Conte's men went into the game having not lost in 49 qualifying matches but fell behind in the 23rd minute courtesy of Alexander Tettey's low strike from the edge of the area.
But Alessandro Florenzi restored parity in the 73rd minute and then, after being denied a second by the offside flag, set up Graziano Pelle for the winner eight minutes from time.
The Southampton striker's rasping volley sealed the points and extended Italy's qualifying unbeaten run to 50 games, while also making sure of qualification for Croatia, who leapfrogged Norway into second place by beating Malta 1-0.
Conte told Rai Sport: "We have to compliment our opponents, as Norway are a strong side and I'm sorry they went into the play-offs. This is football.
"We played the way we had to, trying to give a sense of consistency. We didn't deserve to be a goal down, this is the right path and I want this intensity and grit. We can lose or win, but the important thing is this attitude."
Italy started the match in a 3-5-2 formation but a switch to a 4-2-4 system prompted the fightback.
Conte added: "The plan was always to go 4-2-4 later in the match. This versatility has to be one of our greatest strengths."