Osvaldo, who replaced the injured Mauro Icardi after 25 minutes, found the net from 10 yards with a superb scissor-kick from Fredy Guarin's free-kick after Rodrigo Palacio had missed a penalty and Nemanja Vidic was denied by the woodwork at San Siro.
And another substitute, Hernanes, added a second when he curled in a free-kick with four minutes remaining to leave Inter with two wins and two draws from their first four Serie A games of the new campaign.
The result also represents a first defeat for Atalanta against Inter since April 2010 and means they have now lost their last two league matches.
The Inter home crowd have been spoilt so far this season, with two home games yielding 13 unanswered goals, including a 7-0 drubbing of Sassuolo in their only Serie A game at San Siro earlier this month.
However, a 1-1 draw with Palermo on Sunday left Inter coach Walter Mazzarri to bemoan a lack of sharpness from his forwards.
Mazzarri's solution was to draft in Inter's top-scorer for the last two seasons, Palacio, for his first start of the campaign following an injury he suffered at this year's World Cup, and also switched to a back three with Jonathan and Dodo starting as wing-backs.
Palacio, who played the last 11 minutes of Sunday's game, had two glorious opportunities to open the scoring for Inter in the first half, but he was denied both times by superb saves from Marco Sportiello.
First, the striker was put through on goal only to see his shot parried by Sportiello and then, after Andrea Ranocchia was bundled over by Yohan Benalouane in the penalty area, the goalkeeper produced an even better diving save to push away the Argentine's spot-kick.
Inter, who also hit the post in the first half through a stooping Vidic header, were seemingly dealt a blow when Icardi, who scored a hat-trick against Sassuolo, departed injured.
However, it was his replacement, on-loan Southampton forward Osvaldo, who finally gave Inter a 40th-minute lead with a superb scissor-kick from Guarin's delivery.
Inter continued to dominate after the break, but struggled to add to their lead.
Palacio came close to another goal, but it summed up his evening when his angled shot from inside the box came back off the foot of the post.
The visitors almost made Inter pay for their wastefulness when Marcelo Estigarribia headed his side's best chance of the game just a yard wide at the back post in the 70th minute before Hernanes' free-kick " awarded for a foul on Osvaldo on the edge of the box - wrapped up the points.