Ryan Gosling had put the hosts a goal up in the first-half before Victor Anichebe levelled.
Jermain Defoe's second-half penalty gave 10-man Sunderland their first win at the 11th attempt in the Premier League this season as they beat Bournemouth 2-1 away on Saturday.
The 34-year-old former England striker's sixth goal of the season -- against a team where he spent a productive season on loan as an 18-year-old scoring 18 goals for them -- completed the Black Cats comeback.
Ryan Gosling had put the hosts a goal up in the first-half before Victor Anichebe levelled only for former South African international Steven Pienaar to be sent off but it was the Black Cats who had the final say through Defoe.
Sunderland move on to five points but stay bottom because of a worse goal difference than Swansea who host Manchester United on Sunday.
Victory also ensures Sunderland will not go on to earn the tag of worst start to a Premier League league start.
They shared it with Manchester City, who began the 1995-96 season in identical fashion before losing their 11th match 6-0 against Liverpool.
The hosts made a cracking start as the Sunderland players looked as if the only thing they learnt on their visit to the Nissan car factory this week was how to operate reverse gear as they were on the back foot from the start.
Ireland international Harry Arter produced a superb long range effort inside the first five minutes but Sunderland's young goalkeeper Jordan Pickford saved superbly.
However, he could do nothing about Gosling's goal as he converted via his stomach a great cross by Adam Smith for his third goal of the campaign.
Pickford kept his side in the game as he pulled off another terrific stop from Joshua King who had been set up by the impressive Junior Stanislas.
The Sunderland fans must have seen something that the rest of the onlookers had missed for their rendition of Bob Marley's "Don't worry about a thing, every little thing's gonna be all right," proved prescient as the Black Cats drew level.
Defoe teed up Anichebe and he turned smartly in the area holding off Simon Francis and fired the ball past Artur Boruc.
Sunderland's tails were up and a fine sweeping move involving Anichebe and Defoe led to Paddy McNair whipping in a cross which was turned in by Duncan Watmore only for it to be ruled out for offside.
Sunderland continued in the same vibrant form in the second-half only for their momentum to be halted as Pienaar was sent off for a second bookable offence.
Pickford then prevented the Cherries taking the lead as he denied England midfielder Jack Wilshere a goal on his 100th Premier League appearance.
However, the game swung back in Sunderland's favour when Smith brought down Anichebe and Defoe slotted home the resulting penalty with 16 minutes remaining and ending Boruc's run of three saved spot kicks.
Bournemouth threw everything at the Sunderland defence, substitute Benik Afobe should have done better on two occasions whilst Pickford once again came to his team-mates' aid with a stunning save from Joshua King at close range.