David Warner (109) set up Australia's 288-6 in Basseterre after they elected to bat first in Saint Kitts.
South Africa, who claimed victory in the last meeting between the teams, looked on track for a convincing win.
Faf Du Plessis (63) and Hashim Amla (60) put them in control as they reached 177-2 at Warner Park.
But South Africa collapsed in their chase, losing 75-8 as Mitchell Starc (3-43), Josh Hazlewood (3-52) and Adam Zampa (3-52) did most of the damage.
Even after Amla fell by picking out Steve Smith at cover off Hazlewood, South Africa appeared in control.
Starc took the big wicket of Du Plessis but AB De Villiers (39) kept his side cruising, only to be clean bowled by Hazlewood.
Warner hurt his thumb dropping JP Duminy, yet the collapse would start with South Africa needing 70 off the last 10 overs.
Farhaan Behardien (4) was LBW to Zampa despite a review and the leg-spinner removed Duminy (41) a ball after being hit for six.
Kyle Abbott (1) lasted just three balls and Wayne Parnell (3) quickly followed as South Africa's chances took a huge blow.
Substitute Glenn Maxwell took a screamer to get rid of Aaron Phangiso (3) and Imran Tahir (6) was run out to complete Australia's win.
Warner's injury is the only concern for Australia after the opener's heroics.
His century was his first in ODIs outside of his home nation as he made his 109 off 120 balls, with 11 fours and two sixes.
Usman Khawaja (59) and Smith (unbeaten 52) were also good contributors in a total that appeared lean until the late drama.