The Springboks trailed early after an Israel Folau try in the second minute at Patersons Stadium but looked on course for victory leading 23-14 when Irish referee George Clancy yellow carded Bryan Habana for a high tackle.
Habana, playing in his 100th Test, collected Adam Ashley-Cooper on the shoulder in the 66th minute but Clancy reached for the pocket despite strong protests from Boks skipper Jean de Villiers.
It was a pivotal moment as the Wallabies piled on 10 points while the visitors were reduced to 14 men.
They left it late, though, with Horne bursting clear to touch down after the Boks had made a mess of a late penalty.
It was a much-needed victory for the Wallabies following their 51-20 drubbing by New Zealand last time out, dragging them off the bottom of the Rugby Championship ladder.
South Africa's first defeat of the tournament sees the All Blacks move above them in first place after earlier defeating Argentina.
Morne Steyn kicked six-from-seven with the boot for the tourists including six penalty goals while Foley also missed just one of seven attempts.
It was a dream start for the Wallabies.
Folau stole the ball from the kick-off and scored the opening try in the second minute.
Matt Toomua held the ball up just long enough to attract Jan Serfontein's attention and send Folau sliding over untouched to take an early 5-0 lead.
With a strong wind at their backs the Springboks replied with a Steyn penalty goal and claimed the lead on 13 minutes when right winger Cornal Hendricks finished a sweeping movement out wide.
Steyn narrowly missed the conversion and Foley slotted his first penalty after a good run from Tevita Kuridrani to get back on level terms at 8-8.
Referee Clancy was doing his best to blow the pea out of his whistle and Steyn obliged with two more penalty goals before Foley answered in the 28th minute.
Foley levelled the scores at 14-14 immediately after the resumption but momentum turned immediately back South Africa's way when Toomua kicked the ball dead from his own 30m line.
The Boks won a scrum penalty immediately and Steyn reclaimed the lead in the 45th minute for the first of three straight penalties as the visitors took a stranglehold on the contest.
After Habana was harshly sent from the field, the hosts peppered South Africa's tryline but appeared to have blown their chance when the Boks were awarded a breakdown penalty late in the contest.
But the visitors inexplicably failed to find touch and the Wallabies took full advantage when Horne stepped off his left to complete a 60-metre counter-attack that involved super-sub Kurtley Beale, Folau and Tevita Kuridrani.
Foley, who kicked the winning penalty goal for the Waratahs in the Super Rugby final against the Crusaders, calmly slotted the conversion 15m in from touch to complete an unlikely victory for Ewen McKenzie's side.