The Antarctica attempt, which has only been completed twice before, takes sailors past the three most notorious capes on the plane.
Lisa Blair was 72 days into the non-stop and unassisted 14,000-nautical mile voyage around the icy continent on her yacht "Climate Action Now".
She was aiming to break the record set by Russian adventurer Fedor Konyukhov who braved storms, freezing temperatures, and the threat of icebergs to complete the journey in 102 days, 35 minutes and 50 seconds.
But the conditions have taken their toll, with her support crew saying the yacht's mast broke early Tuesday 895 kilometres (554 miles) south of Cape Town in seven metre (22 feet) swells and 40 knot winds.
"The mast is currently being dragged behind the boat. She is regrouping after making the boat safe and at first light will assess the situation," her spokeswoman Tracey Whittaker told AFP.
"With the damage she has, the record attempt is abandoned," she added, shortly after speaking with Blair, who was unhurt in the incident.
Once the full extent of the damage is known, her team will work on options for Blair to best reach land.
"Of course she is disappointed," Whittaker said. "At 72 days, she was on target to break the record."
So far Blair, who started her voyage at Albany in Western Australia, has only issued a pan-pan call signifying a problem, and not a mayday, which is used in life-threatening situations.
Search and rescue in Cape Town have been notified and will provide assistance if required, her support crew said.
In a blog post on Monday, Blair said it had been a testing 24 hours, including a wave rolling the boat.
"After I battled with the zipper of my sleeping bag I then glanced outside. It was a complete white-out with rain sheeting sideways and the winds howling through the rigging," she said.
"Behind me, I could see crystal clear blue skies and in front of me was this frothing boiling mass of angry wind and rain."
The Antarctica attempt, which has only been completed twice before, takes sailors past the three most notorious capes on the planet -- Cape Leeuwin, Cape Horn and Cape Agulhas.