Sarkozy's hopes of winning back the presidency were crushed as he scored just 21 percent, according to near-complete results.
Sarkozy was beaten into third place after a stunning upset by Francois Fillon, who served as his prime minister, with the veteran Alain Juppe finishing second.
The surprise result puts Fillon in a commanding position for next Sunday's second round of a contest that is widely expected to decide France's next leader.
With the French leftwing in disarray, the rightwing candidate is tipped to face -- and beat -- far-right leader Marine Le Pen in the presidential run-off next May.
But after a wave of populism saw British voters choose to leave the European Union and swept Donald Trump to the White House, no-one is writing off Le Pen's chances.
In a major upset, Fillon, a pro-business conservative, took more than 44 percent of the vote compared to around 28 percent for Juppe, a former prime minister and foreign minister. Sarkozy's hopes of winning back the presidency were crushed as he scored just 21 percent, according to near-complete results.
Sarkozy immediately endorsed Fillon and said he would now withdraw from political life.
"I fought for my beliefs with passion... I did not manage to convince the voters," he told supporters.
"I have great respect for Alain Juppe, but Francois Fillon's political choices are closer to mine," Sarkozy added.
Fillon, 62, pulled off a remarkable come-from-behind victory in the first round after trailing Sarkozy and Juppe in all but the final days of the two-month campaign.
Voters appear to have warmed to Fillon's understated style over the brashness of 61-year-old Sarkozy, who still deeply divides the country four years after being turfed out of office by the Socialist Francois Hollande.
Juppe, 71, was the early frontrunner but Fillon made stunning progress thanks largely to strong performances in three televised debates.
Fillon told his ecstatic camp that his programme was one of "hope". He said he had a "special thought" for Sarkozy, whom he served as prime minister from 2007 to 2012.
It appears that the participation of leftwing voters in the first ever US-style rightwing primary to be held in France could have been a factor in dragging down Sarkozy.
Anyone who paid two euros ($2.1) and signed a statement saying they adhered to "the values of the centre and the right" could take part.
Many Socialist supporters who turned out are thought to have done so to block the former president.
Turnout was high with around four million people casting ballots, according to a preliminary count.
One Socialist voter, a sports teacher in his fifties who identified himself only as Eric, told AFP he had taken part to vote "against Sarkozy".
"I'm fed up of that guy, he thinks he is all-powerful and he has been involved in too many scandals. Juppe, despite everything else, is the opposite," he said as he cast his vote in the Paris suburb of Pantin.
In a final TV debate among the seven candidates on Thursday, Sarkozy angrily ducked a question about fresh claims that he received millions in funding from the late Libyan leader Moamer Kadhafi towards his 2007 campaign.
The case is one of several investigations to dog Sarkozy since he left office after what was dubbed a "bling-bling" presidency because of his flashy lifestyle.
Juppe and Fillon have broadly similar programmes, underpinned by pledges to reinforce domestic security in a country still under a state of emergency following jihadist attacks that killed more than 230 people.
Fillon has promised to slash 600,000 jobs from France's bloated civil service.
The nomination of the right-wing candidate on November 27 is expected to trigger an announcement from Hollande on whether he intends to bid for re-election despite the lowest popularity ratings of any post-war president.
Hollande's former economy minister Emmanuel Macron, 38, has announced he will stand as an independent, further confusing the picture on the left.