The contest is widely seen as a two-way race between Susana Diaz, the president of the southern region of Andalusia, and former party leader Pedro Sanchez who was ousted in a rancorous internal rebellion.
Diaz, 42, is the establishment favourite and is seen as more amenable to striking deals with the government if she wins the primary while Sanchez, 45, promises a more adversarial approach.
Rajoy's minority Popular Party (PP) government has relied on the Spanish Socialist Workers' Party (PSOE), the main opposition party, to pass key measures in parliament since it was sworn in for a second term in October 2016.
But Sanchez would have "a more fraught relationship" with the government and the risk of snap elections would increase with him in charge of the Socialists, Eurasia Group analyst Federico Santi said.
"If Sanchez wins, I expect that within a year, a year-and-a-half at most, Rajoy will call early elections," added a former PSOE strategist.
Nearly 190,000 card-carrying members of the PSOE are eligible to vote in the leadership primary at 2,900 polling stations nationwide, with the results expected after 9:00 pm (1900 GMT).
'Petty squabbles'
At a polling station in Moncloa, west of Madrid, close to the party's main headquarters, a representative said that by noon almost a quarter of the 260 people registered there had cast their ballots.
"We've lost our way for some time," said 71-year-old pensioner Arturo Rivero, who joined the party in 1978, and who said he voted for Diaz.
"We have an opportunity to unite. From tomorrow we have to calm down and engage in national politics, forget petty squabbles, and convince people to vote for us."
Former economics professor Sanchez was forced to resign in October over his staunch refusal to allow Rajoy to form a minority government following two inconclusive national elections.
The architects of the messy rebellion argued it was best to let a conservative PP government through rather than go to the third elections in a year and risk losing even more votes.
Sanchez has called this strategy a huge mistake, deeply critical of corruption scandals impacting Rajoy's PP and of their severe austerity measures.
He argues that the Socialists must move further to the left to have any hope of winning back voters who have drifted to new far-left party Podemos.
He has not ruled out submitting a no confidence motion in Rajoy if he wins the leadership race.
Slump in polls
By contrast Diaz, a plumber's daughter and career politician who was one of the leaders of the rebellion against Sanchez, talks of providing "constructive opposition" to the PP.
After an inconclusive local election in Andalusia in 2015, she made a pact with centre-right Ciudadanos -- Rajoy's allies at the national level -- to reach a parliamentary majority and form a government.
Diaz was widely tipped as the favourite but Sanchez has done better than many expected, gathering around 53,000 signatures for party members in favour of his candidacy, just 7,000 less than his rival.
Whoever wins the leadership faces a tough job ahead.
Like other Socialist parties across Europe, the PSOE has seen its support slump in recent years as voters flock to new formations like Podemos.
"The future of the Socialist Party is being decided at a time when social democracy in Europe is going through very critical times," said 53-year-old oil worker Arturo, who joined the party in 1986 and voted for Sanchez.
The PSOE won just 85 of the 350 seats in parliament in last year's general election, its worst showing since Spain returned to democracy following the death of longtime dictator Francisco Franco in 1975.
While the Socialists were busy with their primary vote, Podemos staged a large protest against corruption in Madrid on Saturday, a day after it filed a motion of no confidence against Rajoy, citing graft scandals affecting his ruling PP.
The motion lacks enough support from other parties to pass but could boost its appeal to voters angry at the government who would otherwise back the Socialists.
Eyewitness? Submit your stories now via social or:
Email: news@pulselive.co.ke
