After the vote Tuesday, Jobbik lawmakers held up a banner reading: "The traitor is he who lets in terrorists for money.
While all 131 MPs of Orban's ruling right-wing coalition voted in favour, the bill failed to pick up an extra two votes to reach a required two-thirds majority in Hungary's parliament.
Jobbik has been a natural ally to Orban in the populist right-winger's bid to stop Hungary taking in any of the hundreds of thousands of migrants who arrived in Europe in 2015.
But Jobbik is currently vying with the Socialists as the second most popular party in the country, and laid a surprise ambush for Orban earlier this month.
Spotting a rare opportunity for leverage, Jobbik's leader Gabor Vona had announced after a recent one-on-one meeting with Orban it would vote for the bill.
But this was on the condition that the government scraps a controversial cash-for-residency bond scheme for wealthy foreigners, particularly from Russia, China and the Middle East.
After the vote Tuesday, Jobbik lawmakers held up a banner reading: "The traitor is he who lets in terrorists for money." It included a logo of Orban's Fidesz party in an Arabic-style font.
Jobbik have long called the residency bonds, generally sold via offshore companies, a "dirty business" and a national security risk that could be exploited even by Islamic State jihadists.
"Neither poor nor rich migrants should be allowed to settle in Hungary," Vona said.
Unused to not getting his own way since winning a supermajority in 2010 and reelection in 2014, Orban was caught offguard by the ultimatum and initially said he would "consider" Vona's gambit.
But Orban later told parliament that the government "would not give in to blackmail" and urged Jobbik not to connect the bond scheme with the change to the constitution which he called "an affair of national importance".
Observers say Tuesday's vote could set the scene for a growing power struggle between Fidesz and Jobbik ahead of the next scheduled general election in 2018.
"If Jobbik do not back the amendment, it would unquestionably wound Viktor Orban. It would be the first time since 2010 that the parliament votes against the PM, or that Orban has failed to get through something important to him," leading news website Index.hu wrote on Monday.
The parliamentary ballot follows an expensive and divisive referendum on October 2, in which 3.3 million voters backed Orban's rejection of the European Union's troubled refugee quota plan.
The ballot, however, was deemed invalid due to low turnout in the nation of nearly 10 million people.
Undeterred, Orban still hailed the outcome as "a sweeping victory" over "Brussels bureaucrats" and vowed to change the constitution to "reflect the will of the people".
Submitting a draft amendment personally to parliament the week after the referendum, Orban proposed a ban on migrant settlement without the approval of the national assembly and authorities.