Critics have slammed the cost involved in hosting the event and said taxpayers should decide how their money is being spent.
The initiative, championed by the sports-mad Orban, was dealt a possibly fatal blow last week after a petition garnered enough signatures to force a referendum on the bid.
Mayor Istvan Tarlos was set to meet the right-wing premier on Wednesday evening to discuss ending Budapest's candidacy given the lack of "national unity" regarding the matter, according to a statement by the municipal council.
A decision could be expected within days.
"There's no point in dragging things out," Tarlos told council members during a meeting on Wednesday morning.
The mayor had previously said he would not "ignore the will of the people" in light of the referendum push.
Over a quarter million signatures of Budapesters were collected within a month by a group of young activists, aided by several political parties critical of the strongman Orban.
Recent polls have also shown clear majorities in favour of withdrawing the bid.
"I am sorry to see a dream dissolve," said Zsolt Borkai, the chief of the Hungarian Olympic Committee, at Wednesday's council meeting.
The chairman of Budapest's bid, Balazs Furjes, pushed for a quick decision.
"If those who have turned their back on us still refuse to support us now, there's no point in continuing... We will ridicule ourselves with months of political mudslinging until the referendum, and won't stand a chance," he said.
Budapest has been vying with powerhouses Paris and Los Angeles for the right to host the summer Games, after Hamburg -- following a referendum -- and Rome dropped out, both citing financial concerns.
Orban has backed the bid, launched in 2015, as a reward for his country's rich Olympic record: only nine countries have won more medals in the history of the Games.
But critics have slammed the cost involved in hosting the event and said taxpayers should decide how their money is being spent.
Bid supporters however insisted that Budapest, which unlike Paris or Los Angeles has never hosted the Games, was more suited than its rivals to the International Olympic Committee's low-cost Agenda 2020 strategy.
The IOC is set to decide between the candidate cities on September 13 in the Peruvian capital Lima.