The European Commission Vice President has been urged to resign from his position.
Hungary's foreign ministry said Timmermans "should resign from his post after having accused Hungary's Prime Minister and the country's government of anti-Semitism".
Foreign Minister Peter Szijjarto said that the comments were an "unfounded accusation" and that Hungary "has done more than anyone in Europe to combat anti-Semitism," a statement said.
It added that Budapest "has major issues" with the Hungarian-born financier and philanthropist but that "these disputes have absolutely nothing to do with George Soros's origins or religion".
Orban on Wednesday told the European Parliament that Soros was an "American financial speculator attacking Hungary" and that the Hungarian-born financier has "destroyed the lives of millions of Europeans".
Asked in an interview with Germany weekly Die Zeit published on Thursday whether he thought the comments sounded anti-Semitic, Timmermans said: "I understood that in exactly the same way as you and was appalled".
Orban was in Brussels to defend education legislation that the Central European University in Budapest, a highly respected institution founded by Soros, believes is aimed at forcing it to close.
The legislation has drawn street protests and international concern including from the US State Department and an open letter signed by more than 900 academics around the world.
Orban has accused the university, set up in 1991 after the fall of communism, of "cheating" and of having an "unfair advantage" over local institutions -- allegations rejected by the CEU as "defamatory".
Orban's government also aims to oblige non-governmental organisations receiving above a certain amount of foreign funding to register and stamp any publication with "foreign-funded organisation".
Mirroring similar rules in Russia, this is also seen as targeting Soros's Open Society Foundation which funds civil society groups and which has also come under fire elsewhere in the region.
Orban, a strong admirer of US President Donald Trump, has long accused Soros, 86, of seeking to undermine Europe by backing immigration, calling it the "Trojan horse of terrorism".