"We are happy our soldiers stood with us, the people. They must finish the good job to make sure Mugabe goes. We have been suffering," said one of those in the euphoric crowd, Baliwe Sibanda.
"We are so happy they freed us from the Mugabe dynasty."
Demonstrators in the city centre swigged from beer bottles and danced in the road as the protest took on a carnival atmosphere.
Zimbabweans were intially stunned when the army seized control on Tuesday apparently over fears Mugabe was laying the foundations for his wife Grace to take power after he abruptly fired his vice president Emmerson Mnangagwa.
But that shock has turned to joy for many Zimbabweans -- the majority of whom have only known life under Mugabe's rule which has been defined by violent suppression, economic collapse and international isolation.
"Just look at the crowds here... Finally we can hope to have jobs when I finish university," said Tafadzwa Musarurwa, a 22-year-old social studies student at the University of Zimbabwe who marched in the central business district.
Nearby a group of young men tore down a green metal street sign bearing Robert Mugabe's name and smashed it repeatedly on the road before trampling it under foot.
Ten kilometres (six miles) away in the blue-collar Highfield suburb of Harare, as many as 10,000 protesters gathered at a vast sports field to voice their opposition to Mugabe.
The march, organised by veterans of Zimbabwe's independence war who were once ferociously loyal to the autocratic leader but now back Mnangagwa, was deeply symbolic.
Highfield was where Mugabe gave his first speech after returning from exile in Mozambique ahead of independence in 1980.
"This is a great day for us. For 37 years we have had nothing to show for participating in the liberation war while his family were living it large," said Sonia Kandemiri, a 59-year-old veteran of the campaign against white-rule.
The growing momentum against Mugabe appeared to have transcended demographic barriers with white and Indian Zimbabweans joining members of the country's Shona and Ndebele communities for the mass gathering.
"I am glad that Mugabe is going during my lifetime. He is the source of every problem in our beautiful country," said Irene Douglas, 67, a former farmer whose land was confiscated during Mugabe's ruinous farm seizure programme from the year 2000.
Several past members of Mugabe's government including former vice president Joice Mujuru and ex-finance minister Patrick Chinamasa also attended the protest.
Morris Craig, a 49-year-old computer programmer who brandished the Zimbabwean flag, said it was "good people are here".
jpegMpeg4-1280x720"The suffering of the people must end," he said as the crowd began to march to Mugabe's opulent private residence, known as the Blue Roof.