The 2017 invasion of Marawi by gunmen waving the black flag of the Islamic State group sparked a five-month battle with the country's army.
More than 1,000 people were killed, and swathes of the southern city reduced to bombed-out rubble.
ICRC president Peter Maurer told reporters he visited Marawi on Monday, and later discussed the reconstruction and rehabilitation efforts at a meeting with Philippine President Rodrigo Duterte.
"It is painfully slow for the victims, painfully slow for those who have expectations, which are legitimate expectations, to go back," Maurer said.
He said this was "not an extraordinary situation" compared to post-war efforts in the Middle East and Africa -- particularly after urban conflicts, where artillery and air strikes cause massive destruction.
"We hope it can be further accelerated and we hope that we don't have to wait for another two years till substantive progress is sizable," he added.
More than 100,000 residents remain homeless, many of them living in squalid relocation camps two years after the start of the conflict, AFP reporters who joined a government-sponsored visit to the area last month saw.
Demolition of blast-pocked buildings has begun, but after several false starts the government does not expect rebuilding to be finished before the end of 2021.
Analysts have warned the anger of the displaced Marawi population could fuel jihadist recruitment.
Maurer said he discussed the Marawi displacement with Duterte, who noted the complex task of clearing the ground of explosives so reconstruction could safely proceed.
"He highlighted that of course, he noticed, as we noticed as well, that there is a comprehensible level of dissatisfaction with the fact that this is taking a long time," Maurer added.
"I've felt that the president is certainly determined to support the Philippine Red Cross and ICRC in our respective efforts to accelerate this process and to encourage his own administration to do the best possible, to go back to a more accelerated pace of reconstruction."
Marawi is the Muslim capital of the nation's south, which has been locked in a cycle of poverty and extremism as separatist insurgencies have raged for decades.
Though Duterte has tried to portray himself as sensitive to the Philippines' Muslim minority, he has sent mixed signals about Marawi.
He has claimed the city was home to illegal drug activity, an incendiary allegation from a leader whose narcotics crackdown has killed over 5,300 alleged dealers and pushers.
"I don't think that I should be spending for their buildings. People there have a lot of money," Duterte said in reference to Marawi in an April speech.