Much of the architecture bears witness to key Afro-Brazilian links in African history, but not a single building, not even the mosque, has been classified as a UN World Heritage Site, Mourchid said.
With a fine ceramic minaret, multicoloured walls and a great, carved wooden door, the Grand Mosque in Benin's capital is a marvel of Afro-Brazilian architecture, yet like much in Porto-Novo it faces collapse.
"It looks just like a church in all ways because the former slaves were used to building churches in Brazil," said Moubarak Mourchid, head of the city's heritage agency.
"They converted to Islam as a show of rebellion against their masters and when they came back to west Africa, they became craftsmen who used the building techniques they had learned over there," the historian told AFP.
"These skills were then transmitted from generation to generation."
Yet unless urgent steps are taken, residents and local specialists say hundreds of these old buildings are at risk.
From the end of the 18th century, Porto-Novo on the Gulf of Guinea was one of the African ports that took in freed slaves who wanted to sail home to the continent of their ancestors.
The streets in what is now Benin's second city and administrative capital call to mind the old Portuguese colonial settlement of Salvador de Bahia in Brazil.
And much of the architecture bears witness to key Afro-Brazilian links in African history, but not a single building, not even the mosque, has been classified as a UN World Heritage Site, Mourchid said.
- 'No political will' -
"It must first be recognised as part of the national heritage by the state. But there's no political will concerning parts of our patrimony," he said.
Without government intervention, the buildings would be left to rot but for work carried out by the House of Patrimony and Tourism of Porto-Novo, an office seen as a big step towards safeguarding Africa's heritage when it was founded in 2009.
Today, a handful of committed enthusiasts are trying to fight against years of neglect, damage from rainy seasons and people ready to let their property disintegrate to cash in on its real estate value.
In 2009, Porto-Novo's director of culture and patrimony Richard Hounsou counted 450 buildings of the Afro-Brazilian type but said today "there are fewer than 400 left, that's obvious."
Ali Moubarak lives in one of the many imposing homes. Dressed in a long white tunic, he receives visitors and shows them the creeping damage to his three-storey house.
The building is divided in two. From the inner courtyard, you can see right into the rooms, like a doll's house. "It was built by my grandfather, around 1910. He was a nurse, he was an important man," Moubarak said.
Like other prominent citizens of his time, Moubarak's grandfather had his home constructed in traditional Afro-Brazilian style as a show of wealth. He had "four or five wives and I don't know many grandchildren," added Moubarak.
"Under Beninese law, we are all heirs and it's the oldest who make decisions."
The walls began to flake as the family grew poorer. With all the comings and goings, Moubarak does not know exactly how many people live under his roof. Cousins and children are among the residents, with teenagers working in a laundry set up in one bedroom. Another has been turned into a sewing shop.
In a magnificent room on a third floor, the old glory of the house is preserved in the wooden and bamboo beams, walls painted in floral patterns and a finely carved chest with a period mirror whose frame may soon be used for firewood.
The city of Porto-Novo has offered to restore the property and Moubarak has accepted, but he needs the consent of other heirs living in the commercial capital of Cotonou or in distant France, who have shown little interest in the architectural heritage.
The alternative is to let the house collapse once and for all, the better to sell a sizeable plot of land or use the site to build several new houses and rake in a profit.
Renovation has started on another home not far away but concrete is used to replace walls, rather than the traditional mix of sand and limestone, while trying to respect "the shape and motifs of the time", said historian Mourchid, who is supervising the work.
"It would cost twice as much to use the original materials."
In all, the city has inventoried 41 homes eligible for renovation if the owners agree.
One belongs to Josephine Rey, an elderly woman who sells coal and wood. She loves the old home her father left her but says she has waited two years already for help repairing her roof.
"I spend my nights praying that it doesn't fall down on my head," she said.
Both money and expertise are problems, conceded another heritage agency official Georges Bankole, who said there are plans to open a school with Brazilian partnership to train artisans.
"The budget for patrimony is the smallest in the entire culture ministry," he said.
And "beyond the problems of money, we've lost building practices," said historian Mourchid, regretting that great houses gradually turning to dust are taking the memories of freed slaves with them.