More than 120,000 Rohingya have languished in camps since they were driven from their homes by sectarian unrest between Buddhists and Muslims that engulfed western Rakhine State in 2012.
Most are not allowed to leave the squalid displacement camps where they live in piecemeal shelters with little access to food, and denied access to basic education and healthcare.
Myanmar's de facto leader Aung San Suu Kyi last year appointed Annan to head a commission tasked with healing long-simmering divisions between Buddhists and Muslims in Rakhine.
"The commission calls for a plan to close all IDP camps in Rakhine state," Ghassan Salame, a member of the body, told reporters at the launch of the body's interim report.
The report also called for the government to ensure "security and livelihood opportunities at the site of return/relocation" for those leaving the camps, including by building new houses.
Rohingya should also be given a transparent path to becoming citizens and restrictions on the movements of those who already have it should be lifted, it added.
Myanmar has long faced international condemnation for its treatment of the Rohingya, who many in the Buddhist majority reject as illegal immigrants from Bangladesh.
The issue has reached boiling point in recent months after the army launched a bloody crackdown in the north of Rakhine after deadly attacks on several police border posts in October.
UN investigators who interviewed escapees in Bangladesh have accused Myanmar's security forces of responding with a campaign of murder, gang rape and arson that may amount to genocide.
Rights envoy Yanghee Lee called the on UN to launch its highest-level probe into the violence, which she said may be part of a government campaign to drive the Rohingya from the country.
But a draft resolution tabled by the UN Human Rights Council stopped short of calling for a Commission of Inquiry into the violence.
Salame said the Annan commission backed calls for an independent investigation into the violence in northern Rakhine in its report, but said anything further would be beyond the body's remit.
It identified three initial camps to close -- one housing over 200 Rohingya along with two others that are home to Buddhist Rakhines and Kaman Muslims who were also displaced in the 2012 violence.
Suu Kyi's office welcomed the report and said it would implement the "large majority of recommendations" without giving more details.
Eyewitness? Submit your stories now via social or:
Email: news@pulselive.co.ke
