President Putin laid red roses at the foot of Karlov's coffin and spoke with his relatives at the ceremony.
Dozens of colleagues and relatives attended the ceremony for Karlov, the ambassador to Turkey whose death was labelled by Moscow as an act of terror while President Recep Tayyip Erdogan said the perpetrator was a member of Fethullah Gulen's group behind the aborted July coup.
Putin laid red roses at the foot of Karlov's coffin and spoke with his relatives but left the ceremony without making a statement.
Foreign Minister Sergei Lavrov praised the deceased envoy, who was 62, and paid his respects to his mother Maria, widow Marina and son Gennady, also a diplomat, as the ambassador's body lay in state in a flower-decked coffin.
"We are saying goodbye to our friend Andrei Karlov who became a victim of a malicious, vile terrorist attack while in the line of duty," Lavrov said at the ceremony held in the foreign ministry headquarters.
"We will never forget Andrei."
A religious service was later expected to be held at Moscow's Cathedral of Christ the Saviour led by the head of the Russian Orthodox Church Patriarch Kirill before the ambassador is laid to rest at a cemetery.
In terrifying scenes captured on photo and video, 22-year-old policeman Mevlut Mert Altintas shot the ambassador nine times in the back on Monday while he was delivering a speech at an exhibition of photographs of Russia in Ankara.
The ambassador fell to the ground and later died in hospital.
The assailant, who was off-duty and managed to circumvent the metal detectors by flashing his police credentials, shouted "Allahu Akbar" (God is greatest) and "Don't forget Aleppo" after targeting Karlov and was himself killed in a subsequent shootout with Turkish guards.
Altintas had no prior criminal record but Turkish authorities have moved to link the murder with Gulen, a preacher living in self-imposed exile in the United States whom Ankara previously blamed for orchestrating a coup against Erdogan.
Pro-government press had reported that police discovered pro-Gulen literature belonging to Altintas and sympathisers of the preacher in his circle.
Erdogan went as far as to say that the killer "was a member of the FETO (Fethullah Terror Organisation)."
Gulen has denied involvement in both the coup and the envoy's assassination, and Moscow has also refrained from assigning blame, with Kremlin spokesman Dmitry Peskov warning against "rushing to conclusions" before the investigation is complete.
A group of Russian investigators has been working on the probe in Turkey since Tuesday.
Turkish prosecutors on Thursday said they have released six relatives of Altintas who were detained for questioning in the wake of the attack.
Thirteen people were arrested in the murder probe and police were looking for 120 people, authorities said.
Russia has bestowed a prestigious Hero of Russia honour on Karlov posthumously, while his alma mater MGIMO Institute of International Relations has initiated a scholarship in his name.
Karlov studied Korean and Japanese as he trained for his diplomatic career and worked for many years in North Korea, including as ambassador between 2001 and 2007. He has served as envoy to Turkey since 2013.