The government is seeking a peace deal with the leftist National Liberation Army (ELN) to seal "complete peace" after signing an accord last year with the bigger Revolutionary Armed Forces of Colombia (FARC).
Following a week of talks in Ecuador, the government's lead delegate Juan Camilo Restrepo read out a statement that announcing a "first accord" had been reached.
It covers agreements on the role of civil leaders in the peace process and humanitarian measures to de-escalate the conflict.
"We want to advance gradually toward a scenario where the conditions will be in place to agree a ceasefire and an end to hostilities with the ELN," Restrepo said in an appearance at the Ecuadoran foreign ministry.i
ELN commander Pablo Beltran said the preliminary accord was "a good start" despite the differences between the two sides.
"We are going to start seeking a bilateral ceasefire."
The ELN is the last active remnant of a half-century civil conflict.
The war broke out in 1964 when the ELN and FARC took up arms for rural land rights.
It drew in various groups and has killed at least 260,000 people, according to official figures.
Shortly before Thursday's statement, Restrepo accused the ELN on Twitter of a bombing against a state army patrol on Tuesday that the government said had injured two soldiers.
The ELN has not responded to that claim.
Eyewitness? Submit your stories now via social or:
Email: news@pulselive.co.ke
