Fredy Moreno Mahecha, was handed over in a rural area of Arauca department in eastern Colombia, where the ELN captured him two weeks ago.
The news came as a new boost to President Juan Manuel Santos's quest to bring "complete peace" to Colombia after sealing a historic peace deal last year with the country's oldest and largest rebel group, the FARC.
The International Committee of the Red Cross said the soldier, Fredy Moreno Mahecha, was handed over in a rural area of Arauca department in eastern Colombia, where the ELN captured him two weeks ago.
"We trust this gesture will reinforce trust between the Colombian government and the ELN before the imminent opening of the public phase of peace negotiations," the head of the Red Cross in Colombia, Christoph Harnisch, said in a statement.
Colombia expert Kyle Johnson of the International Crisis Group called the release a "very important" gesture ahead of the peace talks, which are set to open Tuesday in the Ecuadoran capital Quito.
"The talks would have begun with serious doubts hanging over them" had Moreno not been released, Johnson said.
A trust-building prisoner exchange had already taken place last Thursday, when the ELN released their highest-profile hostage, former lawmaker Odin Sanchez, in exchange for two rebel prisoners.
The ELN's previous refusal to free Sanchez had already derailed the peace talks once, just as they were due to open last October.
His release left Santos, who won the Nobel Peace Prize last year, brimming with optimism going into the talks.
"This conflict is over," he said last Thursday as he opened a summit of Nobel Peace laureates in Bogota.
"The public phase of negotiations between the Colombian government and the ELN... will enable us to achieve complete peace."
Colombia's Cold War-era conflict, which has killed more than 260,000 people and left 60,000 missing, is the last major armed conflict in the Americas.
The territorial and ideological war has drawn in other leftist rebel groups, right-wing paramilitaries, drug gangs and the army since the FARC and ELN were launched in 1964.
Rights groups say massacres, rapes and other atrocities have been committed on all sides.
The ELN, or National Liberation Army -- a leftist guerrilla group inspired by the Cuban Revolution -- has an estimated 1,500 fighters.
The peace talks come after three years of secret preliminary negotiations.
The government is currently implementing a historic peace deal signed last November with FARC, or Revolutionary Armed Forces of Colombia, a Marxist rebel group whose fighters now have six months to disarm.
But experts warn the ELN will likely be a tougher negotiating partner. And elections next year to decide Santos's successor threaten to complicate matters.
The peace process faces ongoing resistance from conservative opponents who accuse Santos of granting impunity to rebels guilty of war crimes.
Santos had to tweak the initial FARC accord after voters narrowly rejected it in a referendum last October, a major embarrassment for the government.
The slightly revised version was ratified in Congress, where Santos enjoys a majority.