"We will continue calling on Congress to de-finance the forces of deportation," Maria Bilbao, a Florida organizer for the pro-immigrant group United We Dream, told AFP on Friday, referring to ICE.
"Now is the time to stop and take to the streets to fight for an end to these abuses."
Starting in early May, President Donald Trump ordered the arrest of adults crossing the border illegally, including those seeking asylum.
As a result, children were separated from their families and held first in "processing centers," then in shelters.
Widely broadcast pictures of young children held in enclosures behind chain-link fences sparked protests across the US -- with far larger demonstrations expected on Saturday -- and international outrage against Trump's "zero tolerance" policy.
"Occupy ICE" camps have also been set up in several US states.
Last week, Trump signed an order ending the separation of families, but some 2,000 children have yet to be reunited with family members, and immigration lawyers say the process of doing so will be long and chaotic.
In this turbulent political context, one of the first voices to call for the abolition of ICE was that of New York gubernatorial candidate Cynthia Nixon, who as an actress played the part of Miranda in the popular TV series "Sex and the City."
"I think we need to abolish ICE. That seems really clear," the Democratic candidate told ABC television's "The View" last week.
"They have strayed so far from the interests of the American people and the interests of humanity."
US lawmakers are voicing the same demand, along with human rights organizations, who note that ICE is a relatively recent creation.
The office was established only 15 years ago, a move intended to help "protect national security and strengthen public safety," in response to the September 11, 2001 attacks, as the agency's website says.
But Hemanth Gundavaram, co-director of the Immigrant Justice Clinic at Northeastern University in Boston, said ICE has been diverted from its stated mission, used as a tool by President Donald Trump to implement his "racist and xenophobic" immigration policy.
"Our immigration system should not exist to focus solely on national security and terrorism, Gundavaram told USA Today.
Cornell Law School immigration professor Stephen Yale-Loehr said it was "extremely unlikely" that Congress will do away with ICE.
"Even if the Democrats take control of Congress in November, the chances of abolishing ICE are slim to none. Every agency has to have an enforcement branch; immigration is no exception," Yale-Loehr said.