Iraqi forces have retaken more than 90 percent of the territory IS seized in the country in 2014.
The launch of the keenly awaited offensive that the US-led coalition fighting IS has dubbed "the last big fight" of the campaign came even as Iraqi troops launched a new operation against the Kurds.
There had been fears that the bitter dispute that has raged between the Baghdad government and Iraqi Kurdish leaders since they held a referendum for independence last month would hamper the battle against the jihadists.
But federal troops and allied paramilitaries pressed ahead with a threatened drive up the Euphrates valley towards the Syrian border in a bid to retake two Sunni Arab towns that have been bastions of insurgency since soon after the US-led invasion of 2003.
Iraqi forces have retaken more than 90 percent of the territory IS seized in the country in 2014, with the jihadists now confined to a small stretch of the valley adjoining some of the last areas they still hold in Syria.
"The heroic legions are advancing into the last den of terrorism in Iraq to liberate Al-Qaim, Rawa and the surrounding villages and hamlets," Prime Minister Haider Al-Abadi said in a statement from neighbouring Iran where he is on a state visit.
"They will all return to the arms of the motherland thanks to the determination and endurance of our fighting heroes," he added.
"The people of IS have no choice but to die or surrender."
Regional operations commander General Qassem al-Mohammedi told AFP that government forces were advancing on four fronts -- from the east, southeast, north and south.
He said that units of the federal police and the elite Counter-Terrorism Service as well as the paramilitary Hashed al-Shaabi (Popular Mobilisation) force were supporting the army.
The Joint Operations Command said that by early afternoon government forces had recaptured several military bases southeast of Al-Qaim, including an airbase.
Crucially for an offensive in an overwhelmingly Sunni Arab region, Sunni tribal volunteers in the Hashed were heavily engaged alongside the Iran-trained Shiite militias that are its mainstay.
The US-led coalition said it had carried out some 15 strikes on IS targets in and around Al-Qaim and the town of Albu Kamal on the Syrian side of the border.
Al-Qaim has been renowned as a bastion of Sunni Arab insurgency for years.
US troops carried out repeated operations with names like Matador and Steel Curtain in 2005 to flush out Al-Qaeda jihadists.
Coalition commanders are convinced that Al-Qaim will be IS's last stand in its ambitions of territorial control of the cross-border caliphate it proclaimed in 2014.
On the Syrian side of the border, Russian-backed government forces have been pushing down the Euphrates valley while US-backed Kurdish and Arab fighters have been attacking the jihadists from their stronghold in the north.
The launch of the offensive against IS's last Iraqi redoubt comes with federal troops and militia engaged in an operation to reassert central government control over thousands of square kilometres (miles) of territory long disputed with the Kurds.
Loss of the territory has dealt a crippling blow to the finances of the autonomous Kurdish region and on Wednesday its leaders reached out for talks, saying they were ready to freeze the outcome of the September 25 independence referendum.
The Iraqi prime minister on Thursday dismissed the offer, saying it did not go far enough.
"We will accept nothing but the annulment of the referendum and respect for the constitution," he said in a statement released by his Baghdad office.
Abadi, whose stock has been massively boosted by the success of the fightback against IS, was in Tehran for talks a day after holding meetings in Ankara.
On Thursday, his forces launched a new assault on Kurdish forces in the disputed oil-rich Zummar area of Nineveh province, Kurdish authorities said.
An AFP correspondent reported heavy artillery exchanges as Kurdish forces put up fierce resistance.
Parts of Nineveh province north and east of Iraq's second city Mosul are some of the last areas that Kurdish forces still hold outside their longstanding three-province autonomous region.
Thursday's assault was close to the route of a strategic oil export pipeline linking the Kirkuk fields retaken from the Kurds with the Turkish Mediterranean port of Ceyhan that fell into disuse during IS's lightning sweep through northern and western Iraq in 2014.
Abadi discussed reopening the pipeline in his talks with Turkish President Recep Tayyip Erdogan on Wednesday.
"We are ready to provide any kind of support to allow the operation of the pipeline," Erdogan said.
The course of the disused pipeline passes through the town of Faysh Khabur, near where the borders of Iraq, Turkey and Syria meet, in territory which lies undisputedly inside the autonomous Kurdish region.