The retired Marine Corps general's stinging criticism of Russia stands in sharp contrast to the views of his future boss. Trump has repeatedly praised "very smart" Russian President, Vladimir Putin's leadership qualities and advocated closer ties.
Asked during his confirmation hearing how he viewed the strains facing the modern world order, Mattis said it was under the biggest attack since World War II.
"And that's from Russia, from terrorist groups and with what China is doing in the South China Sea," Mattis told the Senate Armed Services Committee.
His testimony came a day after Trump's pick for secretary of state, former ExxonMobil boss, Rex Tillerson, said he too would take a firm line with Moscow.
Mattis noted that, like Trump, he was open to greater engagement with Russia, but stressed he only had "very modest expectations about areas of cooperation with Mr Putin."
"I'm all for engagement, but we also have to recognize reality and what Russia is up to," he said.
"There's a decreasing number of areas where we can engage cooperatively and increasing number of areas where we're going to have to confront Russia."
US intelligence agencies say Russia interfered in the US election to help Trump, and an explosive -- but unverified -- dossier published this week alleged Moscow had lengthy ties to senior Trump advisors.
While campaigning, Trump called into question the role of NATO, a bedrock of America's global security apparatus, by suggesting countries would be left on their own if they didn't pony up more cash.
Mattis said Putin was working to undermine NATO, and he called the alliance "vital for our national interests," noting that "if we did not have NATO today, we would need to create it.
"Right now, the most important thing is that we recognize the reality of what we deal with with Mr Putin, and we recognize that he is trying to break the North Atlantic alliance," Mattis said.
With regard to China, which has a rapidly growing military, Mattis said it is "shredding trust along its periphery" as it builds up its military presence in the South China Sea.
In written testimony, he said America "must try to engage and collaborate with China where possible, but also be prepared to confront inappropriate behavior if China chooses to act contrary to our interests."
Trump has frequently spoken out against China's trade policies and ruffled feathers in Beijing when he took a call from Taiwan's leader after the election.
Mattis sailed through his three-hour confirmation with broad support, despite some senators expressing reservations that his appointment runs counter to decades of Pentagon tradition -- and US law.
A cornerstone of US democracy is that civilians, not people in uniform, must have control of the military, and the commander-in-chief is the president.
US law prohibits officers from serving as defense secretary for seven years after leaving active duty -- Mattis only hung up his uniform in 2013.
The 66-year-old Washington state native is expected to get a special congressional waiver -- only granted once before, for the famous World War II General George Marshall who served under President Harry Truman from 1950-1951.
"Civilian control of the military is a fundamental tenet of the American military tradition," Mattis said.
US politicians have lauded Mattis, and his appointment appears to be a formality.
Democratic Senator, Richard Blumenthal said he would vote against the waiver because he worried about the precedent it could set.
"But let me say that very bluntly, if ever there were a case for a waiver of that principle, it is you at this moment in our history," Blumenthal said.
"Your appreciation for the costs of war in blood, treasure and lives and the impact on veterans afterwards will enable you to be a check on rash and potentially ill-considered use of military force by a president-elect who perhaps lacks that same appreciation."
A colorful commander famed for his pugnacious aphorisms, the media dubbed Mattis "Mad Dog" for his battle-hardened swagger and the sort of blunt language Marines are famous for.
He has been quoted as saying, "Be polite, be professional, but have a plan to kill everyone you meet."
Mattis would replace technocrat Ashton Carter, President Barack Obama's fourth Pentagon chief.
jpegMpeg4-1280x720Senators also grilled Mattis over a slew of security issues, including the pace of the fight against the Islamic State group, North Korea's nuclear ambitions, women in combat and budget constraints.
Trump has said his administration "will begin a major national effort to rebuild our badly depleted military."
He wants more ships, troops, planes and weaponry.