Abe, who has seen his popularity plummet in recent weeks over controversies including claims he showed favouritism to a friend in a business deal.
Abe, who has seen his popularity plummet in recent weeks over controversies including claims he showed favouritism to a friend in a business deal, immediately apologised to the nation over the saga.
"I feel a keen responsibility," Tomomi Inada told a press conference where she announced her resignation. She called the results of an internal ministry probe which found that some officials had acted inappropriately "extremely regrettable".
She also said she would return a month's salary, despite the probe finding she had not been directly involved in the affair.
The chief of staff of Japan's Ground Self-Defense Force, the army, also resigned, as did the top ranking civil servant in the defence ministry.
Abe said that Fumio Kishida, the foreign minister, will also serve as interim defence chief until a replacement for Inada is named.
The shakeup in the top echelon of Japan's defence establishment comes as the country remains on alert over possible further North Korean missile launches.
Inada, a close confidante of Abe who shares his staunchly nationalist views, was appointed defence minister in August 2016, a time when she was touted as a possible future leader.
But her nearly year-long stint has been characterised by repeated controversy, including criticism of the ministry's handling of log reports filed by Japanese peacekeepers in South Sudan showing worsening security.
The troops, part of a United Nations mission, returned to Japan in May after five years.
Abe, who became prime minister for a second time in December 2012 with a vow to rejuvenate Japan's economy, has been hit by scandals and gaffes involving members of his ruling Liberal Democratic Party (LDP).
"I apologise from the bottom of my heart to the people for this situation in which a minister resigns," Abe told reporters after the resignation.
"All responsibilities lie with me, the prime minister, in the appointment of cabinet ministers," Abe said, adding he "must seriously accept the people's severe criticism".
The prime minister this week dodged opposition questioning in parliament over the claims of favouritism in a business deal, an accusation he has vigorously denied.
His government's public support ratings have fallen precipitously over the summer, with voters punishing the LDP in local Tokyo elections early this month.
The party suffered a drubbing in the vote for Tokyo's municipal assembly that media and analysts chalked up to a growing perception of "arrogance" on the part of his government.
It lost more than half of its seats, with the result seen as a bellwether for national political sentiment.
Abe will reportedly revamp his cabinet next week in a bid to reboot the government.
During her tenure, Inada delighted conservatives but drew domestic and international criticism in December when she prayed at a controversial war shrine in Tokyo -- the day after accompanying Abe on a symbolic visit of reconciliation to Pearl Harbor in the United States.
Yasukuni Shrine honours millions of mostly Japanese war dead, but is contentious for lionising senior military and political figures convicted of war crimes by an international tribunal.
Days before the Tokyo election Inada called on voters to support the LDP in the name of her ministry and the military.
The comments drew widespread condemnation, forcing Inada to retract her remarks, which were seen as a key reason for her party's unprecedentedly bad performance in the poll.