Zwayer has not refereed since December 9, five days after his performance in Dortmund's 3-2 home defeat by Bayern was blasted by Bellingham.
Zwayer added the abuse he has received has taken a toll on his wife.
"You try to pack your rucksack and say goodbye to your wife at the door, as you have done for 15 years. And then you see her burst into tears," he said.
"Not because she misses me when I'm gone, but because she worries about me and maybe about what's going on at home.
"This is a situation that is very difficult to bear, to be honest."
Dortmund's Bellingham was fined 40,000 euros ($45,000) by the German Football Association (DFB) for slamming Zwayer in a post-match interview.
The 18-year-old England midfielder was fuming when Zwayer awarded Bayern a penalty for a Mats Hummels handball, which resulted in the winning goal.
Shortly before, Zwayer had waved on play and refused Dortmund a penalty after Bayern defender Lucas Hernandez appeared to foul Marco Reus in the area.
"You can look at a lot of the decisions in the game. If you give a referee who has match-fixed before the biggest game in Germany, what do you expect?" Bellingham told Norwegian channel Viaplay.
He was referring to Zwayer's alleged role as a 23-year-old linesman in a 2005 scandal involving referee Robert Hoyzer, who was later jailed after confessing he took money from the Croatian mafia to influence match results.
Zwayer was suspended for six months for keeping silent though he knew what the referee was doing and for allegedly accepting 300 euros ($340), which he denies.
"I have never received any money from Robert for any involvement in any manipulation of a game," Zwayer said.