Bosz, 55, who steered Ajax to the Europa League final in 2017, was dismissed by current league leaders Borussia Dortmund last December after just five months.
Leverkusen have given Bosz a contract until June 2020 and he will take charge after the Christmas holidays. The German league resumes on January 18 following the winter break.
After beating Hertha Berlin 3-1 on Saturday, Leverkusen are ninth in the table and seven points off the Champions League places.
Leverkusen's sports director Rudi Voeller justified the decision to fire Herrlich after 17 months because of the "stagnation of the team's development".
Under the 47-year-old Leverkusen finished fifth in Germany's top flight last season, missing a Champions League place on goal difference.
"After a series of unsatisfactory performances in the first half of the season, we are in a situation that calls for a change of coach, from our point of view," said Voeller.
Bosz had been Voeller's first choice to coach Leverkusen in July 2017, but when the Dutchman opted for Dortmund, Herrlich got the job.
"Heiko gave our team an important boost last year after a very difficult season and got us back in international competition," said Voeller.
Leverkusen finished top of their Europa League group and will face Russian side Krasnodar in the last 32.