Mladenovic, of France, fell to Karatantcheva 5-7 6-4 7-5 at the WTA International hard-court event in Quebec City, Canada.
Karatantcheva has spent over five hours on court on her way to the quarter-finals, after winning in two hours 33 minutes against Mladenovic.
Ranked 195th, Karatantcheva came from a set and 3-1 down to triumph over 82nd-ranked Mladenovic - although she almost blew it after wasting a match point in the 10th game of the third set.
Mladenovic managed to break back to level at 5-5, but folded in the very next game before Karatantcheva held to love to seal the upset.
Fourth seed Shelby Rogers survived an almighty challenge from American qualifier Asia Muhammad, eventually progressing to the quarter-finals with a 6-7 (5-7) 7-6 (7-2) 7-6 (7-5) victory that took two hours 54 minutes.
Rogers, of United States, will take on Germany's Tatjana Maria in the last eight, after Maria eclipsed American Madison Brengle 4-6 6-4 6-0.
Fifth seed Julia Goerges joined countrywoman Maria in the quarters, with a 6-2 6-2 drubbing of American Melanie Oudin.
Goerges' next opponent will be either Croatian second seed Ajla Tomljanovic or Czech Andrea Hlavackova.
Hlavackova progressed past the first round on Wednesday, beating Poland's Paula Kania 6-3 6-0.