The 27-year-old Slovak had been hoping for a reprieve after his Bora team appealed against his sanction.
The 27-year-old Slovak had been hoping for a reprieve after his Bora team appealed against his sanction for flicking an elbow towards Mark Cavendish who crashed into the barriers during the sprint finish to Tuesday's fourth stage.
But on Wednesday morning, Sagan admitted defeat in his bid to ride on.
"I can only accept the decision of the jury, but I disagree. I don't think I've done anything wrong in the sprint," said Sagan.
Cavendish was taken to hospital after the crash where he was diagnosed with a broken shoulder blade, forcing him out of the Tour.
Sagan had appeared to elbow him into the barriers during a muscular sprint finish, with the race commission later deciding to expel the Bora rider for having "endangered some of his colleagues seriously".
Bora reacted late on Tuesday saying they had appealed the decision and asked that Sagan be reinstated, although that scenario was always unlikely.
"In the sprint I didn't know that Mark Cavendish was behind me," protested Sagan. "Mark was coming really fast from the back and I just didn't have time to react and to go left."
"He came into me and he went into the fence."
For his part, Cavendish said he found Sagan's elbow inexplicable.
"I was a little bit confused with the elbow, that's something I'd like to speak to him about," said the 32-year-old winner of 30 Tour stages.
"I'm obviously massively disappointed to get this news about the fracture," said Cavendish.
As for Wednesday's 160.5km fifth stage from Vittel to La Planche des Belles Filles, it has reigning champion Chris Froome licking his lips in anticipation.
It was there in 2012 that he won his first ever stage on the Tour, when he went on to finish second overall to British compatriot and Sky team-mate Bradley Wiggins.
"It was a really memorable victory for me," said Froome, a three-time Tor winner since.
"I'm certainly looking forward to going back there."
According to Australia's 2011 Tour winner Cadel Evans, this stage will give a big clue as to who will win the Grand Tour when it reaches Paris on July 23.
"If you look back at all the results, certainly of the Tours I rode, the result of the first mountain finish is often very close to the GC (overall standings) in Paris," he said.
Now retired, Evans is the only Australian and only rider for the BMC team to have won the Tour.
But he has in Richie Porte a more than capable successor for both those roles.
And current race leader Geraint Thomas, Froome's Sky team-mate, believes Porte will be looking to attack at the end of Wednesday's stage, which starts off quite flat but climbs to more than 1,000 metres by the finish.
"Richie is going to try for sure. He is in great form and the climb is perfect for him," said Thomas.
Despite wearing the yellow jersey, the 31-year-old Welshman insisted he has no personal designs in this Tour other than helping his team leader.
"I'm fully committed to Froomey and winning the Tour with him," he said.
The stage ends with a steep first category climb, 5.9km long at an average 8.5 percent gradient, although the very end rises to 20 percent.
"It's not a long climb," said Froome.
"We shouldn't see big time differences, but definitely it's tough enough to show exactly where all the rivals are at."