Cavendish, 32, suffered a broken shoulder blade in the incident and had to pull out of the Tour.
Sagan was kicked out after his elbow sent Cavendish crashing into metal safety barriers and down to the tarmac in Tuesday's sprint finish to the Tour's fourth stage.
Cavendish, 32, suffered a broken shoulder blade in the incident and had to pull out of the Tour, but he revealed in a video posted on Twitter that he has since been subjected to the ire of some social media users.
"Please know that it's sport and I have a family," he said.
"Everyone is entitled to their opinions. But vile and threatening comments on social media to myself and my family aren't deserved."
Cavendish said he was paying now for the "petulant" attitude he had as a younger rider.
"Unfortunately this will never leave me," he added.
"But I'd ask you all to respect that and please not send threatening or abusive language to myself and my family."
He also praised Sagan for both coming to see him after the incident on Tuesday and calling him later that evening, insisting that the two remain friends.