Instagram's new TV service recommended videos of potential child abuse

Instagram's new TV service recommended a crop of graphic and disturbing videos, including what appeared to be child exploitation and genital mutilation.

That's the finding of a Business Insider investigation into IGTV, which launched in June as Instagram attempts to muscle in on rivals like YouTube and Snapchat.

BI spent nearly three weeks monitoring the Facebook-owned video service, during which time IGTV's algorithm recommended questionable content, including sexually suggestive footage of young girls and an explicit video of a mutilated penis.

Two of the videos discovered by BI were reported to the police by the National Society for the Prevention of Cruelty to Children (NSPCC), a British children's charity, and were eventually removed by Instagram five days after BI reported them through the app's official reporting channel.

Instagram apologised to users who saw the videos and said it wants to make IGTV a safe space for young people.

The findings come at a time when Facebook is under extraordinary scrutiny over inappropriate content on its platforms. Facebook and Instagram share a community operations team, and Mark Zuckerberg's company has hired an army of 7,500 moderators and is using AI to snuff out posts that break its guidelines.

Earlier this year, former Facebook moderator Sarah Katz told Business Insider that she had to review 8,000 posts a day and it made her numb to child abuse. But despite the intense oversight, and resources Facebook is ploughing into policing its platforms, disturbing content appears to be slipping the net — and, as in the case of IGTV, even being suggested to users.

IGTVs content recommendation machine

Instagram launched IGTV in June, and was viewed by many as Facebook moving in on YouTube's territory. It allows users to set up their own channels and upload video of up to an hour in length. Anyone with an Instagram account can make a channel, and users swipe through them much like flicking through channels on a television.

IGTV recommends content in three ways: A "For You" tab, which plays videos as soon as you open IGTV; a "Popular" section; and a "Following" menu, which offers videos from people you follow.

Instagram did not answer Business Insider's questions on how IGTV's algorithm recommends certain videos and why videos were suggested that appeared to be child exploitation. But it appears that the For You section recommends things users will like, possibly based on past activity. The Popular tab seems to gather trending content from across IGTV.

Users can scroll through the recommended videos by swiping left, or IGTV will automatically play the next video. It is clearly designed to encourage scrolling and continued viewing, in much the same way that the YouTube algorithm recommends content through its Up Next bar.

Disturbing videos of young girls

Business Insider monitored the For You and Popular tabs for almost three weeks to establish what kinds of content IGTV's algorithm was serving up for users.

We did so in two ways: Firstly through the account of this author and other BI journalists, and secondly an anonymous login set up as a child's account. This second account had no activity history on Instagram and a user age set to 13, which is the earliest people can officially sign up on the app.

Within days of monitoring IGTV through Business Insider accounts, a video appeared in the For You section, titled "Hot Girl Follow Me." It showed a young girl, we speculate to have been 11 or 12, in a bathroom. She glanced around her before going to take her top off. Just as she's about to remove her clothing, the video ends.

The video, uploaded by a user Business Insider is not naming for legal reasons, also appeared under the Popular tab on IGTV. It was also one of the first videos recommended under the For You section to the child account set up by BI, which had no prior history of activity on Instagram.

The same user that uploaded the "Hot Girl Follow Me" video posted another video, titled "Patli Kamar follow me guys plzz," which was also recommended to our child Instagram account under the For You section. It featured another clearly underage girl, exposing her belly and pouting for the camera.

The same two videos were separately uploaded by a different user, who again Business Insider has chosen not to identify. The video named "Hot Girl Follow Me" was called "Follow me guys" by this second user, and was also circulating on IGTV's suggested posts.

Comments on the videos show they were being recommended to other IGTV users. They were also being interpreted by other users as sexually suggestive.

Some condemned the videos and questioned why they had been suggested. "BRO SHE'S LIKE FUCKING 10 WHY THE FUCK IS THIS IN MY INSTAGRAM RECOMMENDED," said one user, commenting on the "Hot Girl Follow Me" video.

Others were more predatory in tone. "Superb," one user commented on the "Patli Kamar follow me guys plzz" video. "Sexy grl," another added.

The NSPCC, which is frequently involved in law enforcement activities around child abuse, reviewed the videos and reported them to the police. It was concerned that they could constitute illegal indecent images under UK law because they appeared to feature footage of erotic posing.

"This is yet another example of Instagram falling short by failing to remove content that breaches its own guidelines," an NSPCC spokeswoman said.

Business Insider reported the videos through Instagram's official reporting function. Because there was no obvious criteria for alerting the company to potential child exploitation, they were logged as "nudity or pornography."

The videos remained online for five days. It was only after Business Insider contacted Instagram's press office that the content was removed. By this time, the two videos — and other versions uploaded by the second user — had racked up more than 1 million views.

Instagram left the accounts that posted the videos active, however. Business Insider asked why the accounts were left up, as Instagram has a "zero tolerance policy" on child abuse. Instagram said the policy applies to the content itself and not to the account that uploads it.

"Instagram’s zero tolerance policy towards child abuse content is the right one, and it must make sure its policy is enforced in practice," an NSPCC spokeswoman said. "Where Instagram has removed child abuse content from an account, we would expect that account to be reviewed by a moderator to establish whether the account should also be suspended."

As of this week, the two accounts remain active. They continue to post sexually suggestive content, but not of the same nature as the "Hot Girl Follow Me" and "Patli Kamar follow me guys plzz" videos.

A graphic video of genital mutilation

Potential child exploitation is not the only questionable content being recommended by IGTV's algorithm.

One of the first videos recommended to Business Insider's anonymous account, registered to a user aged 13, was graphic footage of a penis undergoing an operation involving a motorized saw.

The penis appeared to have a metal lug nut affixed around its middle, above which it was extremely swollen and dark red in colour. The bolt was being removed with a round electric saw by what appeared to be a medic.

It was quickly wiped from IGTV after being reported by Business Insider as nudity, although the account that uploaded it remained live.

Another recommended video showed a baby lying on the floor, wailing inconsolably, with a monkey standing over and touching it. Adults were stood around in a circle, shouting and filming the scene on their phones while the monkey occasionally lashed out at them.

Instagram found that the video was not in breach of guidelines, but discovered the account that uploaded it was linked to a different account that had previously been taken down for breaching community guidelines. For this reason, Instagram took the account down.

"Very disturbing"

There was a multitude of other questionable recommendations being pushed by IGTV's algorithm. Examples included a video in which a group of men deceived a sex worker into thinking she was going to be arrested, a video of a woman pulling something long and bloody out of her nose, and various sexually suggestive scenes.

Business Insider presented its investigation to MP Damian Collins, the British lawmaker who is leading an inquiry into fake news and Facebook's data breach involving Cambridge Analytica. He described the findings as "very disturbing," and said big tech companies need to sink more investment into enforcing their own rules.

"It's a question of the responsibility of the companies to monitor the content that's on their platforms. A lot of the problematic content is already in breach of the community guidelines of these services, but what it shows is that there's not effective enforcement," he said.

An Instagram spokeswoman said: "We care deeply about keeping all of Instagram — including IGTV — a safe place for young people to get closer to the people and interests they care about.

"We have Community Guidelines in place to protect everyone using Instagram and have zero tolerance for anyone sharing explicit images or images of child abuse. We have removed the videos reported to us and apologise to anyone who may have seen them.

"We take measures to proactively monitor potential violations of our Community Guidelines and just like on the rest of Instagram, we encourage our community to report content that concerns them. We have a trained team of reviewers who work 24/7 to remove anything which violates our terms."

"Its not so different from where YouTube was 10 years ago"

It's no secret that social media has a problem with disturbing and illegal material, and tech giants like Facebook have come under fire recently for failing to effectively moderate content at scale. For IGTV, however, the problem isn't just that this material exists, but rather it's being actively suggested by the algorithm.

Mike Henry, CEO of video analytics firm OpenSlate, which works with Facebook, said IGTV is still a young service. "While Instagram is relatively mature, IGTV is a brand new social video platform and will need time to develop its policies and technology. It’s not so different from where YouTube was 10 years ago," he said.

YouTube's child safety policy is broader than Instagram's, for example. Instagram's report function is limited to child nudity, while YouTube's endangerment policy bans "sexualization of minors," which allows scope for reporting images users suspect might be child exploitation.

Henry also said Instagram will have to figure out how to better filter its new platform if it hopes to monetize it, especially considering IGTV was touted as a space for influencers.

"Influencers make great video producers with compelling economics and, at scale, a viable canvas for video ad dollars. With the right policies and infrastructure, IGTV has the potential to become a major player," he said.

For Collins, IGTV's early missteps are evidence that governments need to do more to regulate tech firms.

"These companies are ad services, they make money out of understanding every single thing you could ever want to know about your users so you can target them with advertising. That same technology should surely very easily be able to root out harmful content as well," he said.

"They don't do it because there’s not been a commercial incentive for them to do it, so they’ve just not bothered. But what we have to do through regulation is create that incentive, to say 'you’ve actually got an obligation to do it and if you don’t do it then there will be costs for you for not complying, so you need to invest in doing this now.'"