Humans are reviewing Copilot users’ bizarre and abusive image-editing requests

A reminder for anyone who thinks their AI chats are private: A human may see what they upload. A report from 404 Media this week describes contractors reviewing Copilot users’ photos, editing requests, and the images Copilot produces. Some of what they see is disturbing. The third-party companies hired to conduct these reviews present their workers with an original photo and a user’s request for an edit. They are also shown two potential AI edits and asked to judge each on technical merit. Because this is the internet, this goes well beyond requests to put your cat in a Halloween costume. Many of the requests are sexual in nature and some involve children, the report said. The faces in the original photos are not censored, meaning reviewers can see who the subjects are. Editing requests and results that reviewers are asked to evaluate include requests to shorten skirts, enlarge body parts, and place people in sexual positions. In many cases, the edits appear nonconsensual. For example, one contractor mentioned being shown multiple sets of upskirt photos. Another said they were asked to rate foot-fetish images of children’s cartoon characters. Users had asked Copilot to create pictures of Ariana Grande suffering from anorexia. Images depicting animal sacrifice crossed one reviewer’s desk. Interviewees told 404 Media that contractors are paid to rate the output quality of an image, but not to flag inappropriate content. They check things like whether instructions were followed properly, whether parts of the image were preserved or distorted, and the overall look of the image. They don’t make legal or safety judgments about the content. Microsoft uses customer data from some accounts to help improve its products. But when 404 Media asked, it didn’t say whether users can exclude uploads from these rating tasks. In its Privacy FAQ for the older consumer version of Copilot, Microsoft says it keeps conversations for 18 months by default, although users can delete them at any time. It also says it may use data “including the images or files you upload” for AI training. Eligible users can opt out of training, but the FAQ says an opt-out from certain human reviews is not available. It does not say whether that includes the image-rating tasks described by 404 Media. Microsoft says the u p dated consumer Copilot app, available since August 18, does not use prompts, responses, or uploaded file contents to train foundation models. That still leaves the question 404 Media put to Microsoft: Can users keep their uploads out of human image reviews? Copilot’s previous image issues Microsoft insiders have warned about Copilot’s propensity to produce questionable images in the past. In early 2024, machine learning engineer Shane Jones went public with concerns over the ability to bypass guardrails in the software’s text-to-image tool DALL-E 3. He had previously asked the company to remove Copilot Designer, which used the tool, from public use for three months but said Microsoft had refused. Its feedback team received over 1,000 product feedback messages daily and was only dealing with the most serious of them, he told CNBC at the time. At the same time that this was happening, Microsoft fixed a loophole in Copilot Designer that enabled someone to make nonconsensual intimate deepfakes of Taylor Swift . Microsoft isn’t the only company that hires contractors to review AI chats. OpenAI does the same as part of its Project Lily initiative. If you want more information about how AI assistants use your conversation data to improve themselves and what you can do to stop them, read Malwarebytes’ guide to AI training on user content. From reporting threats to removing them. Cybersecurity risks should never spread beyond a headline. Keep threats off your devices by downloading Malwarebytes today . About the author Danny Bradbury has been a journalist specialising in technology since 1989 and a freelance writer since 1994. He covers a broad variety of technology issues for audiences ranging from consumers through to software developers and CIOs. He also ghostwrites articles for many C-suite business executives in the technology sector. He hails from the UK but now lives in Western Canada.