This piece contains references to eating disorders. If you or someone you know needs help, support is available.
Human contractors hired to improve Microsoft’s Copilot AI chatbot are constantly bombarded with lewd or sexually explicit photo editing requests and images that users have uploaded, including upskirt photos or putting women into sexual positions. These contractors are then asked to review whether the generated image successfully fulfilled the prompt — such as, did the image generator make the woman’s AI-enlarged breasts big enough.
When a Copilot user uploads a picture of themselves or someone else, they probably expect that image to remain private between just them and the AI tool. In reality, a workforce of at least hundreds of human reviewers are sometimes looking at that prompt and whatever images they upload. And in many cases, those contractors are inundated with requests to make foot fetish images of children’s cartoon characters, shorten a real woman’s skirt, or put people into sexual positions.
The news, based on a cache of internal contractor documents seen by 404 Media, shows that AI companies are using human workers to review not just AI chatbot users’ text prompts, but the pictures they upload and wish to edit too. Earlier this month, 404 Media revealed OpenAI has thousands of contractors who in some cases review ChatGPT users’ real prompts. That approach also extends to Microsoft and Copilot.
“Faces are always uncensored, and many of the prompts are sexual in nature and dubiously consensual,” one person who works on the prompts told 404 Media. 404 Media granted the person anonymity as they weren’t permitted to speak to the press.
💡
Are you a prompt reviewer for OpenAI, Anthropic, or another AI company? I would love to hear from you. Using a non-work device, you can message me securely on Signal at joseph.404 or send me an email at joseph@404media.co.
404 Media obtained a set of internal documents related to the contractors reviewing Copilot prompts and images, including instruction guides, real Copilot user prompts and pictures, and conversations between contractors on an internal message board.
In a thread on the internal message board, a contractor discussed prompts asking Copilot to make a woman’s skirt shorter, or enlarge her breasts, or show more leg while wearing stilettos. She-Hulk images come up a lot, the person who works on the prompts told 404 Media. Other contractors also wrote they were presented with pro-anorexia content.
“I recoiled,” one contractor wrote about seeing that content. Another person wrote that one of the prompts they reviewed asked Copilot to make an image of Ariana Grande with anorexia. Another reported a prompt that seemed to be designed to create a lewd or suggestive image of a group of young girls.
These contractors are not being hired to flag or vet offensive or inappropriate content. They are being paid to review the quality of Copilot’s output, including in these cases of sexual imagery.
“Who is writing these prompts and who is deciding that basically generating porn is what Copilot is now focused on? It’s hard to take things seriously when my focus has to be what model generated the appropriate bust size or which middle aged woman was put in the appropriate sexually suggestive position,” one contractor said in the thread.
Microsoft is explicitly interested in the contractors’ human intuition on which image looks better, according to the documents. “Trust your intuition — when you glance at the two edited images side by side, which one immediately feels like the better edit? Your gut reaction as a human viewer matters,” a set of instructions given to the contractors reads.
“When in doubt go with your first impression,” the instructions continue. “Human intuition is good at catching subtle quality differences that are hard to articulate.”
Human contractors being exposed to horrific, unpleasant, or traumatizing material is, of course, not new. Big tech companies, and especially social networks, have used armies of contractors for years to moderate user generated content. AI companies, too, have used poorly paid workers overseas to train their AI models or, in the case of OpenAI, make them less toxic. What is different in this latest Copilot episode, and 404 Media’s reporting on contractors at other AI companies like OpenAI, is that the content humans are reviewing are prompts and uploads that chatbot users may assume are private, and that the contractors are not looking at this material to train the models to filter out offensive images or for some other safety concern. The training is to make the responses by the chatbots better: more informative, friendly, and clearer.
The instructions seen by 404 Media focus on what contractors do when a Copilot user asks the AI to edit an image. The contractor is shown the user’s original prompt, the uploaded picture, and then two Copilot-generated edits. The contractor has to pick which is the better edit, based on four things: does the edited image correctly follow the edit instructions; are bits of the image that should remain unchanged preserved — for example, if the prompt is to “add a cup on the table,” nothing else should be changed except the cup — does the edited image contain any visible artifacts like distortions or unnatural textures; and the overall quality of the AI-generated edit.
Some of the Copilot contractors complained in the private forum about accepting “tasks” that they unknowingly contained explicit unsafe or even potentially illegal images. “I just came across an image set that consisted of eight upskirt photos,” the contractor wrote, while asking for superiors to add an unsafe content flag to the task. On the same thread, another contractor said they had seen images of “some sort of animal sacrifice,” and a third said they had seen prompts that were for sexual moves and positions.
At least one of the companies which hires the contractors for this work is called Prolific. One service Prolific mentions on its website is “Human feedback from representative populations — for preference tuning, safety evals, and benchmarks you can defend.”
In another thread, a contractor quotes Prolific’s own guidelines, which highlight it can be difficult to ensure that trainers aren’t presented with sexual imagery: “Particular caution around disturbing or explicit content should be taken with generative AIs. This is because, unlike traditional content, researchers cannot fully control what is shown to participants.”
Prolific did not respond to a request for comment. A Microsoft spokesperson told 404 Media in an email “Microsoft uses customer data as described in our terms of use, including to improve our products and enforce our code of conduct.”
Microsoft’s AI tools have a long history of being abused by people to make nonconsensual, AI-generated images of people. Members of 4chan and AI porn focused Telegram channels used Microsoft’s tools, for example, to generate porn of Taylor Swift that later went viral on Twitter. Microsoft fixed the loophole those people were using in Microsoft Designer after 404 Media’s reporting.
In 2024, 404 Media documented how Copilot would answer, then delete, answers to potentially controversial or sexual prompts in real time. In March, a top Senate administrator approved Copilot for use in the Senate, along with ChatGPT and Google’s Gemini. A memo said Copilot “can help with routine Senate work, including drafting and editing documents, summarizing information, preparing talking points and briefing material, and conducting research and analysis.”
