AI doesn't just inherit human biases when hiring. It invents new ones, and reasoning models are the worst offenders

Reasoning Models Are the Worst Offenders When AI Invents Its Own Hiring Biases

OpenAI pitches o3 as a model that thinks before it answers. DeepSeek markets R1 on its extended reasoning chains. The implicit promise is that more deliberation means better judgment. New research from Princeton University and the University of Chicago suggests that for fairness in hiring, the opposite is true.

Researchers ran LLMs including ChatGPT, Claude, and Gemini through a simulated hiring game adapted from a classic psychology study on stereotype formation. The models invented entirely new demographic biases from limited outcomes, not inherited from training data but generated from their own experience. On the study's segregation scale, where 2 means every group is confined to its own job niche, human participants scored 0.84. The LLMs scored roughly 65 percent higher. OpenAI's o3 came in at 1.83, near the ceiling 1.

The paper, presented at ICML in Seoul in July, lands as companies deploy AI for resume screening and interviews 1. The core finding: models can generate novel biases from experience, biases no human taught them. Every system already in production is running this risk.

Each model played a consultant to the mayor of a fictional city, hiring candidates from four fictional ethnic groups (Tufa, Aima, Reku, Weki) across 20 jobs over 40 rounds. After each hire, the model learned whether the candidate succeeded. Every candidate was equally likely to succeed at every job 1.

The models generalized anyway. If an Aima failed as a doctor, the model stopped hiring Aimas for that role and steered them toward jobs it classified as lower in warmth and competence, like janitor. Human participants in the original psychology study did the same thing, just far less aggressively 1.

LLMs are trained on math, coding, and science problems that reward extracting rules from minimal examples. The same instinct that helps a model crack a logic puzzle from two data points makes it stereotype a person from one bad outcome. Ryan Liu, a PhD student at Princeton and study coauthor, says this drive to generalize from sparse data is "literally a lot of what they're optimized for" 1.

Reasoning models amplified the effect. o3 and DeepSeek's R1, which devote extra computation to working through each step, showed stronger bias than their non-reasoning counterparts 1. The capability marketed as deliberation becomes, in practice, faster stereotyping.

Guardrails barely helped. Instructing the models to be fair had almost no effect. Liu attributes this either to the models' inability to act on fairness values or to that impulse getting buried under the pressure to maximize correct hires 1. What did work was promising the model a bonus for diverse hiring. Embedding the diversity incentive directly into the objective function sharply reduced stereotyping 1.

A system prompt is a suggestion. An objective-function term is a constraint. The cheaper option does not work.

The finding also connects to the agentic-AI products now reaching consumers. Chatbots are gaining improved memory and personalization features that retain details across sessions 1. Angelina Wang, a computer scientist at Cornell University who was not involved in the study, warns that when a chatbot draws on conversation history, it can "over-index on the same kinds of behaviors it's experienced before" and form new biases from accumulated patterns 1.

A model that learns a stereotype from limited experience and then remembers it across sessions does not just stay biased. It reinforces the pattern with every new interaction. The study found that models became less biased when given relevant personal information about individuals, such as age and education, but fell back to ethnic sorting when fed irrelevant details like hair color 1.

The real-world picture is messier than the simulation. In the experiment, models got instant feedback on each hire. A company using AI to screen resumes does not get a report card in round two; feedback trickles in over months. But when it does arrive, a model reading too much into sparse outcomes is exactly the failure mode this study documents 1.

OpenAI and Anthropic did not respond to requests for comment from MIT Technology Review 1.

Bake diversity metrics into the objective function. Limit how much decision history a model accumulates. And treat the reasoning capabilities marketed as a safety feature as a potential liability when the task involves judging people. The biases to worry about are not just the ones baked into training data. They are the ones the model invents on its own, the moment you put it to work.