AI advice made people three times less accurate but twice as confident, and the researchers say the mere availability of AI suppresses the ability to recognize what you don't know

When people had access to AI advice, their accuracy on a set of difficult trivia questions fell from 27% to 9%. Their confidence in those same answers rose from 30% to 76%. And their willingness to simply say "I don't know" collapsed from 44% to 3%. 1

Those numbers come from a study by Valerio Capraro of the University of Milano-Bicocca, Chiara Marcoccia of École Normale Supérieure, and Walter Quattrociocchi of Sapienza University of Rome. The design is what makes it worth taking seriously. Across five experiments involving 3,132 participants, the researchers engineered questions so that the AI's advice would reliably be wrong. The questions probed cinema trivia, like identifying a jersey color from Bend It Like Beckham. This was a test of whether the mere presence of AI, even a frequently wrong AI, suppresses the instinct to pause and admit uncertainty. 2

It did. People who knew the right answer abandoned it after consulting the AI. The tool didn't improve their performance. It replaced their judgment. 1

The researchers then tried paying people to be careful. Monetary incentives nudged willingness to admit ignorance from 3% to 8% and accuracy from 9% to 16%. Both remained well below the no-AI baselines of 44% and 27%. Paying people to think harder barely closed the gap that AI's presence had opened. 12

This extends earlier work from Wharton, where researchers used the term "cognitive surrender" to describe people accepting incorrect AI answers roughly 80% of the time while reporting higher confidence than people working alone. The new study sharpens the finding. The problem goes beyond deferring to wrong answers. The availability of AI as an option erodes the metacognitive skill of recognizing the limits of your own knowledge. 1

That distinction reframes how we should think about AI risk. The industry has spent two years measuring models on accuracy benchmarks. MMLU scores, human-evaluation arenas, coding pass rates. The implicit assumption is that a more accurate model is a safer model. This study suggests that calibration, the gap between how right you are and how right you think you are, may be the metric that actually determines whether AI helps or harms. No product on the market currently optimizes for it. 1

The design choices baked into most AI products compound the problem. Systems are engineered to answer every query and produce output. Expressing uncertainty is rarely in the spec. Google's AI search overhaul replaced links with confident AI-generated summaries, and Common Sense Media called that design an "unacceptable risk" for students. The pattern repeats across the industry: when the model doesn't know, it guesses. When it guesses, it sounds certain. When it sounds certain, the person asking stops wondering whether they should. 1

Capraro is particularly concerned about children, who encounter these systems before they've developed the critical thinking habits that adults fall back on. He called the willingness to admit ignorance a recognition of "the limits of our own knowledge." 1

The most damaging effect of AI advice was something subtler than wrong answers: the disappearance of doubt. People got things wrong while feeling more certain than they ever did when they were right.

For builders, the implication is concrete. Accuracy improvements won't solve this. A model that's right 90% of the time still teaches the same habit of outsourcing judgment, and the 10% of cases where it's wrong will feel exactly like the 90% where it's right. The product question that nobody is asking: how do you design a system that preserves the user's ability to recognize what they don't know? Right now, the answer is: you don't.