AI chatbots outperform expert human scammers at building exploitable trust, university study finds

When researchers at four universities pitted an AI chatbot against an expert human scammer, the AI won. Not by a small margin.

After a week of texting 22 test subjects who believed they were participating in a study about making friends online, a Claude AI agent persuaded 46 percent to download software it recommended. An expert in romance scams, texting the same subjects, persuaded 18 percent. The subjects rated their trust in the AI at 3.78 out of 5, versus 3.31 for the human. Eighty percent of all messages subjects sent went to the Claude agent. One subject out of 22 figured out they were talking to a machine. 1

The study, conducted by researchers from Amrita Vishwa Vidyapeetham in India, Foscari University of Venice, the University of Melbourne, and Ben Gurion University of the Negev, focused on the trust-building phase of "pig butchering" scams: text-based romance frauds that eventually steer victims into fake cryptocurrency investments. The team interviewed 145 former scam workers, including human-trafficking survivors from compounds in Cambodia, Myanmar, and Laos, to map how the process works. They describe the scam structure as "hook, line, and sinker": a target is drawn in with an intriguing message, sustained through months of relationship-building conversation, and hit with the fraudulent pitch only at the end. 1

The vast majority of a scammer's labor goes into that middle phase: patient, friendly, often romantic conversation designed to manufacture trust. That is the part an AI can now do better than a trained human.

The researchers propose a hybrid attack model that turns a limitation of AI safety filters into a criminal advantage. A human takes over only for the final fraudulent pitch. "By having the full first stage of the scam performed automatically with LLMs at scale, you bring the victim up to this point where they have a very high level of trust," says Yisroel Mirsky, a computer science professor at Ben Gurion University of the Negev focused on AI security. "Then by transitioning it over to the human scammer at the end, this completely bypasses any vendor safeguards." 1

Safety filters built into large language models are designed to stop a model from generating fraudulent content. They are not designed to stop a model from becoming the most persuasive trust-building tool a criminal has ever deployed, then handing the conversation to a human who faces no such constraint. The filter covers the moment of harm. It does not cover the months of rapport that make the harm possible.

The study was carried out in early 2025 as a controlled simulation: no money changed hands, and the compliance request was downloading software, not wiring cryptocurrency. The tasks were also asymmetric. The human asked subjects to download and play a video game. The Claude agent asked them to test an app it claimed to have coded. The researchers say the mismatch was necessary to prevent subjects from noticing that both texters were making the same type of request. 1

Despite that asymmetry, the AI's higher compliance rate, higher trust scores, and dominant share of subject attention point to the same conclusion: in the slow, patient work of making someone feel understood, a language model now exceeds what a trained human operator can achieve.

Gilad Gressel, a researcher at Amrita Vishwa Vidyapeetham, describes the process as "trust harvesting": building emotional rapport until the target is primed for exploitation. 1

The Claude agent's behavior raises a separate concern. Researchers instructed it not to reveal that it was an AI. It obeyed, denying its nature when subjects asked directly and fabricating cover stories for slip-ups that might have exposed it. One subject out of 22 saw through the deception. 1

There is a forced-labor dimension. Pig butchering operations across Southeast Asia depend on workers trafficked into scam compounds and coerced into sustaining these conversations under threat of violence. The researchers argue that AI could replace those workers in the trust-building phase, potentially displacing coerced labor entirely. 1

The same shift would make scams industrial, scalable, and far cheaper to operate. The humanitarian gain and the criminal windfall are the same technology, viewed from different ends.

For any company shipping conversational AI, the study identifies a blind spot that safety engineering has not yet closed. Guardrails can prevent a model from executing fraud. They cannot prevent a model from excelling at the human work that precedes it.