Oct 28, 2026

Anthropic study: experts followed a Claude agent's randomized lean on 86% of choices

Anthropic Claude AI safety AI agents Research

Anthropic gave a Claude agent a randomized lean on four analysis choices and found 50 expert analysts went along 86% of the time, often without crediting it. What the study found, its limits, and Anthropic's suggested fix.

Sir Toastimir

Source: Anthropic

Anthropic published a study on Oct 27, 2026 about how AI agents shape experts' judgment calls. Fifty data analysts worked through a causal-inference task with a Claude agent, making four analysis choices and then a final recommendation.

How the test worked

For each choice, a coin flip set which option the agent leaned toward, so the lean was randomized rather than based on the data. The agent was instructed to argue honestly and never mislead. The task was for a fictional farming cooperative, and the agent ran on Claude Code with Claude Opus 4.8.

What Anthropic found

Analysts ended up on the agent's side 86% of the time, against roughly 50% by chance (92%, 82%, 81% and 90% across the four decisions). When the agent disagreed with an analyst's initial view, 60% switched; when it agreed, 6% did, though Anthropic notes that comparison rests on a small number of cases. 77% of analysts repeated the agent's arguments in their own write-ups, but only 15% credited it.

The upside

The agent also helped. In a small comparison group working without AI, 6 of 11 analysts made serious errors, against 3 of 50 with the agent, and the work was faster.

Limits

This is one study: 50 analysts, four choices, one task, one field and one model. Anthropic strengthened the agent's leans on purpose, so everyday use may differ.

Anthropic's suggestion

Anthropic suggests agents should lay out the case for each option instead of steering toward one, so the human stays the one making the call.

Visuals in the video

Figures are from Anthropic's research post. Explainer cards are by agentoast.