Special Report

AI gets over-trusted until it errs once. Then it gets under-trusted for good.

Two opposite failures get called the same thing: not trusting AI correctly. Show someone a confident, fluent AI answer and they defer to it, often past the point their own judgment would say to stop. Show that same person one visible mistake from the same tool and they swing the other way, avoiding it even once it's genuinely more accurate than they are.

Both failures are well documented, in real research spanning coding, medicine, and everyday knowledge work. The usual fix, keep a human in the loop, doesn't reliably work either, and three separate studies below explain why a human technically watching isn't the same as a human actually catching anything.

The story in four parts

The over-trust

Developers using an AI coding assistant wrote less secure code, and felt more confident it was secure, than developers with no assistant at all.

The under-trust

People who watched an algorithm make one visible mistake stopped using it, even once it was still outperforming their own forecasts.

The real tension

The same people who abandon an algorithm after one error often prefer algorithmic advice over a human's, before any error happens at all.

The fix that isn't automatic

Adding a human review step doesn't reliably catch what the AI got wrong, unless that human actually has the time and reason to disagree.

The evidence at a glance

Confident, and wrong more often

47 developers using an AI coding assistant wrote less secure code on 4 of 5 tasks, yet felt more confident their code was secure.

One error was enough

After a single mistake, people trusted an algorithm less than a human who made the identical error, even when the algorithm still outperformed them on average.

No better than the humans it was meant to help

220 clinicians using AI-generated antidepressant recommendations were no more accurate than expert consensus alone.

Confidence, not correctness, killed the scrutiny

319 professionals described 936 real instances of AI use at work. Higher confidence in the output meant less critical thinking applied to it.

People defer to confident AI output, even when it's wrong

Automation bias isn't new, and it isn't specific to generative AI. In 1999, Skitka, Mosier and Burdick found people using an automated decision aid missed errors they would have caught working alone, deferring to the system's judgment over their own senses.

Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). “Does Automation Bias Decision-Making?” International Journal of Human-Computer Studies, 51(5), 991–1006

Generative AI reproduces the identical pattern, with real, current stakes attached to it. A 2023 Stanford study gave developers access to an AI coding assistant for security-related programming tasks, then measured what they actually shipped.

Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023). “Do Users Write More Insecure Code with AI Assistants?” Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS '23)
47 participants completed five security-related programming tasks across Python, JavaScript, and C, with or without access to an AI coding assistant.
ParaphrasedParticipants with access to the AI assistant wrote measurably less secure code than participants without it, on four of the five tasks.
ParaphrasedParticipants who used the AI assistant were more likely to believe their code was secure than participants who didn't, the opposite of what the actual vulnerability rate would justify.

The same pattern shows up with a patient in the room, not just a pull request. A 2021 study gave radiologists chest X-rays alongside diagnostic advice, secretly varying only whether that advice was labelled as coming from a person or an AI system.

Gaube, S., Suresh, H., Raue, M., et al. (2021). “Do as AI Say: Susceptibility in Deployment of Clinical Decision-Aids.” npj Digital Medicine, 4, Article 31
The diagnostic advice shown to radiologists and other physicians was always written by human experts. Only its labelled source, a person or an AI system, and its accuracy were varied.
ParaphrasedDiagnostic accuracy dropped significantly when participants received inaccurate advice, regardless of whether that advice was labelled as coming from a person or an AI system.
ParaphrasedRadiologists specifically rated advice as lower quality when it was labelled as AI-generated, even though the advice itself, accurate or not, was never actually written by an AI.

The opposite failure is just as real, and it doesn't fade with time

That's one failure, trusting a tool past the point its accuracy earns. The opposite failure runs the other way, and it's the one most trust research actually built its reputation on.

Berkeley Dietvorst, Joseph Simmons and Cade Massey ran a forecasting task where participants could use their own judgment or an algorithm's, then watched what happened once each side made a visible mistake.

Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). “Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err.” Journal of Experimental Psychology: General, 144(1), 114–126
Multiple incentivised forecasting experiments comparing participants' willingness to rely on their own judgment versus an algorithm's, after each had been shown making real errors.
ParaphrasedAfter seeing an algorithm make a forecasting error, participants were significantly less likely to use it than a human forecaster who had made the identical error.
ParaphrasedThis held even when the algorithm, despite its visible mistake, was still outperforming the participant's own forecasts on average.

That aversion doesn't just fade with reassurance. Dietvorst's own follow-up found a real, practical way to shrink it instead.

Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018). “Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them.” Management Science, 64(3), 1155–1170

Letting participants adjust the algorithm's forecast by even a small, fixed amount, without giving them any real control over its actual logic, restored their willingness to use it. The adjusted algorithm still outperformed participants relying on judgment alone.

Over-trust and under-trust are the same miscalibration, pointed in different directions

Put those two failures side by side and they look contradictory: over-trust in one study, under-trust in the next. A third study explains why they're actually the same finding.

Jennifer Logg, Julia Minson and Don Moore found that before anyone has watched an algorithm make a mistake, people often prefer its advice over a human's, rating identical advice as more trustworthy purely because it was labelled algorithmic.

Logg, J. M., Minson, J. A., & Moore, D. A. (2019). “Algorithm Appreciation: People Prefer Algorithmic to Human Judgment.” Organizational Behavior and Human Decision Processes, 151, 90–103

None of this describes a population that's stably too trusting or too skeptical of AI. It describes trust that swings on the most recent visible outcome, not on the tool's actual long-run accuracy. A single visible error erases confidence built over many quietly correct answers, and a confident-sounding answer with no visible error yet earns trust it hasn't actually demonstrated.

None of this gets fixed by simply adding a human to check the output

The standard response to both failures is the same instruction: keep a human in the loop. Three real studies found that instruction alone doesn't do what it promises.

Madeleine Elish studied cases where a human was nominally supervising an automated system, an aircraft autopilot, an early self-driving car, and found the human absorbed the blame when something went wrong without ever having had the real time or information to prevent it.

Elish, M. C. (2019). “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction.” Engaging Science, Technology, and Society, 5, 40–60

Ben Green and Yiling Chen studied judges using an algorithmic risk score in real bail and sentencing decisions, and found the human-plus-algorithm combination didn't behave the way the “human in the loop” framing assumed it would. Judges' actual use of the score varied in ways its own designers hadn't accounted for.

Green, B., & Chen, Y. (2019). “Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments.” Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* '19), 90–99

Maia Jacobs and colleagues gave 220 clinicians machine-learning treatment recommendations for depression and measured whether the recommendations actually improved decisions against expert consensus, a different question from whether clinicians used them at all.

Jacobs, M., Pradier, M. F., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021). “How Machine-Learning Recommendations Influence Clinician Treatment Selections: The Example of Antidepressant Selection.” Translational Psychiatry, 11, Article 108
A within-subject factorial experiment with 220 clinicians reviewing patient vignettes, comparing treatment selections made independently against selections made with access to machine-learning recommendations.
ParaphrasedAccess to machine-learning recommendations did not significantly improve clinicians' treatment selection accuracy, measured as agreement with expert psychopharmacologist consensus, compared to clinicians deciding independently.
ParaphrasedClinicians without the recommendations available performed comparably overall, meaning the recommendation step added to the workflow without a matching gain in decision quality.

A 2025 study explains why a human “in the loop” so often fails to catch anything

Three studies above show human oversight failing without saying why. The most current research on generative AI specifically names the mechanism.

Lee, H.-P., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025). “The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers.” Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25)
319 professionals across a range of occupations described 936 real, recent instances of using generative AI at work and reflected on the critical thinking involved in each.
ParaphrasedHigher confidence in the AI's output was associated with less critical thinking applied to that output.
ParaphrasedHigher confidence in one's own ability to do the task was associated with more critical thinking, at a higher perceived mental cost, than deferring to the AI.

The finding reframes what “human in the loop” actually needs to mean. It isn't a role on a process diagram. It's a level of scrutiny that has to survive exactly the confidence the AI's fluent output is designed to produce. A review step staffed by someone who trusts the tool is not the same safeguard as one staffed by someone who doesn't, even when the org chart looks identical.

Four questions worth asking about any AI-assisted decision

  1. Does the review step force independent judgment, or does it let someone approve confident-sounding output without re-deriving the answer themselves?Perry et al. found developers felt more confident in AI-assisted code precisely when it was less secure, and Gaube et al. found accuracy tracked the advice's real quality, not its labelled source.
  2. Has this tool ever visibly erred in front of the people using it, and did trust swing to the opposite extreme afterwards rather than settling near its real accuracy?Dietvorst et al. found one visible mistake was enough to erase trust in an algorithm that was still, on average, more accurate than the people avoiding it.
  3. Does the human “in the loop” actually have the time, information, and authority to disagree, or do they just carry the blame if something goes wrong?Elish's “moral crumple zone” and Green and Chen's judges both describe a human present in the process without the real capacity that presence implies.
  4. Does anyone check how confident the reviewer feels in the tool, or does a review step existing on paper count as enough?Lee et al. found confidence in the AI, not the presence of a review step, is what predicts how much scrutiny an output actually gets.