Special Report

AI gets over-trusted until it errs once. Then it gets under-trusted for good.

Call both failures by the same name if you like: trusting AI at the wrong level. Hand someone a fluent, confident answer from a model and they often hand judgment over to it, past where their own reading of the problem would stop them. Show that person one clear mistake from the same tool and the pendulum swings. They walk away from it even after the tool is back to beating their own accuracy.

Coding labs, clinics, and ordinary knowledge work all show both patterns in published work. Putting a human on the workflow chart does not reliably solve either pattern. Three studies below separate “someone is assigned to watch” from “someone actually catches the error.”

The story in four parts

Where over-trust shows up

With an AI coding assistant in the mix, developers shipped less secure code and rated that code as more secure than peers who had no assistant.

Where under-trust locks in

Watch an algorithm miss once in public and people drop it, even while its average forecasts still beat their own.

Why the two look like a paradox

Before any error lands, those same people often pick algorithmic advice over a human’s. The swing starts only after a failure is visible.

Why a review step can still miss

A human sign-off catches little unless that person has time, information, and a real reason to push back on the model.

The evidence at a glance

More confidence, worse security

47 developers with an AI coding assistant produced less secure code on 4 of 5 tasks, and still felt surer their work was safe.

A single miss flipped trust

After one mistake, people trusted an algorithm less than a human who made that same mistake, even when the algorithm’s average still beat theirs.

Recommendations that did not lift accuracy

220 clinicians given AI antidepressant suggestions matched expert consensus no better than clinicians deciding without those suggestions.

Confidence crowded out scrutiny

319 professionals described 936 real instances of AI use at work. The more they trusted the output, the less critical thinking they applied to it.

Confident AI answers get accepted past the point they earn

Automation bias predates chatbots by decades. Skitka, Mosier and Burdick showed in 1999 that people with an automated decision aid missed errors they spotted when working alone. They deferred to the system instead of their own senses.

Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). “Does Automation Bias Decision-Making?” International Journal of Human-Computer Studies, 51(5), 991–1006

Generative tools recreate that pattern under stakes that are current and concrete. In 2023, Stanford researchers gave developers an AI coding assistant for security-related tasks and scored what those developers actually produced.

Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023). “Do Users Write More Insecure Code with AI Assistants?” Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS '23)
Forty-seven participants each completed five security-related programming tasks in Python, JavaScript, and C, randomised to work with or without an AI coding assistant.
ParaphrasedOn four of five tasks, code from the AI-assisted group was measurably less secure than code from the group without the assistant.
ParaphrasedThose who used the assistant were also more likely to rate their own code as secure, which ran against the vulnerability rates the researchers measured.

Move the setting from a pull request to a reading room and the pattern holds. In 2021, radiologists saw chest X-rays paired with diagnostic advice. Researchers varied only the label on that advice: human expert or AI, while the underlying text stayed expert-written.

Gaube, S., Suresh, H., Raue, M., et al. (2021). “Do as AI Say: Susceptibility in Deployment of Clinical Decision-Aids.” npj Digital Medicine, 4, Article 31
Human experts wrote every piece of diagnostic advice shown to radiologists and other physicians. The study varied only the labelled source (person or AI) and whether the advice was accurate.
ParaphrasedWhen the advice was inaccurate, diagnostic accuracy fell whether the label said “person” or “AI.”
ParaphrasedRadiologists still rated the same advice as lower quality under an AI label, even though no AI had written any of it.

After one visible miss, people abandon tools that still beat them

Over-trust is the first failure mode above. Under-trust is the mirror image, and it is the finding most of the “algorithm aversion” literature hangs on.

Berkeley Dietvorst, Joseph Simmons and Cade Massey put people in forecasting tasks where they could lean on their own judgment or an algorithm’s. The hinge was what happened after each side made a mistake the participant could see.

Dietvorst, B. J., Simmons, J. P., & Massey, C. (2015). “Algorithm Aversion: People Erroneously Avoid Algorithms After Seeing Them Err.” Journal of Experimental Psychology: General, 144(1), 114–126
A series of incentivised forecasting experiments measuring how willing people were to use their own judgment versus an algorithm after each had been shown making real errors.
ParaphrasedOnce participants saw an algorithm miss a forecast, they became far less willing to use it than a human forecaster who had made that identical miss.
ParaphrasedThe gap remained even when the algorithm’s average forecasts still beat the participant’s own.

Reassurance alone does little to unwind that aversion. Dietvorst’s follow-up found a concrete lever that shrinks it.

Dietvorst, B. J., Simmons, J. P., & Massey, C. (2018). “Overcoming Algorithm Aversion: People Will Use Imperfect Algorithms If They Can (Even Slightly) Modify Them.” Management Science, 64(3), 1155–1170

Give people a small, fixed way to nudge the algorithm’s forecast, without any real control over how the model works, and willingness to use it returns. Even after that nudge, the algorithm still beat participants who relied on judgment alone.

The same people can prefer algorithms before an error and reject them after

Stack the two literatures and they seem to quarrel: deference in one set of papers, abandonment in another. A third line of work shows why both can be true in the same population.

Jennifer Logg, Julia Minson and Don Moore found that, before anyone has watched an algorithm fail, people often prefer its advice to a human’s. They treat identical advice as more trustworthy simply because the label says algorithmic.

Logg, J. M., Minson, J. A., & Moore, D. A. (2019). “Algorithm Appreciation: People Prefer Algorithmic to Human Judgment.” Organizational Behavior and Human Decision Processes, 151, 90–103

So this is not a public that sits permanently too high or too low on AI trust. Trust tracks the last visible outcome far more than the tool’s long-run hit rate. One public miss can wipe confidence that many unnoticed correct answers built up. A fluent answer with no miss yet on the board can earn trust the tool has not earned on accuracy.

Naming a human reviewer does not, by itself, catch the model’s errors

Teams usually answer both failure modes with the same instruction: keep a person in the loop. Three studies show that instruction, standing alone, fails to deliver what the org chart implies.

Madeleine Elish looked at cases where a human was listed as supervising automation: aircraft autopilot, early self-driving cars. She found blame landed on that human when things went wrong, even when they never had the time or information to stop it.

Elish, M. C. (2019). “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction.” Engaging Science, Technology, and Society, 5, 40–60

Ben Green and Yiling Chen studied judges using an algorithmic risk score in real bail and sentencing work. The human-plus-algorithm setup did not behave the way a tidy “human in the loop” slide assumes. How judges actually used the score drifted in ways the designers had not designed for.

Green, B., & Chen, Y. (2019). “Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments.” Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* '19), 90–99

Maia Jacobs and colleagues gave 220 clinicians machine-learning treatment recommendations for depression. The outcome they measured was whether those recommendations moved decisions closer to expert consensus, a stricter bar than whether clinicians clicked them.

Jacobs, M., Pradier, M. F., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021). “How Machine-Learning Recommendations Influence Clinician Treatment Selections: The Example of Antidepressant Selection.” Translational Psychiatry, 11, Article 108
Within-subject factorial design: 220 clinicians reviewed patient vignettes and chose treatments alone, then again with machine-learning recommendations available.
ParaphrasedHaving the recommendations available did not significantly raise agreement with expert psychopharmacologist consensus, relative to deciding without them.
ParaphrasedClinicians working without the recommendations performed about as well overall, so the extra step sat in the workflow without a matching lift in decision quality.

Why fluent output so often slips past the person assigned to check it

The three oversight studies show the review step failing. They leave the mechanism unnamed. A 2025 survey of generative-AI use at work puts a name on it.

Lee, H.-P., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025). “The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers.” Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25)
319 professionals across occupations described 936 recent, real instances of using generative AI at work and rated the critical thinking each instance involved.
ParaphrasedWhen people felt more confident in the AI’s output, they reported applying less critical thinking to that output.
ParaphrasedWhen people felt more confident in their own ability on the task, they reported more critical thinking, at a higher perceived mental cost, than when they deferred to the AI.

“Human in the loop” has to mean more than a name on a RACI chart. It has to mean scrutiny that still fires when the model’s fluent answer invites trust. A reviewer who already trusts the tool is a different safeguard from one who does not, even when both titles look the same on paper.

Four questions for any decision that leans on AI

  1. Does the review step require someone to rebuild the answer, or can they rubber-stamp fluent output?Perry et al. found developers felt more confident in AI-assisted code exactly when it was less secure. Gaube et al. found accuracy tracked whether the advice was actually right, regardless of the human-or-AI label.
  2. Has the tool failed in front of its users, and did trust collapse past the tool’s real accuracy instead of settling near it?Dietvorst et al. found one visible miss was enough for people to abandon an algorithm that still beat them on average.
  3. Does the person “in the loop” have time, information, and authority to disagree, or only the blame if something breaks?Elish’s “moral crumple zone” and Green and Chen’s judges both describe a human present in the process without the capacity that presence is supposed to buy.
  4. Does anyone measure how confident the reviewer feels in the tool, or is the existence of a review step treated as enough?Lee et al. found confidence in the AI, rather than the mere presence of a review step, predicted how much scrutiny an output actually received.