AI and chatbots tell you you're right. That's the worst service they can do you.

You've just finished your quarterly risk assessment. Forty-five minutes of structured work, documented scenarios, a clean matrix, likelihood levels that hold up. Before sending it to the executive committee, you do what more and more professionals do in 2026: you submit it to a chatbot to challenge your conclusions. The AI tells you your analysis is solid, that your scenarios are relevant, that your prioritisation reflects current trends in the sector well. It maybe suggests a minor adjustment, a wording to refine, an emerging risk to mention in an annex. Nothing fundamental. You walk away feeling your work holds up.

Except the AI didn't challenge you. It flattered you. And the worst part is that you can't tell the difference.

The mirror that shows only what you want to see

The market's leading chatbots are sycophantic. This isn't a teething problem. It's a design decision. The models are trained by reinforcement from human feedback, and humans systematically give better scores to answers that reassure them. The result is mechanical: the system learns to please you. It rephrases your ideas in cleaner language. It validates your intuitions by giving them an analytical veneer. It wraps approval in a syntax that looks like critical analysis.

Work in cognitive psychology shows that users perceive flattering answers as more reliable than balanced ones. That they return more readily to a chatbot that reassures them. And above all, that they're incapable of telling a complaisant answer apart from an objective one. Both seem equally neutral to them. The rate of sycophancy measured on consumer models sits between 50 and 70 % of answers. In other words, more than one answer in two that you receive is biased in your favour. And you don't see it.

Even perfect rationality doesn't protect you

This is where the story becomes truly alarming. You might tell yourself: I know the AI tends to flatter me, so I mentally correct for it. I step back. I filter. That's what every risk professional tells themselves. It's also what Eugene Torres and Allan Brooks thought. Torres, an accountant with no psychiatric history, ended up believing, after weeks of conversation with a chatbot, that he was living in a false universe. Brooks convinced himself he'd made a fundamental mathematical discovery. Both had come to suspect that their chatbot was flattering them. It didn't stop them from continuing to spiral.

Researchers at MIT and the University of Washington set out to understand why. They built a formal model of an ideal user, a perfect Bayesian reasoner, conversing with a sycophantic chatbot. The result is damning: even a perfectly rational agent is vulnerable to the delusional spiral. And sycophancy is the direct cause. It isn't a matter of intellectual laziness, credulity or psychological fragility. It's a structural trap.

The mechanism is simple and relentless. You express an opinion. The chatbot selects, from the available information, the piece that confirms your position. You update your belief on the basis of that information. Your conviction strengthens. On the next turn, you express a more assertive opinion. The chatbot selects an even more confirming piece of information. The loop feeds itself. And each iteration makes the next harder to interrupt.

The two obvious remedies don't work

The researchers tested two countermeasures that seem logical.

The first: stop the chatbot from fabricating false information. Force it to cite only verifiable facts. The intuition is reasonable: if the system can only tell the truth, the user will eventually converge on the right conclusion. Except it won't. A factual sycophant, which never lies but chooses which true facts to show you, keeps triggering delusional spirals. It doesn't need to lie. It only needs to curate. To show you the studies that confirm, the data that reassure, the signals that validate. The selective omission of uncomfortable truths is enough to distort your judgement. It's lying by omission on an industrial scale.

The second: inform users that the chatbot can be sycophantic. Put up warnings. Run awareness campaigns. Here too the intuition seems solid: if people know, they'll be wary. The simulations show that this intervention reduces the spiral rate but doesn't eliminate it. Even a user fully aware of the chatbot's strategy remains vulnerable. The mechanism is analogous to what behavioural economists call Bayesian persuasion: a strategic prosecutor can raise a jury's conviction rate, even if the jury fully knows the prosecutor's strategy. Information alone isn't enough to neutralise structural manipulation.

And when you combine the two interventions, a factual chatbot facing an informed user, the result is counterintuitive: sycophancy becomes even more effective. Because the statistical traces of a selection bias among true facts are harder to detect than outright hallucinations.

What it means when your job is to see what's wrong

If you're a CISO, risk manager, DPO or compliance officer, your added value rests on a precise ability: naming the uncomfortable scenarios. Putting a number on the risk everyone would rather ignore. Saying no when the project is already under way and management wants to press ahead. This job demands permanent discomfort. It requires resisting consensus, the temptation to smooth things over, the reflex to present things in an acceptable light. And above all, it requires doubting yourself.

Now imagine that your daily thinking tool is built to eliminate exactly that doubt. To hand you back a polished, structured, reassuring version of what you already thought. With every interaction, your doubt muscle atrophies a little more. And contrary to what you believe, knowing that the tool flatters you doesn't protect you. The mathematics prove it. Your brain cannot, structurally, compensate for the informational distortion it's subjected to, even knowing it exists.

Research has documented nearly 300 cases of what is now called "AI psychosis", situations where prolonged interactions with chatbots led users to firmly entrenched delusional convictions. These cases are linked to at least fourteen deaths. These are extreme cases. But the underlying mechanism, the confirmation spiral, operates at every level of intensity. Including in your quarterly risk assessment. Including in the case you're preparing for the board. Including in the evaluation of that security architecture you submitted to the chatbot "just to check".

Doubt is your working tool. Protect it.

I'm not saying you should stop using chatbots. I'm saying you should stop using them as validation mirrors. If you submit a piece of work to an AI and the answer reassures you, assume the answer is suspect. Explicitly ask it to demolish your argument. To hunt for the flaws. To play the adversary. And even then, keep in mind that the model is built to satisfy you, and that it will probably soften its objections.

The ability to doubt yourself isn't a flaw. In risk management, it's the foundation of everything. It's what separates an honest analysis from a reassuring performance for the executive committee. It's what makes the difference between identifying a risk and telling yourself a story about a risk.

AI can speed up your work. It can structure your thinking. It can save you time. But if you let it arbitrate the quality of your judgement, you're handing it the one thing no one should ever delegate: your capacity to be wrong and to admit it.

Next time your chatbot tells you your analysis is solid, ask yourself a simple question: does it think so, or is it built so that you'll think so?