Why AI gets things wrong with such confidence (and how to manage it)

Most of us have had the same experience with an AI chatbot. You ask it something, it answers with complete confidence, and the answer is wrong. You point out the mistake, it apologises, explains why it happened, and then (sometimes in the very same conversation) makes the same mistake again.

It is easy to read that as the tool being faulty, something the next version will fix. I think that is the wrong way to look at it. And getting the frame right matters, because it changes what you actually do about it.

Hallucination is not a bug to be fixed

The thing we call ‘hallucination’ is not a glitch sitting on top of an otherwise reliable system. It is baked into how these models work.

A large language model (LLM) produces text by predicting, word by word, the most plausible continuation of what has come before. That is also why it can write a fluent email, summarise a report or draft a policy in seconds. But a system that can generate fluent, original text can, for the exact same reason, generate text that sounds right and is completely wrong. The ability to produce something new and plausible is the same one we praise as creativity when it lands, and call hallucination when it does not (we are not so different: the human imagination that lets us invent a story also lets us misremember one with complete conviction). There is no internal switch that lights up to tell it ‘I am making this up’. It is optimising for plausibility, not for truth, and most of the time plausible and true happen to line up (the problem is the times they do not).

So ‘how do we stop it hallucinating’ is, I think, the wrong question. You cannot stop it, any more than you can ask a calculator to stop being a calculator. What you can do is manage it. And managing it starts with noticing that we are usually talking about three different problems at once.

Three different things we call ‘hallucination’:

1. Invented facts

This is the real hallucination: the model states something that is simply not true. A statistic that does not exist, a law that was never passed, a quote nobody said, a reference to a report you will never find because it was never written. These are dangerous because they are delivered in the same confident tone as everything else.

2. Failures of reasoning and context

This is a different problem, and it gets mislabelled as hallucination all the time.

Take the car wash question that went viral earlier this year. You ask: ‘I want to wash my car. The car wash is 50 metres away. Should I walk or drive?’ most of LLMs answered, with full confidence, that you should walk (it is only 50 metres after all). As we (humans) know, you cannot wash a car in the car wash if you leave your car at home. Not one fact was invented by the LLM here. The model simply latched onto the obvious pattern, short distance means walk, and missed the one thing that actually mattered. That is a failure of reasoning, not a hallucination, and lumping the two together is exactly what muddies the water.

Here is an everyday version that might feel more familiar. Imagine you tell a chatbot, ‘I have an exam tomorrow and I am really nervous’. The next day, exam done, you message it again: ‘I am still feeling anxious’. A person would immediately fit that into a picture of your week. The exam has happened, so they would ask how it went. The model, in most cases, will not. Unless you tell it that a day has passed and the exam is over, it simply continues the most plausible reply to ‘I am anxious about my exam’, and carries on reassuring you about tomorrow.

The point is not that the model is stupid or is failing. It is that it doesn’t build and update a picture of you and your situation the way a person does. It reacts to the text in front of it. Even when something like today’s date is technically available to it, it does not make the human inference that the rest of us make without thinking.

3. False self-justification

This one is the most slippery. When a model gets something wrong and you challenge it (‘why did you say that?’), it gives you an explanation. The problem is that the explanation is not a true account of how it reached its answer. It is, once again, the most plausible-sounding text it can produce for the question ‘why did you say that’.

A model does not have reliable access to its own workings. So when it tells you why it made a mistake, treat that with the same caution as the mistake itself. It is not a confession. It is another guess that happens to sound convincing. (This is also why ‘telling it off’ so rarely works: you are not correcting a reasoning process, you are just adding more text for it to respond to plausibly.)

Managing it, not avoiding it

If you accept that getting things wrong is part of how these tools work, the practical question changes. The job stops being about eliminating errors, and becomes about working in a way that errors cannot cause harm. Here, a few habits go a long way:

• Treat every fact, figure, name, date, quote, and legal or funding detail as unverified until you have checked it against a reliable source. For a charity, a wrong deadline in a bid or a misquoted regulation is not a small thing.
• Give it the context a new colleague would need. Do not assume it knows your situation, today’s date, or what has changed since last time. Spell it out in the prompt.
• Ask for sources, then check that those sources actually exist and say what the model claims they say. Invented references are common.
• When you challenge it and it explains itself, stay sceptical. The explanation is not evidence.
• Match the tool to the stakes. Use it freely where being wrong is cheap, such as early drafts, brainstorming or reformatting. Slow right down where being wrong is costly: anything beneficiary-facing, safeguarding, compliance, funding applications or public statements.
• Keep a named person responsible for anything that leaves the organisation. The model can draft. It cannot be accountable.

None of this requires you to be technical. It requires a shift in posture: from trusting a confident answer to verifying it, and from expecting a tool that is always right to using one that is often useful and sometimes wrong, on purpose and with care.

The useful question for your charity is not ‘how do we stop the AI getting things wrong’. It is ‘where in our work can we afford for it to be wrong, and where can we not’. Answer that one honestly, and you are most of the way to using these tools well.

Want to explore AI in your charity in a more conscious and practical way?

The Charity AI Conference takes place online on Thursday 9 July – Book your tickets here.