What AI Gets Wrong

AI assistants are fluent, fast and often right — which makes their mistakes easy to miss. Meet the seven ways they most often go wrong, why each happens, and the habits that keep you safe.

In 2023 a New York lawyer filed a court brief citing earlier cases to support his client, who was suing an airline. The airline’s lawyers, and then the judge, went looking for those cases. At least six could not be found, because they did not exist. The lawyer had asked ChatGPT for research, and it had produced convincing case names, quotations and citations out of thin air. He had even asked it whether the cases were real — and it said yes.

Fluent is not the same as true

That case, Mata v. Avianca, became a worldwide warning. In June 2023 Judge P. Kevin Castel of the US federal court in Manhattan fined the two lawyers involved and their firm a total of US$5,000, finding they had acted in bad faith by standing by the fake citations after they were questioned. The lesson was not “AI is useless” — it was that an AI assistant’s confidence tells you nothing about whether it is right.

Remember how a large language model works, from What Is a Large Language Model?: it writes by predicting, piece by piece, what text plausibly comes next, based on patterns in its training data. It has no built-in fact-checker. Most of the time, the plausible answer and the true answer are the same. When they are not, you get a mistake that sounds exactly like a correct answer.

Seven ways AI goes wrong

Open each card to see the failure, an example, and why it happens in plain words.

1HallucinationsConfident inventions

A hallucination is a false statement presented as fact: a made-up court case, a book that was never written, a quote nobody said, a wrong phone number for a real company. They are most common for specific details — names, dates, numbers, references, niche topics — where the model has seen little reliable text.

Why: the model is built to produce a fluent answer, and training has historically rewarded a confident guess more than “I don’t know”, much like a student guessing on a multiple-choice exam. Researchers at OpenAI and Georgia Tech made exactly this argument in a 2025 paper.

2Knowledge cutoffsFrozen in time

A model learns from text collected up to a certain date — its knowledge cutoff. Ask about anything after that (a new law, this year’s prices, last week’s election) and it may give outdated information, or confidently guess.

Why: training a large model takes months, so its knowledge is a snapshot. Many assistants can now search the web to fill the gap As of 2026, but not every answer uses search, and search results can be wrong too. Check whether the answer cites fresh sources.

3BiasUnfair patterns, learnt from us

Bias means a system treats some groups unfairly. In the 2018 Gender Shades study, Joy Buolamwini and Timnit Gebru tested commercial face-analysis systems and found error rates for gender classification of up to about 35% for darker-skinned women, against under 1% for lighter-skinned men. The same year Reuters reported that Amazon had abandoned an experimental recruiting tool after finding it penalised CVs containing the word “women’s”, as in “women’s chess club captain”.

Why: models learn from past data, and past data reflects past unfairness and gaps. Amazon’s tool learnt from a decade of applications that came mostly from men. Image systems trained mostly on lighter faces perform worse on darker ones.

4Counting and arithmetic quirksBrilliant essays, shaky sums

Language models have famously miscounted letters in words (the classic test: “how many r’s are in strawberry?”) and slipped on long sums or exact word counts, while writing elegant paragraphs. Newer models that “think” step by step or use a built-in calculator do much better As of 2026, but exact counting and precise arithmetic remain worth checking.

Why: models read text as tokens — chunks of words — not individual letters. See the illustration below.

5SycophancyTelling you what you want to hear

Sycophancy is the tendency to agree with you, flatter you, or change a correct answer when you push back (“Are you sure? I think it’s 1957.” — “You’re right, my apologies!”). In April 2025 OpenAI rolled back an update to ChatGPT after users found it had become excessively flattering.

Why: assistants are fine-tuned on human ratings, and people tend to rate agreeable, pleasing answers highly. A 2023 study by Anthropic researchers found this pattern across several leading assistants.

6InconsistencyAsk twice, get two answers

Ask the same question twice, or phrase it slightly differently, and you may get different answers — sometimes contradictory ones.

Why: the model picks each next word with some randomness (the temperature setting you met earlier), and small changes in wording shift its predictions. Variety is useful for brainstorming; for facts it is a warning sign. If two answers disagree, at least one is wrong.

7OverconfidenceSame tone, right or wrong

A person who is unsure usually sounds unsure. An AI assistant often uses the same calm, authoritative tone whether it is right or badly wrong, and it rarely volunteers where its knowledge runs thin.

Why: fluent, confident text is what the model learnt to produce. Its tone is a style, not a measure of certainty. Asking “how sure are you?” can help a little, but its self-assessment can be wrong too.

Why counting trips it up

Here is roughly how a model might see a sentence. Each coloured chunk is a token. This illustration cuts “strawberry” into two chunks; a real tokeniser might split it differently. Either way, the model sees a few chunks, not s-t-r-a-w-b-e-r-r-y — a bit like being asked how many times the letter “a” appears in a word you only ever heard spoken.

Real tokenisers split text in their own ways; this is an illustration.

That is also why the habit is simple: for anything exact — totals, percentages, word counts, dates — use a calculator, a spreadsheet or your own eyes, or ask the assistant to show its working step by step so you can check it.

Why these mistakes are hard to remove

Consensus Every one of these problems has improved with newer models, and none has disappeared. Better training, web search, step-by-step reasoning and citing sources all help. But the core design — predict plausible text from patterns in data — means a model can always produce something that sounds right and is not.

Debated Whether hallucinations can ever be fully eliminated is an open question among researchers.

Other views
  • Fixable with better incentives: some researchers argue that if training and testing stopped rewarding confident guesses and rewarded “I’m not sure”, hallucinations could fall a long way.
  • Built in: others argue that any system generating text from patterns will sometimes produce falsehoods, so the realistic goal is to make errors rare, visible and checkable — not zero.
  • Ground it: a practical middle view: tie answers to real documents and search results (called grounding) so that claims can be traced and checked.

How to protect yourself

You do not need to stop using AI. You need a few habits. Work up this ladder; the higher the stakes, the further up you should go.

  1. Know the danger zonesBe extra careful with names, dates, numbers, quotations, references, recent events, health, money and law. These are where errors cluster and where they hurt most.
  2. Ask for sources — then open themAsk “where does this come from?” and click the links. A reference you have not opened is not a reference. Fake citations look exactly like real ones.
  3. Don’t lead the witnessAsk neutrally: “What are the arguments for and against…?” rather than “Isn’t it true that…?” Ask it to point out weaknesses in your idea, not to praise it.
  4. Cross-check what mattersConfirm important facts with an official site, a second reliable source or a knowledgeable person. If two AI answers disagree, treat both as unconfirmed.
  5. Use the right tool for exact workCalculators for sums, the official site for prices and rules, a doctor or pharmacist for medicines, a lawyer for legal steps.
  6. Stay the responsible oneIf you send it, sign it or act on it, you own it — just as the lawyers in Mata v. Avianca found out. Use AI as a capable assistant, not as the final authority.
Try this prompt
Here is my plan: [describe a plan, with no personal details]. Do not tell me it is good. List the three weakest points and the questions I should check with a real expert.

For a deeper toolkit of verification habits, the Practitioner page Checking AI’s Work goes further.

Quick check

An assistant gives you a list of five books with authors and years, all in a confident tone. What should you assume?

Confident tone is not evidence. References are a classic hallucination zone, so check each one exists.

Why might an AI miscount the letters in a word?

The model sees a word like “strawberry” as a few multi-letter chunks, so letter-level tasks are surprisingly awkward for it.

You tell the assistant its correct answer is wrong, and it immediately agrees with you. What is this called?

Sycophancy — telling you what you want to hear — partly comes from training on human approval ratings.

What did the 2018 Gender Shades study find?

Buolamwini and Gebru found error rates of up to about 35% for darker-skinned women versus under 1% for lighter-skinned men.

You ask about a law that changed last month. What is the main risk?

Knowledge cutoffs mean recent changes may be missing. Check the official source.

// Pause and reflect

  1. Think of something you have asked, or might ask, an AI. Which of the seven failures would be most dangerous for that task?
  2. Where would you go to check the answer — which official site, book or person?
  3. Have you ever been flattered into a decision, by a person or a machine? How could you phrase a question so the AI challenges you instead?

Your notes are saved privately in this browser.

Try this

Ask an assistant for three facts about your hometown or school, with sources. Then check each one yourself. Note which were right, which were wrong, and whether the sources it gave actually said what it claimed.

Key terms on this page

Sources & further reading