AI confidence can outrun its evidence

Language models can learn how a convincing answer sounds without having the fact needed to support it, while training and evaluation can reward guessing. Treat confidence as a feature of the wording and verify the claims that could change a decision.

A 2025 analysis by Adam Tauman Kalai and colleagues explains why predicting plausible text can produce factual errors even when the training material contains no mistakes. Grammar repeats across countless examples, but an obscure birthday offers little structure from which to recover the right date. Their example is telling: DeepSeek-V3 gave three different, incorrect birthdays for Kalai despite instructions to answer only if it knew.

Scores can make guessing look attractive because a lucky answer earns credit while declining to answer earns none. In an OpenAI SimpleQA comparison published in 2025, gpt-5-thinking-mini answered 22% correctly and 26% incorrectly, while o4-mini answered 24% correctly and 75% incorrectly. Those are historical benchmark results, rather than everyday error rates, but they show how two extra percentage points of accuracy can hide a much larger increase in wrong answers.

Asking for reassurance supplies no independent evidence: Mrinank Sharma and colleagues found that tested assistants could shift toward users' beliefs, sometimes replacing correct answers after a challenge. Agreement across repeated answers also has limits, because the Oxford researchers studying semantic entropy distinguish inconsistent guesses from stable misconceptions their method cannot resolve. A second answer can flag a problem, but agreement cannot establish that the first answer was true.

A practical check starts by extracting the names, dates, numbers, and causal claims that matter, then locating the original evidence for each. Read the supporting passage, confirm its date and scope, recalculate arithmetic, and run code in an appropriate test environment; a working citation only proves that a page exists. Keep the evidence beside the conclusion so an unsupported detail remains visible instead of gaining authority through repetition.

If the supporting source, calculation, or test is missing, leave that part unresolved before acting on it.