Maya's daughter asks a homework helper how many groups of four fit in her class of twenty-four.
The answer comes back fast, in friendly words, with a neat check at the end. It says eight groups.
Maya counts on her fingers. Four, eight, twelve, sixteen, twenty, twenty-four. That is six groups, not eight.
The answer did not sound unsure at all. How can something so confident be so wrong?
Last week you saw that text tools work by predicting the next word. NIST notes that this can give accurate answers.
It can also give answers that are wrong, or that contradict themselves.
NIST has a name for one kind of wrong answer. Confabulation is content an AI tool states with confidence that is wrong or false.
Many people call this a "hallucination". NIST notes that users may be misled or deceived by it.
NIST names a second problem. Generated content may not separate fact from opinion or fiction.
It also may not say when it is unsure.
So a chatbot can mix a fact, a guess and an invented detail in one smooth paragraph. All three can sound the same.
| What you read | What it might really be |
|---|---|
| "The answer is eight." | A fact, or a wrong guess stated as a fact |
| "This is the best method." | An opinion presented as a fact |
| "Experts agree that..." | A claim with no named source |
| No "I am not sure" anywhere | Not proof that the tool is sure, or right |
| Statement | True or false? |
|---|---|
| Next-word prediction can produce accurate answers. | ? |
| Next-word prediction can produce answers that contradict themselves. | ? |
| A chatbot always tells you when it is unsure. | ? |
| Generated content always keeps fact and opinion apart. | ? |
Sources: NIST AI 600-1, Generative AI Profile (2024)
Tomorrow: made-up reasons and sources, and why we believe them so easily.