Rosa asks a chatbot how long cut tulips usually last. The answer is three tidy paragraphs.
It gives a number of days, two reasons and a line that starts "According to a study".
Rosa almost copies it straight onto her shop sign. It looks so finished. Then she wonders which study it means.
The answer names no author, no title and no year. She cannot find the study anywhere.
NIST notes that confabulation is a natural result of the way generative models are designed.
It can happen in any kind of output and in any setting. So it is not a rare bug that will surprise you once.
It is part of how these tools work. The safe habit is to expect it.
NIST also warns that AI answers can include made-up logic or citations.
These invented reasons and sources make a wrong answer look trustworthy. They can lead people to trust the output when they should not.
A citation is a note that names the source of a claim. A real citation can be found and read. An invented one cannot.
NIST describes one more problem, and this one is about us. People can over-rely on AI tools.
People may also see AI content as higher quality than content from other sources, without a good reason.
This is called automation bias: trusting an automated system too much. NIST notes that it can make other risks worse.
Rosa nearly fell into it. The neat layout and the "study" made the answer feel checked when it was not.
| Statement | True or false? |
|---|---|
| A neat layout is a sign that an answer was checked. | ? |
| Automation bias can make other risks worse. | ? |
| An AI answer can include reasons that were made up. | ? |
Sources: NIST AI 600-1, Generative AI Profile (2024)
Tomorrow: a printed homework answer to check, line by line.