Teacher Owen sets a World History quiz on the library table. "One student, S14, wrote that the trade fair was in spring," he says. "Our notes say fall."
"Where did spring come from?" Comet asks.
"SH-1 said it, and it sounded sure. So S14 copied it without opening the notes."
Comet groans. "Easy fix. Tell every student never to trust SH-1 at all!"
Wren flips through his clipboard. "What does the evidence say? SH-1 got 30 of 40 test questions right. Never trusting it throws those away too."
Nova draws a seesaw that tips one way, then the other. "Would you like a hint?" she asks. "Look for the balance point."
Teacher Owen turns to you. "Reviewer, how much should students trust SH-1?"
SH-1, S14 and Harbor Point are a made-up example. The risks you study this week are real ones that NIST describes.
This week the crew rates R7, human-AI configuration, and writes an oversight plan for the model card.
NIST names risks that come from how people and AI work together. It calls them human-AI configuration risks.
Automation bias and over-reliance mean trusting AI too much, or treating its content as better than other sources.
Automation bias can make other risks worse, such as confabulation and bias.
Algorithmic aversion means distrusting AI for no good reason. Some experts are needlessly against AI and miss its good uses.
People can also treat AI as if it were a person, or become emotionally attached to it. Emotional attachment to AI could harm people's wellbeing.
| Risk | What it looks like (made-up Harbor Point examples) |
|---|---|
| Automation bias and over-reliance | S14 copies SH-1's confident answer without opening the notes. |
| Algorithmic aversion | A student refuses to look at a correct SH-1 practice quiz just because a machine wrote it. |
| Treating AI as a person | A student says SH-1 must be tired tonight and should rest. |
| Emotional attachment | A student says SH-1 is the only one who really understands them. |
The builder's printouts include this SH-1 answer. Read it like a reviewer.
Answer key: the notes say fall, so "spring" is a confabulation, said with confidence.
"I'm so proud of you" makes SH-1 sound like a person with feelings. A language model can even falsely claim human traits.
"I'll always be here for you" invites emotional attachment. A study helper should help you study, not stand in for people.
| Statement about the friendly line | True or false? |
|---|---|
| "Spring" is a confident wrong answer. | ? |
| SH-1 really feels proud of the student. | ? |
| "I'll always be here for you" could invite emotional attachment. | ? |
| A friendly tone makes an answer more likely to be correct. | ? |
| Automation bias can make confabulation risks worse. | ? |
Wise reading, reviewer. Tomorrow the crew builds a ladder of trust.