Teacher Dev opens the grade book on the library screen. "This program has done my averages for years," he says. "Why does SH-1 need a whole panel?"
Comet shrugs. "They are both software. If one is fine, the other should be too."
Wren flips through the packet. "What does the evidence say? Teacher Dev can read every formula in his grade book. Can anyone read SH-1's?"
Nova projects the two programs side by side. "Would you like a hint?" she asks. "Ask where each one gets its answers."
Teacher Dev turns to you. "Reviewer, help me see the difference."
Teacher Dev's grade book follows formulas a person wrote. The same scores always give the same average, and anyone can check the formula.
SH-1 is an AI system built on a large language model. Its answers come from what it learned from data, not from formulas a teacher wrote.
That difference brings risks that ordinary software does not have.
NIST lists risks of AI that ordinary software does not share. Here are four of them in plain words.
The data used to build an AI system may not truly represent the place where it is used.
The true answer may not exist, or it may not be known.
Training data can go stale, falling out of date for the place the system is used.
AI systems can be hard to see into, and their results can be hard to reproduce.
| Risk | How it could show up with SH-1 (made up) |
|---|---|
| Data may not match the setting | P3 says "public text and study material," not Harbor Point's own class notes. |
| The true answer may not exist | A student asks, "What is the best way to study?" There is no single true answer. |
| Training data can go stale | SH-1 may have learned old library hours from an old school page. |
| Hard to see into and reproduce | The builder's printout shows SH-1 giving two different answers to the same question. |
AI systems may be trained on data that changes over time. That can change how well they work in ways that are hard to understand.
So AI systems may need fixing more often than ordinary software.
Many AI systems are also huge. Some contain billions or even trillions of decision points, housed inside more ordinary software.
Teacher Dev can read his grade book's formulas one by one. Nobody could read billions of decision points that way.
| Statement | True or false? |
|---|---|
| Data that changes over time can change how well an AI system works. | ? |
| An AI system never needs fixing once it is built. | ? |
| Some AI systems contain billions or even trillions of decision points. | ? |
| The grade book and SH-1 get their answers in the same way. | ? |
| For some questions, the true answer may not exist. | ? |
Sharp thinking, reviewer. Tomorrow is Impact Friday: who does SH-1 touch, even people who never use it?