Ben hears about a reading app that listens to students read aloud and scores them.
He thinks of his class. Some students have the local accent. Some grew up in other regions, and some speak two languages at home.
Would the app hear all of them equally well? Ben realizes he does not know, and that the answer matters.
NIST's Generative AI Profile lists harmful bias as one of the risks of generative AI.
It says these tools can amplify historical, social and systemic biases. Amplify means make louder or bigger.
It also names performance gaps. A tool can work worse for some groups or some languages.
NIST says this may come from training data that does not represent everyone the tool is used for.
This table is a way to think about it, not a list of proven causes for any one tool.
| Stage | What can go wrong | A question to ask |
|---|---|---|
| Collecting data | Some people or languages are rare or missing. | Who is missing from the data? |
| Learning patterns | Old, unfair patterns in the data get copied. | What past habits might it repeat? |
| Making outputs | The tool works worse for some groups. | Who does it get wrong more often? |
| Using outputs | People trust the result without checking. | Who checks before anyone acts on it? |
The last row links to week 4. NIST describes automation bias as trusting a machine too much, and says it can make other risks worse.
A biased output that nobody checks can do more harm than one a person questions.
| Statement | True or false? |
|---|---|
| NIST says generative AI can amplify historical and social biases. | ? |
| NIST says tools can work worse for some languages. | ? |
| Bias can only enter when data is collected. | ? |
| Checking an output before acting on it can stop some harm. | ? |
Sources: NIST AI 600-1, Generative AI Profile (2024) · AI4K12, Five Big Ideas in Artificial Intelligence (poster, version 2)
Tomorrow: a printed output with biased lines to find.