Comet reads the description of SH-1's Check feature aloud. "Check My Answer: marks a student's answer right or wrong."
"So let's test it," says Wren. "We write 20 student answers. Ten really right, ten really wrong."
The builder returns Check's marks. Comet counts. "Fifteen of twenty right! Same as TS-40: 75 percent."
Wren is not done. "But which five did it get wrong? A false alarm is not the same as a missed error."
Nova hovers over the results and projects a grid with four boxes. "Would you like a hint?" she asks. "Sort each mark by what was true and what Check said."
Check's job is to flag wrong answers. So a positive means "Check flags this answer as wrong."
A false positive is a false alarm: Check flags an answer that is really right.
A false negative is a missed error: Check says a wrong answer is right.
Accuracy measures should include false positive and false negative rates.
| What is true | SH-1 flags it wrong | SH-1 says it is right | Total |
|---|---|---|---|
| Really wrong | 7 | 3 (false negatives: missed errors) | 10 |
| Really right | 2 (false positives: false alarms) | 8 | 10 |
| Total | 9 | 11 | 20 |
This is CK-20, the crew's answer-checker test. It is a made-up example from the SH-1 review story.
Some kinds of failure cause more harm than others.
A person may need to step in when a system cannot find or fix its own errors.
The crew judges the false negatives worse for learning. A student told a wrong answer is right keeps a wrong idea.
A false alarm is annoying, but the student will probably check with the teacher and find out.
Robustness is keeping performance up in many different conditions.
A system that is inaccurate, unreliable or works badly outside its training conditions increases risk.
All 20 CK-20 answers were typed. Wren asks: would Check work as well on answers a student writes by hand and photographs?
The crew has no evidence either way, so they write it down as "not tested."
| Statement | True or false? |
|---|---|
| A false positive for Check means flagging a right answer as wrong. | ? |
| If two tests both score 75 percent, they found the same kinds of mistakes. | ? |
| Some kinds of failure cause more harm than others. | ? |
| CK-20 shows that Check works well on handwritten answers. | ? |
Sharp thinking, reviewer. Tomorrow is Impact Friday: what a score on the wrong test can do to a whole school.