The review board spreads out twenty fresh snapshot cards. Sky Sorter has never seen any of them.
"This is the final review test," Raven says. "We count, we score, we write feedback."
Rocket deals out the cards. "And if Sky Sorter does badly?"
Nova hovers over the scorecard. "Then the board writes down exactly what to fix," she says.
"A review is not about passing or failing. It is about making the next version better."
Rocket picks up the first card, a dim, fuzzy smudge. "Here we go," he says.
Here are the board's results on 20 new cards. These numbers are for this review only.
| Label | Cards in test | Sorted right |
|---|---|---|
| Star | 5 | 5 |
| Planet | 5 | 5 |
| Comet | 5 | 2 |
| Satellite streak | 5 | 4 |
| Total | 20 | ? |
| Characteristic | Evidence | Strong or needs work |
|---|---|---|
| Valid and reliable | 16 of 20 right | |
| Fair, with harmful bias managed | Comets: 2 of 5 right, all 3 misses dim | |
| Explainable and interpretable | Decision tree path can be followed | |
| Privacy-enhanced | Names and locations removed |
| Board idea | True or false? |
|---|---|
| A review test should use cards the system has never seen. | ? |
| A review is only about passing or failing. | ? |
| Feedback should say exactly what to fix. | ? |
| If overall accuracy is high, the board can skip checking each label. | ? |
A thorough review. Tomorrow you use the feedback to write Sky Sorter's design brief.