Rocket wants to change the comet cut-off right away. "T7 was size 7, so I will set it to 7. Done!"
Raven stops him. "If we tune Sky Sorter on the test cards, they are not a fair test any more."
"Then how do we improve it?" Rocket asks.
Nova shows two paths on the wall. "Add more labeled comets to the training set," she says. "Let the training data show the cut-off."
"Then test on a fresh envelope of cards nobody has used yet."
Raven nods and reaches for a new envelope.
To refine a sorter, change it using the training set, not the test set.
The crew labels twenty more comets. Several are only 7 pixels wide, and no planet in the training set is that big.
Counting again, a comet cut-off of size 7 now fits the training data best.
Then they test version 3 on ten fresh cards nobody has used before.
SKY SORTER, VERSION 3 IF streak is yes THEN label = "satellite streak" ELSE IF size >= 7 THEN label = "comet" ELSE IF size >= 4 THEN label = "planet" ELSE label = "star"
| Card | Brightness | Size | Streak? | True label | Version 3 label |
|---|---|---|---|---|---|
| N1 | 140 | 2 | no | star | star |
| N2 | 225 | 6 | no | planet | planet |
| N3 | 110 | 7 | no | comet | comet |
| N4 | 190 | 1 | yes | satellite streak | satellite streak |
| N5 | 80 | 11 | no | comet | comet |
| N6 | 215 | 4 | no | planet | planet |
| N7 | 70 | 2 | no | star | star |
| N8 | 160 | 5 | no | planet | planet |
| N9 | 95 | 3 | no | planet | star |
| N10 | 200 | 1 | yes | satellite streak | satellite streak |
| Statement | True or false? |
|---|---|
| Refining a sorter should use the training set. | ? |
| Version 3 lowered the comet cut-off from 8 to 7. | ? |
| After refining, the crew retested on fresh cards. | ? |
| Version 3 is now correct on every possible card. | ? |
Excellent refining! Tomorrow we see what happens when the data itself changes.