← Back to course
AI Explorers 6-8 / Week 06 / Explorer Quest
6/6
Week 06 · Train, Test, Refine

Explorer Quest

Explorer Quest: Train and test a guesser
// Measuring how well Sky Sorter really works
⏱ about 20 min

Explorer Quest: Train and Test a Guesser

It is game night at the station. Raven has written twenty index cards, each with an animal and two features.

"Rocket, you get twelve training cards with labels," she says. "Find a rule for land or water animals."

She tucks the other eight cards into an envelope. "These are the test set. You do not get to peek."

Rocket studies the training cards and writes a rule. Nova reads it aloud and opens the envelope.

"Six out of eight," she announces. "Seventy-five percent. Want to refine and try fresh cards?"

  1. Machine learning needs tremendous amounts of training data, usually supplied by people.
  2. Labels must be checked, because a wrong label teaches a wrong pattern.
  3. A test set is held back and never used to build the sorter.
  4. Accuracy is the number right divided by the total tested.
  5. Refine using training data, then test again on fresh cards. Data can change over time, so keep testing.
◇ TRAIN AND TEST A GUESSER
Write twenty index cards. Each card names an animal and two features, like "has fur: yes, number of legs: 4."
Label each card land or water. Check each other's labels.
Shuffle and deal twelve cards into a training pile. Seal the other eight in an envelope.
Using only the training pile, write one if-then rule that sorts land from water.
Open the envelope. Run your rule on each test card and count how many are right.
Work out your accuracy as a percent.
To refine, write eight new cards. Use four to fix your rule, and seal four as a fresh test set.
Run the refined rule on the four fresh cards and work out the new accuracy.
For a grown-up
Help write the animal cards if needed. Any common animals work well.
Ask: why does peeking at the test cards make the score less honest?
Ask: what would happen if one of the training cards had a wrong label?
QUEST CHECK
  • Read the question.
  • Tap your answer.
Your rule gets 6 of 8 test cards right. What is the accuracy?
Someone peeked at the test cards before writing the rule. What happens to the score?
The cards kept in the envelope are called the test ____.
WHY THIS EXERCISEThe held-back test set is what makes the accuracy honest.
← Friday