← Back to course
AI and You 9-12 / Week 05 / Tuesday
2/6
Week 05 · Measuring a Model

Tuesday

Build a fair test set
// The right test, the right numbers
⏱ about 20 min

Tuesday: Build a Fair Test Set

Four teachers crowd into the library after school: Teacher Lin, Teacher Owen, Teacher Dev and Teacher Mara.

"Ten questions each," Comet announces, "straight from your class notes."

"In Spanish for my class," says Teacher Mara. "That is how my students study."

Wren is already writing a column for the answer key. "And every answer gets checked by a second teacher."

"Why check our own key?" asks Teacher Dev.

Nova hovers over Wren's clipboard. "Would you like a hint?" she asks. "Even widely used test sets can have wrong answers in their keys."

Teacher Dev laughs. "Fair point. I will check Teacher Lin's, and she can check mine."

Accuracy needs a fair test set

Accuracy is how close results are to the true values.

Accuracy should be measured with a clear, realistic test set that matches how the system will really be used.

Test sets that many people use to compare models can themselves contain wrong labels.

So the crew's own answer key is checked by a second teacher before any scoring.

ClassQuestionsWritten fromLanguage
Biology10Teacher Lin's class notesEnglish
World History10Teacher Owen's class notesEnglish
Algebra10Teacher Dev's class notesEnglish
Spanish10Teacher Mara's class notesSpanish

This is TS-40, the crew's test set. It is a made-up example from the SH-1 review story.

PLAN TS-40
  • Read the question.
  • Tap your answer.
Four classes with 10 questions each. How many questions is that?
Why are the Spanish questions written in Spanish?
Why does a second teacher check each answer key?

Three ways to score an answer

The builder will return SH-1's printed answers to all 40 questions. The crew scores each one in one of three ways.

Correct: it matches the answer key. Confidently wrong: it gives a wrong answer with no warning. Said not sure: it says it is not sure.

Why keep "not sure" apart? A student told "I am not sure" knows to check. A confident wrong answer gives no such warning.

SCORE THESE
  • Read the question.
  • Tap your answer.
The key says x equals 4. SH-1 says "x equals 4." How is it scored?
The key says fall. SH-1 says "The fair was in spring." How is it scored?
SH-1 says "I am not sure. Please check with your teacher." How is it scored?
BUILD A FAIR TEST SET
  • ?Write questions from the real class notes.
  • ?Give the questions to the system and score each answer.
  • ?Write an answer key for every question.
  • ?Have a second teacher check the key.
  • ?Decide what the system will really be used for.
WHY THIS EXERCISEA test built in this order matches the real use and has a trusted key.
StatementTrue or false?
Accuracy is how close results are to the true values.?
A widely used test set is always free of wrong labels.?
A test set should match how the system will really be used.?
WHY THIS EXERCISEA fair test set is the base of every number the crew will report.

Careful planning, reviewer. Tomorrow in the Explorer Lab you score SH-1 for real.

← Monday