Comet could not wait. Last night she ran a quick trial of one question with 20 members.
"Big news," she says. "Almost everyone visits in the evening! Fourteen out of twenty!"
Wren looks at the time on her sheet. "What do you notice? You asked between six and nine at night."
Comet's face falls. "Oh. Everybody I asked was there in the evening."
Nova projects a second sheet. "Would you like a hint?" she asks. "Wren ran the same question at four different times. Compare the two."
Wren smiles at you. "Data analyst, which trial should the crew trust more?"
Both trials asked 20 members "When do you usually visit?" These trial numbers are made up for the Commons Count story.
Comet asked only in the evening. Wren asked in the morning, the afternoon, the evening and on the weekend.
| Usual visit time | Trial A: Comet, evening only | Trial B: Wren, four times |
|---|---|---|
| Morning | 1 | 5 |
| Afternoon | 3 | 5 |
| Evening | 14 | 6 |
| Weekend | 2 | 4 |
| Total | 20 | 20 |
Bias often comes from the type or source of the data collected. Collecting more data does not remove it.
If Comet asked 40 evening members instead of 20, her trial would still lean toward the evening.
The Census Bureau does not let addresses picked for its survey drop out. If they could, the sample's data would quickly become less valid.
It also keeps its sample as small as it can, to lighten the load on the people who answer.
| Statement | True or false? |
|---|---|
| Asking only in the evening changed the trial's result. | ? |
| Collecting more data removes bias that comes from the source. | ? |
| Bias often comes from the type or source of the data. | ? |
| Both trials asked 20 members. | ? |
Careful work. Tomorrow is Impact Friday, where you look at bias in what people build.