Comet points at packet section P4. "The builder tested SH-1 and got 92 out of 100. Testing done!"
Wren tilts his head. "What does the evidence say? Who wrote that test? Who scored it?"
Comet flips the page. "The builder. Oh. The builder checked its own work."
Nova projects a chart of job titles beside the lifecycle arrow. "Would you like a hint?" she asks. "Match each job to its stage, then notice who checks whom."
Teacher Mara leans over the table. "I know Spanish class better than any builder," she says. "Reviewer, where do teachers like me fit?"
NIST describes the jobs at each lifecycle stage and the people who do them. Here they are in plain words.
| Stage | The work | Some of the people who do it |
|---|---|---|
| Design | Write down goals, setting and requirements; gather and clean data; document the data | Data scientists, domain experts, members of affected communities, user-experience designers, privacy experts |
| Development | Build, choose, tune, train and test models | Machine learning experts, data scientists, developers |
| Deployment | Pilot the system, check it works with older systems, follow the rules, manage the change, check what using it is like | System integrators, software developers, end users, operators, evaluators |
| Operation and monitoring | Run the system and regularly check its output and effects with others | System operators, domain experts, users who act on the output, evaluators, auditors, researchers |
| TEVV | Examine the system, find problems and fix them, starting as early as design | People who examine the system, ideally not the people who build and use it |
NIST says that, ideally, the people who check a model are not the same people who build and use it.
A builder who scores its own test may not notice what it left out. A separate checker brings fresh eyes.
P4's score of 92 came from the builder's own benchmark. The crew will test SH-1 with its own questions in week 5.
Domain experts appear in both design and operation and monitoring. They know the field where a system is used.
At Harbor Point, the teachers are the domain experts. Teacher Mara knows Spanish class better than any builder.
| Statement | True or false? |
|---|---|
| Gathering and cleaning data is design work. | ? |
| End users can be part of deployment work. | ? |
| Ideally, the people who check a model are the same people who built it. | ? |
| TEVV can start as early as design. | ? |
| Teachers can be domain experts for a study helper. | ? |
Nice matching, reviewer. Tomorrow in the Explorer Lab you build a risk grid on paper.