Raven flips through the crew's mission log. "Look how often we write the same kinds of sentences."
She reads aloud. "Rocket spots a comet. Raven spots a planet. Rocket spots a comet again."
Rocket laughs. "We are not very creative log writers."
Nova taps the page with a beam of light. "But you are very predictable," she says.
"If I say 'Rocket', what word do you expect next?"
"Spots!" Rocket and Raven answer together.
"Then you just did a tiny version of next-word prediction," Nova says. "Let us count it carefully."
NIST explains that generative models make outputs that approximate the statistical patterns of their training data.
For example, large language models predict the next word in a sentence.
You can try a tiny paper version. Count which word follows which in some text, then pick the most common next word.
| Mission log line |
|---|
| Rocket spots a comet. |
| Raven spots a planet. |
| Rocket spots a star. |
| Rocket logs a comet. |
| Raven logs a star. |
| Rocket spots a comet. |
Here is a tally of the word that comes right after "Rocket" and the word that comes right after "spots a".
| After this | Next word | Tally |
|---|---|---|
| Rocket | spots | 3 |
| Rocket | logs | 1 |
| spots a | comet | 2 |
| spots a | planet | 1 |
| spots a | star | 1 |
| Statement | True or false? |
|---|---|
| Large language models predict the next word in a sentence. | ? |
| The paper predictor can choose a word it never saw in training. | ? |
| A bigger tally means a word is more likely to be picked. | ? |
Great predicting. Tomorrow in the Explorer Lab, you build a full paper predictor and let it write.