Wren deals six index cards onto the library table. Each card holds one short sentence about the school.
"Meet Tiny Text," he says. "A model so small we can build it by hand."
Comet reads the first card aloud. "The library opens at eight. That is not exactly exciting."
"It doesn't have to be exciting," Wren says. "It has to be countable. Which word follows which?"
Nova projects an empty tally table above the cards. "Would you like a hint?" she asks. "Take two words at a time, then slide over one word."
Comet grabs a pencil and passes another to you. "Reviewer, let's count the first three cards together."
Generative models make outputs that follow the statistical patterns of their training data.
Large language models predict the next word, or token, in a sentence.
Tiny Text is a made-up next-word model the crew trains by hand. Its training data is six sentences on index cards.
Real language models are enormously bigger. Some AI systems contain billions or even trillions of decision points.
| Card | Training sentence |
|---|---|
| T1 | the library opens at eight |
| T2 | the library has quiet rooms |
| T3 | the lab opens at nine |
| T4 | the lab has new desks |
| T5 | the library opens at eight |
| T6 | the gym opens at seven |
A next-word pair is two words side by side in a sentence. T1 has four pairs: the library, library opens, opens at, at eight.
Every card has five words, so every card has four pairs.
Tiny Text's model is a table that counts how often each word follows another. A model can be a table of patterns.
| After this word | Next words in T1 to T3 (tally) | Total |
|---|---|---|
| the | library 2, lab 1 | 3 |
| library | opens 1, has 1 | 2 |
| lab | opens 1 | 1 |
| opens | at 2 | 2 |
| at | eight 1, nine 1 | 2 |
| has | quiet 1 | 1 |
| quiet | rooms 1 | 1 |
| Statement | True or false? |
|---|---|
| Large language models predict the next word in a sentence. | ? |
| Generative models make outputs that follow patterns in their training data. | ? |
| Tiny Text is a real AI product. | ? |
| A model can be as simple as a table of patterns. | ? |
| Real language models are about the same size as Tiny Text. | ? |
Nice counting, reviewer. Tomorrow in the Explorer Lab you train all of Tiny Text.