Module 1 · Lesson 4 of 4
Predict, run, compare
- What happenedThe author predicted the unique count before each course run and was wrong both times, by a wide margin.
- Why it changedA prediction made before a run exposes what you believed; one made after is shaped by the result.
- How it is measuredA prediction file with a timestamp earlier than the first call, compared by the check script against the count in the run file.
- How to improve itWrite the prediction down first, save raw output, report the miss, and let a script do the comparison.
Your guess against the record
Try it. Move the slider to your own prediction of the unique count on a current model, then reveal the record. You cannot change your number afterwards.
Where this data comes from · claude-opus-5 · 2026-11-08
Read from the book's saved record. Nothing here is simulated.
runs/ch1-course-prediction-2026-11-08-mt1000.jsonruns/ch1-course-check-2026-11-08-mt1000.txtruns/ch1-sampling-2026-09-07.txt
What happened
Before the course run on claude-opus-5, a prediction was written to a file: 12 distinct answers. The run came back with 20 of 20. The check script printed the miss, off by 8 (runs/ch1-course-check-2026-11-08-mt1000.txt).
The check still passed. Being wrong about the count is not a failure in this course. Not having written the prediction down is.
Try it now with the slider above, before you look at anything else on this page. Then reveal.
Why it changed
Compare two ways of meeting that result:
| Without a prediction | With a prediction recorded first |
|---|---|
| You see the count and it seems reasonable | You see the count next to the number you committed to |
| "Newer models vary more, of course" | "I expected about half as much variation as I got" |
| Nothing to follow up | A specific question: what did I assume that was wrong? |
The run is identical in both columns. Only the second one teaches you anything, because hindsight has nothing to rewrite.
The book run on the older model gave 10 of 20 (runs/ch1-sampling-2026-09-07.txt). A reasonable person would have guessed near that for the newer one. The record says otherwise, and now it is a fact with a model and a date on it.
How it is measured
Every module in this course uses the same three steps, and the check script enforces the order:
- Predict. The harness writes
prediction.jsonwith your number and a UTC timestamp. - Run. Real calls on your key, saved one line per call, with the text, the hash, the stop reason and the token usage. The harness refuses to start without a prediction.
- Compare. The check script confirms the prediction is dated before the first call, recomputes the unique count from the file and prints how far off you were.
The script also checks the things from the first three lessons: the full set of calls is there, the default arm only, the text matches its hash so the file was not edited, no answer is empty, none stopped on max_tokens.
One limit, stated plainly: the timestamps come from your own machine. The check enforces the habit, not honesty. When this test moves into Measured Lab, the Lab records the prediction on its side.
How to improve it
- Predict a number, not a direction. "More than last time" cannot be wrong enough to teach you anything.
- Save the raw output. A screenshot or a pasted summary cannot be recomputed. A JSONL file can.
- Report the miss. Write down how far off you were and what you had assumed. The book reports the findings that went against its own drafts; the course asks the same of you.
- Let a script compare. If the comparison is manual, it will drift toward what you hoped to see.
You are ready for the hands-on test. The exercise it is built on is at the end of Chapter 1 of the book.
Every number in this lesson, and the file it came from (4)
| Value | File in the book's record | Conditions |
|---|---|---|
| 12 | runs/ch1-course-check-2026-11-08-mt1000.txt | |
| 20 of 20 | runs/ch1-course-check-2026-11-08-mt1000.txt | claude-opus-5, max_tokens 1000, 2026-11-08 |
| 8 | runs/ch1-course-check-2026-11-08-mt1000.txt | |
| 10 of 20 | runs/ch1-sampling-2026-09-07.txt | claude-opus-4-6, 2026-09-07 |
Reference: Chapter 1, "Hands-on exercise".