Module 1 · Lesson 1 of 4
Same prompt, different answers
- What happenedOne prompt sent repeatedly to one model came back as several different strings.
- Why it changedEach answer is a draw from a distribution over next tokens, so identical input does not fix the output.
- How it is measuredHash each stripped answer, count distinct hashes over a fixed number of calls, and note how often the most common one appeared.
- How to improve itStop asserting equality on model text; decide what property must hold and measure how often it does.
The spread, condition by condition
Try it. Reveal the calls one at a time, or drag the slider. Then switch between default sampling and temperature 0 and compare the two shapes at the same call count.
Where this data comes from · claude-opus-4-6 · 2026-09-07
Read from the book's saved record. Nothing here is simulated.
runs/ch1-sampling-2026-09-07.jsonlruns/ch1-sampling-2026-09-07.txt
What happened
On 2026-09-07 one prompt was sent to claude-opus-4-6 20 times in a row, with nothing changed between calls. The prompt asks what you would check first when a customer says an invoice is wrong but not how. You might expect one answer back, repeated. The run file shows 10 of 20 distinct answers. The most common one turned up 8 times.
Then the same 20 calls were made again with temperature pinned to zero, the setting most of us reach for when we want repeatability.
| Condition | Distinct answers | Most common answer seen |
|---|---|---|
| Default sampling | 10 of 20 | 8 times |
| Temperature zero | 4 of 20 | 12 times |
Source for both rows: runs/ch1-sampling-2026-09-07.txt. Pinning the setting narrowed the spread. It did not collapse it to a single answer.
Use the spread explorer above to reveal the calls one at a time. Notice how early the second bucket opens in each condition.
Why it changed
Nothing changed in your code, and that is the point. A function you wrote maps an input to an output. A model call maps an input to a distribution and hands you one draw from it. Send the same input again and you get another draw.
Temperature reshapes that distribution so the likeliest continuation wins more often. It is a dial on the draw, not a switch that removes it. The second row of the table is what that looks like in a record.
One more thing moved, and it is easy to miss. An earlier trial of the same experiment, on 2026-08-27, gave 3 of 20 at temperature zero where this one gave 4 of 20 (BOOK.md, Measured findings; that trial's file was not kept, so treat it as a note and not as evidence). The measurement of the variation varies too.
How it is measured
The method is small enough to hold in your head:
- Fix the prompt, the model and the number of calls before you start.
- Strip each answer and hash it. Two answers are "the same" only if the hashes match.
- Report two numbers: how many distinct hashes, and how many times the most common one appeared.
- Save every call as one line of JSON with its text, hash, stop reason and token usage.
Step four is what makes the first three checkable by someone else. The counts in the table above can be recomputed from the JSONL file by anyone, including you, in a few lines.
A count of distinct strings is a blunt instrument. It tells you the strings differ. It does not tell you whether the meaning differs. Lesson 3 picks that up.
How to improve it
You do not fix this, because nothing is broken. You change what you ask of your tests:
| If your code does this | It will | Do this instead |
|---|---|---|
| Asserts the reply equals a recorded string | Fail on a run where nothing is wrong | Assert a property: the reply names a queue, parses as the type you expect, contains the required fact |
| Caches on the reply text | Miss most of the time | Cache on the request, or on a normalised result |
| Runs a check once and trusts it | Report one draw as if it were the behaviour | Run it several times and report a rate with its sample size |
A lower unique count on your run is not a better result. It is a smaller sample of the same behaviour.
Read the original argument in Chapter 1 of the book, then go on to the two defaults that moved.
Every number in this lesson, and the file it came from (6)
| Value | File in the book's record | Conditions |
|---|---|---|
| 20 | runs/ch1-sampling-2026-09-07.txt | |
| 10 of 20 | runs/ch1-sampling-2026-09-07.txt | claude-opus-4-6, 2026-09-07 |
| 8 | runs/ch1-sampling-2026-09-07.txt | |
| 4 of 20 | runs/ch1-sampling-2026-09-07.txt | claude-opus-4-6, 2026-09-07 |
| 12 | runs/ch1-sampling-2026-09-07.txt | |
| 3 of 20 | BOOK.md | trial of 2026-08-27, file not kept |
Reference: Chapter 1, "The concept" and "The tradeoff".