Measured Academy Measured Academy
Français Pilot Progress 0

Module 1 · Lesson 1 of 4

Same prompt, different answers

  1. What happenedOne prompt sent repeatedly to one model came back as several different strings.
  2. Why it changedEach answer is a draw from a distribution over next tokens, so identical input does not fix the output.
  3. How it is measuredHash each stripped answer, count distinct hashes over a fixed number of calls, and note how often the most common one appeared.
  4. How to improve itStop asserting equality on model text; decide what property must hold and measure how often it does.
Side by side

The spread, condition by condition

Try it. Reveal the calls one at a time, or drag the slider. Then switch between default sampling and temperature 0 and compare the two shapes at the same call count.

Where this data comes from · claude-opus-4-6 · 2026-09-07

Read from the book's saved record. Nothing here is simulated.

  • runs/ch1-sampling-2026-09-07.jsonl
  • runs/ch1-sampling-2026-09-07.txt

What happened

On 2026-09-07 one prompt was sent to claude-opus-4-6 20 times in a row, with nothing changed between calls. The prompt asks what you would check first when a customer says an invoice is wrong but not how. You might expect one answer back, repeated. The run file shows 10 of 20 distinct answers. The most common one turned up 8 times.

Then the same 20 calls were made again with temperature pinned to zero, the setting most of us reach for when we want repeatability.

Condition Distinct answers Most common answer seen
Default sampling 10 of 20 8 times
Temperature zero 4 of 20 12 times

Source for both rows: runs/ch1-sampling-2026-09-07.txt. Pinning the setting narrowed the spread. It did not collapse it to a single answer.

Use the spread explorer above to reveal the calls one at a time. Notice how early the second bucket opens in each condition.

Why it changed

Nothing changed in your code, and that is the point. A function you wrote maps an input to an output. A model call maps an input to a distribution and hands you one draw from it. Send the same input again and you get another draw.

Temperature reshapes that distribution so the likeliest continuation wins more often. It is a dial on the draw, not a switch that removes it. The second row of the table is what that looks like in a record.

One more thing moved, and it is easy to miss. An earlier trial of the same experiment, on 2026-08-27, gave 3 of 20 at temperature zero where this one gave 4 of 20 (BOOK.md, Measured findings; that trial's file was not kept, so treat it as a note and not as evidence). The measurement of the variation varies too.

How it is measured

The method is small enough to hold in your head:

  1. Fix the prompt, the model and the number of calls before you start.
  2. Strip each answer and hash it. Two answers are "the same" only if the hashes match.
  3. Report two numbers: how many distinct hashes, and how many times the most common one appeared.
  4. Save every call as one line of JSON with its text, hash, stop reason and token usage.

Step four is what makes the first three checkable by someone else. The counts in the table above can be recomputed from the JSONL file by anyone, including you, in a few lines.

A count of distinct strings is a blunt instrument. It tells you the strings differ. It does not tell you whether the meaning differs. Lesson 3 picks that up.

How to improve it

You do not fix this, because nothing is broken. You change what you ask of your tests:

If your code does this It will Do this instead
Asserts the reply equals a recorded string Fail on a run where nothing is wrong Assert a property: the reply names a queue, parses as the type you expect, contains the required fact
Caches on the reply text Miss most of the time Cache on the request, or on a normalised result
Runs a check once and trusts it Report one draw as if it were the behaviour Run it several times and report a rate with its sample size

A lower unique count on your run is not a better result. It is a smaller sample of the same behaviour.

Read the original argument in Chapter 1 of the book, then go on to the two defaults that moved.

Every number in this lesson, and the file it came from (6)
ValueFile in the book's recordConditions
20runs/ch1-sampling-2026-09-07.txt
10 of 20runs/ch1-sampling-2026-09-07.txtclaude-opus-4-6, 2026-09-07
8runs/ch1-sampling-2026-09-07.txt
4 of 20runs/ch1-sampling-2026-09-07.txtclaude-opus-4-6, 2026-09-07
12runs/ch1-sampling-2026-09-07.txt
3 of 20BOOK.mdtrial of 2026-08-27, file not kept

Reference: Chapter 1, "The concept" and "The tradeoff".