Measured Academy Measured Academy
Français Pilot Progress 0

Module 1 · Lesson 3 of 4

Strings vary, meaning holds

  1. What happenedThe distinct answers in the saved run differ in wording and agree on what to check first.
  2. Why it changedSampling picks between near-equivalent phrasings far more often than between different ideas, so form drifts before content does.
  3. How it is measuredRead the distinct answers side by side and mark the words that differ; count strings and count meanings as two separate numbers.
  4. How to improve itCompare on the property you need, normalise before comparing, and constrain the shape of the output where your code depends on it.
Side by side

Two answers, word by word

Try it. Choose two of the recorded answers. Compare them first as strings, then as meaning, and decide for yourself whether they would send you to check the same thing.

Where this data comes from · claude-opus-4-6 · 2026-09-07

Read from the book's saved record. Nothing here is simulated.

  • runs/ch1-sampling-2026-09-07.jsonl

What happened

Open the default-sampling answers from the saved run and read them, not their count. The file holds 10 of 20 distinct strings. Put any two of them into the diff tool above.

What differs is small: "order or contract" in one and "order or agreement" in another, "prices" against "unit prices", a clause added at the end about asking the customer for more detail. What they tell you to do first does not differ. Each one says to set the invoice's line items, quantities and prices beside the original order and look for the mismatch.

So there are two counts in that file, and only one of them was computed for you:

Count Value Who computed it
Distinct strings 10 of 20 The harness (runs/ch1-sampling-2026-09-07.txt)
Distinct meanings Yours to judge by reading You

Why it changed

When a model continues a sentence, many continuations are almost equally likely and mean the same thing. A draw lands on one of them. Choosing a different idea altogether is a much rarer draw on a prompt this plain.

That gives a useful rule of thumb for where to expect trouble first: form drifts before content does. Wording, punctuation, casing, a pair of asterisks around a word. You will meet exactly that in Module 2, where the right queue name arrives wrapped in markdown.

It is a rule of thumb from these runs, not a law. On a harder or more ambiguous prompt the meaning can split too, which is why you read the answers.

How it is measured

Counting strings is cheap and automatic, so it comes first. Counting meanings costs a human read, or a grader you trust, so decide how much of it you need:

  • Read a sample. Ten answers is enough to see whether any two would lead you to do different things. This is the second reflection question in the hands-on test.
  • Normalise, then count again. Strip case, whitespace and punctuation, hash again, and see how far the count drops. The gap between the two counts is how much of the variation was form.
  • Check a property. If what you need is "the reply is one of four queue names", test that, and report how many of the calls passed.

Each of these gives a rate over a known number of calls. None of them gives a yes or no.

How to improve it

Symptom in your system Likely cause First move
A test that passes on some runs and fails on others It compares strings Replace the assertion with a property check and run it several times
A lookup keyed on the reply raises on some replies The reply's form drifted Normalise before the lookup, then constrain the output (Module 3)
Two reviewers disagree about whether an answer is right "Right" was never written down Write the property first, then grade against it (Module 10)

The systems that look flaky here are usually comparing strings where they meant to compare meaning. The fix is in the comparison, not in the model.

The chapter shows the same failure in code, with a router whose return value is used as a dictionary key: Chapter 1 of the book, "Before / After". Next: predict, run, compare.

Every number in this lesson, and the file it came from (1)
ValueFile in the book's recordConditions
10 of 20runs/ch1-sampling-2026-09-07.txtclaude-opus-4-6, 2026-09-07

Reference: Chapter 1, "Before / After" and "Edge cases" (format drift).