Measured Academy Measured Academy
Français Pilot Progress 0

Module 1 · Lesson 2 of 4

Two defaults that moved

  1. What happenedBetween the book's run and the course run, the temperature setting was rejected by the current model and the model began thinking before answering.
  2. Why it changedThe SDK signature and the model's defaults are versioned things, and both changed between releases.
  3. How it is measuredA probe call records the error class and status, and the harness logs thinking tokens separately from answer tokens.
  4. How to improve itAttach the model, the SDK version and the date to every claim, log the output split, and size max_tokens for thinking plus the answer.
Side by side

The same harness, two models

Try it. Pick one of the three runs to see what was sent and what came back. Then tick the box to add temperature 0 to that request and see what the record says happened.

Where this data comes from

Read from the book's saved record. Nothing here is simulated.

  • runs/ch1-sampling-2026-09-07.jsonl
  • runs/ch1-course-sampling-2026-11-08.jsonl
  • runs/ch1-course-sampling-2026-11-08-mt1000.jsonl
  • runs/ch1-course-temperature-probe-2026-11-08.txt

What happened

The same experiment was run again for this course on 2026-11-08, on claude-opus-5. Two things that had been true in September were no longer true.

The temperature arm could not run. A call that passed temperature through extra_body came back with status 400 and the message that the setting is deprecated for the model (runs/ch1-course-temperature-probe-2026-11-08.txt).

Every answer came back empty. With the book's max_tokens of 100, 20 of 20 answers had no text at all (BOOK.md, Measured findings). Each call stopped on max_tokens. The first version of this module's check script passed that run with warnings.

Raising the budget to 1000 fixed the empty answers, and the result was different again from September:

Book run, claude-opus-4-6 Course run, claude-opus-5
max_tokens 100 1000
Distinct answers, default sampling 10 of 20 20 of 20
Temperature zero 4 of 20 distinct rejected, status 400
Source runs/ch1-sampling-2026-09-07.txt runs/ch1-course-check-2026-11-08-mt1000.txt

Why it changed

Two separate layers moved.

The first is the SDK. The Python client dropped temperature from the messages.create signature in version 1.0.0, so passing it raises a TypeError before any request is sent (BOOK.md, Measured findings). For a while the value still reached the API through extra_body. The second layer is the model: the current one refuses the setting outright.

The empty answers have a different cause. The current model thinks before it answers, by default, and thinking is paid for out of the same max_tokens budget as the answer. At the small budget the thinking used all of it. Nothing was left for text.

Neither change was announced by an exception in your own code. One arrived as a rejected request. The other arrived as a successful request with nothing in it, which is the more dangerous of the two.

How it is measured

Three small instruments, each of which you can read in the companion repo:

  • A probe. One call with the suspect setting, saving the error class, the status and the message. That is the whole of the temperature evidence.
  • The stop reason. Every record carries stop_reason. A run where every call stopped on max_tokens is telling you the budget ran out, whatever the text looks like.
  • The output split. The course harness subtracts thinking tokens from output tokens for each call. Over the course run, 4,619 of 6,636 output tokens were thinking and 2,017 were answer (runs/ch1-course-sampling-2026-11-08-mt1000.txt). BOOK.md records that as 70% thinking.

Thinking is where most of the money went. The run cost $0.17 in total (BOOK.md, Measured findings, at the Opus rates listed in COST.md), against $0.04 for both arms of the book's run on the older model (COST.md, Ch 1 row).

How to improve it

  • Date your claims. "Temperature zero narrows the spread" is true of one model on one date. Write the model, the SDK version and the date next to it. This course does that for every number, and the check script prints the model you ran.
  • Never read the first content block blindly. On a model that thinks, the first block may not be text. Find the text block.
  • Budget for thinking. Set max_tokens for thinking plus the answer, and treat a max_tokens stop as a failure until you have looked at it.
  • Fail on empty. An empty answer should fail a check, not warn. The current check script fails both an empty answer and a max_tokens stop.
  • Log the split. If you log only output tokens, you cannot say what you paid for.

The dated SDK claim is in Chapter 1 of the book under "Edge cases". Next: strings vary, meaning holds.

Every number in this lesson, and the file it came from (14)
ValueFile in the book's recordConditions
400runs/ch1-course-temperature-probe-2026-11-08.txtclaude-opus-5, 2026-11-08
100runs/ch1-sampling-2026-09-07.txt
20 of 20BOOK.mdclaude-opus-5, max_tokens 100, 2026-11-08
1000runs/ch1-course-sampling-2026-11-08-mt1000.txt
10 of 20runs/ch1-sampling-2026-09-07.txtclaude-opus-4-6, 2026-09-07
20 of 20runs/ch1-course-check-2026-11-08-mt1000.txtclaude-opus-5, max_tokens 1000, 2026-11-08
4 of 20runs/ch1-sampling-2026-09-07.txtclaude-opus-4-6, 2026-09-07
1.0.0BOOK.md
4,619runs/ch1-course-sampling-2026-11-08-mt1000.txt
6,636runs/ch1-course-sampling-2026-11-08-mt1000.txt
2,017runs/ch1-course-sampling-2026-11-08-mt1000.txt
70%BOOK.md
$0.17BOOK.mdclaude-opus-5, Opus rates in COST.md, 2026-11-08
$0.04COST.mdclaude-opus-4-6, both arms, 2026-09-07

Reference: Chapter 1, "Edge cases" (the dated SDK claim).