Module 1 · Lesson 2 of 4
Two defaults that moved
- What happenedBetween the book's run and the course run, the temperature setting was rejected by the current model and the model began thinking before answering.
- Why it changedThe SDK signature and the model's defaults are versioned things, and both changed between releases.
- How it is measuredA probe call records the error class and status, and the harness logs thinking tokens separately from answer tokens.
- How to improve itAttach the model, the SDK version and the date to every claim, log the output split, and size max_tokens for thinking plus the answer.
The same harness, two models
Try it. Pick one of the three runs to see what was sent and what came back. Then tick the box to add temperature 0 to that request and see what the record says happened.
Where this data comes from
Read from the book's saved record. Nothing here is simulated.
runs/ch1-sampling-2026-09-07.jsonlruns/ch1-course-sampling-2026-11-08.jsonlruns/ch1-course-sampling-2026-11-08-mt1000.jsonlruns/ch1-course-temperature-probe-2026-11-08.txt
What happened
The same experiment was run again for this course on 2026-11-08, on claude-opus-5. Two things that had been true in September were no longer true.
The temperature arm could not run. A call that passed temperature through extra_body came back with status 400 and the message that the setting is deprecated for the model (runs/ch1-course-temperature-probe-2026-11-08.txt).
Every answer came back empty. With the book's max_tokens of 100, 20 of 20 answers had no text at all (BOOK.md, Measured findings). Each call stopped on max_tokens. The first version of this module's check script passed that run with warnings.
Raising the budget to 1000 fixed the empty answers, and the result was different again from September:
Book run, claude-opus-4-6 |
Course run, claude-opus-5 |
|
|---|---|---|
max_tokens |
100 | 1000 |
| Distinct answers, default sampling | 10 of 20 | 20 of 20 |
| Temperature zero | 4 of 20 distinct | rejected, status 400 |
| Source | runs/ch1-sampling-2026-09-07.txt |
runs/ch1-course-check-2026-11-08-mt1000.txt |
Why it changed
Two separate layers moved.
The first is the SDK. The Python client dropped temperature from the messages.create signature in version 1.0.0, so passing it raises a TypeError before any request is sent (BOOK.md, Measured findings). For a while the value still reached the API through extra_body. The second layer is the model: the current one refuses the setting outright.
The empty answers have a different cause. The current model thinks before it answers, by default, and thinking is paid for out of the same max_tokens budget as the answer. At the small budget the thinking used all of it. Nothing was left for text.
Neither change was announced by an exception in your own code. One arrived as a rejected request. The other arrived as a successful request with nothing in it, which is the more dangerous of the two.
How it is measured
Three small instruments, each of which you can read in the companion repo:
- A probe. One call with the suspect setting, saving the error class, the status and the message. That is the whole of the temperature evidence.
- The stop reason. Every record carries
stop_reason. A run where every call stopped onmax_tokensis telling you the budget ran out, whatever the text looks like. - The output split. The course harness subtracts thinking tokens from output tokens for each call. Over the course run, 4,619 of 6,636 output tokens were thinking and 2,017 were answer (
runs/ch1-course-sampling-2026-11-08-mt1000.txt).BOOK.mdrecords that as 70% thinking.
Thinking is where most of the money went. The run cost $0.17 in total (BOOK.md, Measured findings, at the Opus rates listed in COST.md), against $0.04 for both arms of the book's run on the older model (COST.md, Ch 1 row).
How to improve it
- Date your claims. "Temperature zero narrows the spread" is true of one model on one date. Write the model, the SDK version and the date next to it. This course does that for every number, and the check script prints the model you ran.
- Never read the first content block blindly. On a model that thinks, the first block may not be text. Find the text block.
- Budget for thinking. Set
max_tokensfor thinking plus the answer, and treat amax_tokensstop as a failure until you have looked at it. - Fail on empty. An empty answer should fail a check, not warn. The current check script fails both an empty answer and a
max_tokensstop. - Log the split. If you log only output tokens, you cannot say what you paid for.
The dated SDK claim is in Chapter 1 of the book under "Edge cases". Next: strings vary, meaning holds.
Every number in this lesson, and the file it came from (14)
| Value | File in the book's record | Conditions |
|---|---|---|
| 400 | runs/ch1-course-temperature-probe-2026-11-08.txt | claude-opus-5, 2026-11-08 |
| 100 | runs/ch1-sampling-2026-09-07.txt | |
| 20 of 20 | BOOK.md | claude-opus-5, max_tokens 100, 2026-11-08 |
| 1000 | runs/ch1-course-sampling-2026-11-08-mt1000.txt | |
| 10 of 20 | runs/ch1-sampling-2026-09-07.txt | claude-opus-4-6, 2026-09-07 |
| 20 of 20 | runs/ch1-course-check-2026-11-08-mt1000.txt | claude-opus-5, max_tokens 1000, 2026-11-08 |
| 4 of 20 | runs/ch1-sampling-2026-09-07.txt | claude-opus-4-6, 2026-09-07 |
| 1.0.0 | BOOK.md | |
| 4,619 | runs/ch1-course-sampling-2026-11-08-mt1000.txt | |
| 6,636 | runs/ch1-course-sampling-2026-11-08-mt1000.txt | |
| 2,017 | runs/ch1-course-sampling-2026-11-08-mt1000.txt | |
| 70% | BOOK.md | |
| $0.17 | BOOK.md | claude-opus-5, Opus rates in COST.md, 2026-11-08 |
| $0.04 | COST.md | claude-opus-4-6, both arms, 2026-09-07 |
Reference: Chapter 1, "Edge cases" (the dated SDK claim).