Measured Academy Measured Academy
Français Pilot Progress 0

Module 4 · Lesson 2 of 4

Two exits and a cap

  1. What happenedThe step cap never fired on the six questions, the longest of them came within one step of it, and the harness that produced the table would have hidden a cap hit because it had no raise.
  2. Why it changedThe loop ends either when the model stops asking for tools or when your counter runs out, and only the second is under your control; a cap that returns quietly turns a truncated process into a normal-looking answer.
  3. How it is measuredSteps per request are recorded and compared with the cap, and every run is labelled with the exit it took.
  4. How to improve itWrite the cap as a number with a reason, raise when it is reached, handle every stop reason by name, and set the number from recorded step counts instead of by feel.
Slider

Replay the recorded run under a different cap

Try it. Drag the step cap down and watch which questions get cut off. Then switch between raising at the cap and returning quietly.

Where this data comes from · claude-opus-4-6, 2026-08-28, one run; claude-opus-5, 2026-11-20, the course's passing run

Read from the book's saved record. Nothing here is simulated.

  • BOOK.md Measured findings, rows "Ch 4 exercise, steps per prompt" and "Ch 4 cap headroom"
  • runs/ch4-course-agent-2026-11-20-run3.jsonl

What happened

The agent's loop was allowed at most 5 steps per question. On the six questions it never reached that limit. On the book's run the longest legitimate request used 4 (BOOK.md, Measured findings, the row on cap headroom); on the course's passing run on claude-opus-5 the headroom was 2 steps (runs/ch4-course-check-2026-11-20-run3.txt).

So a number chosen by feel had one step to spare on the older model and two on the newer one, on a set of six. Switch the replay above between the two runs.

There is a second finding in the chapter that is easy to read past. The harness that produced the results table had no raise in it. Had a question run out of steps, the script would have fallen out of the loop and printed whatever the last response held. Nothing hit the cap that day, and that is the only reason nothing was lost.

Why it changed

The loop has two ways to end, and they are not equally trustworthy.

Exit Who decides What it means
The model stops asking for tools the model It considers the task finished. Usually true.
The counter runs out you The task did not finish in the steps you allowed.

"Usually" is the word to watch. Module 1 measured what a sampling system does with the same input twenty times. A loop that ends when the model says so ends on its own most of the time, and "most of the time" is not a guarantee. The failure is not a crash. It is a request that keeps going, paying for a call per step, until something else notices.

The cap is the guarantee. What you do when it fires decides whether it helps:

At the cap The caller receives Can the caller tell?
raise an exception yes
break, then return the last text a well-formed reply from an unfinished process no

The second row is Module 1's worst case again: a failure that is silent and plausible. Try it in the replay above. Lower the cap by one, switch to "return quietly", and find the question that was cut off without reading the label.

Why write this yourself? Because nothing else will. BOOK.md records that the SDK's messages client has no loop or step-cap helper. The loop is your code, so the stop condition is too.

One more exit hides in the usual way of writing the test. A loop that asks "is the stop reason anything other than tool_use?" treats every other stop reason as a finished answer, including a reply cut off at max_tokens. On a model that thinks before answering, Module 1 showed how quickly a budget goes.

What happened when the course ran it: three attempts

The course ran this exercise on claude-opus-5 with a harness that raises at the cap, returns an answer only on end_turn, and then forces one extra run to hit the cap on purpose. It took three attempts to pass, and each failure is this lesson in small.

Attempt Budget per call What happened Check
First 1,024 tokens The reorder-policy question wrote a long answer and stopped on max_tokens, cut off in the middle of a list. A loop that asks only "was it a tool request?" would have returned that half-answer as the answer. failed: Question 6 stopped on 'max_tokens': the reply was cut off.
Second 2,048 tokens All six questions finished on end_turn; the long answer needed 1,432 output tokens. Then the forced run failed: it re-ran the question that had taken the most steps with its cap set one below, question 5, cap 3, exit end_turn. The repeat took a shorter path, so the cap never fired. failed: The forced run finished with exit 'end_turn'
Third 2,048 tokens All six finished; the long answer needed 1,355. The forced run was now the webcam-value question at a cap of one: its answer needs the stock record, so the first reply must be a tool request, and a cap of one stops the loop there whatever path the model takes. question 3 at cap 1, exit cap. passed

Sources: runs/ch4-course-agent-2026-11-20.txt, -run2.txt, -run3.txt and their check files, and the Course rows of BOOK.md.

Read the three attempts together:

  • The first was an exit nobody named. max_tokens is not end_turn. The harness kept them apart, so the check could fail the run instead of passing a truncated answer.
  • The second was our cap, not the model. "One less than it took last time" is a guess about the next run, and Module 1 measured that the next run differs. A cap that depends on the model repeating itself is not a guarantee. The fix was to choose a cap the model cannot beat.
  • The third passed because both fixes were in place: a budget that fits the longest answer, and a cap chosen by construction.

The check refused the first two, which is its job. The predictions for the second and third attempts were written by an agent working for the author after seeing the earlier attempts, so the third attempt's step predictions matching on 6 of 6 is not a blind prediction.

How it is measured

  • Steps per request, against the cap. The headroom is the cap minus the longest legitimate request: 5 minus 4 on the book's run, and 2 steps on the course's passing run.
  • The exit taken, per run. Record whether each request ended on end_turn, on the cap, or on anything else.
  • What this evidence can carry. One run, one day, six prompts. It shows the cap was tight for those six. It does not show that the same number suits your workload.
  • The measurable version, as the chapter puts it: record steps per request for a week, take a high percentile, and compare it with the number you guessed. A cap far above that percentile was never protecting anything.

How to improve it

  • Write the cap as a number and say why, in a comment, in one sentence.
  • Raise at the cap. Not a warning, not a fallback answer.
  • Name the exits you accept. Return an answer on end_turn. Treat any other stop reason as something to handle on purpose.
  • Bound the cost in your head. The cap is the most model calls one request can make. Multiply before you ship.
  • Revisit the number with data. A cap that never fires and a cap that fires weekly are both telling you something.

The argument for the raise is in Chapter 4 of the book, "The tradeoff". Next: steps you did not write.

Every number in this lesson, and the file it came from (12)
ValueFile in the book's recordConditions
5BOOK.md
4BOOK.md
2runs/ch4-course-check-2026-11-20-run3.txt
1,024BOOK.mdclaude-opus-5, 2026-11-20, the course harness, first attempt
Question 6 stopped on 'max_tokens': the reply was cut off.runs/ch4-course-check-2026-11-20.txt
2,048runs/course1-paid-runs-2026-11-20-run2-env.txt
1,432BOOK.md
question 5, cap 3, exit end_turnruns/ch4-course-agent-2026-11-20-run2.txt
The forced run finished with exit 'end_turn'runs/ch4-course-check-2026-11-20-run2.txt
1,355BOOK.md
question 3 at cap 1, exit capruns/ch4-course-check-2026-11-20-run3.txt
6 of 6runs/ch4-course-check-2026-11-20-run3.txtpredictions written after seeing two attempts, by an agent working for the author: not a blind prediction

Reference: Chapter 4, "The tradeoff".