Measured Academy Measured Academy
Français Pilot Progress 0

Module 4 · Lesson 3 of 4

Steps you did not write

  1. What happenedThe model split two dependent tool calls across steps, sent two independent ones together in a single response, retried a failed lookup with a different word, used the calculator for bare arithmetic and called nothing for a general question.
  2. Why it changedThe shape of each question decided the sequence, because the model can only ask for a tool whose input it already has, and it recovered from a miss by itself because nothing in the loop told it to stop.
  3. How it is measuredThe trace lists, per step, each tool request with its full input, which is the only place the retry and the parallel calls are visible.
  4. How to improve itLog inputs and not just names, make the tool accept what a question naturally contains, decide in your own code what a failed tool means, and test the description with a question that should not use the tool.
Side by side

Two questions, two shapes

Try it. Press "Next step" to advance both questions together. Tick the box to see what the model knows before each step.

Where this data comes from · claude-opus-4-6, 2026-08-28; claude-opus-5, 2026-11-20

Read from the book's saved record. Nothing here is simulated.

  • BOOK.md Measured findings
  • chapters/chapter-04-your-first-agent.md "How you verify"
  • scripts/ch03_trace.py
  • runs/ch4-course-agent-2026-11-20-run3.jsonl

Step counts and tool requests are the chapter's results table. The run's output was not kept, so each result shown is recomputed by running the tool's own code from scripts/ch03_trace.py on the recorded input.

Switch

The retry, names only and with inputs

Try it. Switch the trace between tool names only and names with their inputs and results.

Where this data comes from · claude-opus-4-6, 2026-08-28; claude-opus-5, 2026-11-20

Read from the book's saved record. Nothing here is simulated.

  • BOOK.md Measured findings, row "Model-side retry after a failed lookup"
  • scripts/ch03_trace.py
  • runs/ch4-course-agent-2026-11-20-run2.jsonl

Step counts and tool requests are the chapter's results table. The run's output was not kept, so each result shown is recomputed by running the tool's own code from scripts/ch03_trace.py on the recorded input.

What happened

Read the results table from Lesson 1 again, this time for what the model did that no line of the loop asked for. All of it is from BOOK.md, Measured findings, one run.

It split dependent calls. For the value of the webcams on hand, the model looked up the webcam, waited for the result, then asked the calculator to multiply. That took 3 steps.

It sent independent calls together. For "more keyboards or headsets", it put two lookup requests in one response. Both results went back in one turn, and the question was done in 2 steps.

It retried. Asked what it would cost to bring mice up to ten units, it looked up "mice" and got back a "no record" message. It then looked up "mouse" without being told to, got the record and ran the calculation: 4 steps. There is no retry logic in the loop.

It used a tool for plain arithmetic. A bare multiplication with no inventory in it went to the calculator, 2 steps, instead of being answered directly.

It used nothing. The reorder-policy question was answered in 1 step with no tool call.

The course ran the same exercise three times on claude-opus-5 (Lesson 2 says why three). Two of these behaviours moved:

Behaviour Book run, claude-opus-4-6 Course attempts on claude-opus-5: first, second, third
Steps for the six questions 2, 2, 3, 2, 4, 1 2, 1, 3, 2, 3, 1; 2, 1, 3, 2, 4, 1; 2, 1, 3, 2, 3, 1
Bare multiplication went to the calculator answered with no tool, all three times
Mice question missed on "mice", retried with "mouse" looked up "mouse" directly; missed and retried; looked up "mouse" directly

Sources: BOOK.md; runs/ch4-course-agent-2026-11-20.txt, -run2.txt and runs/ch4-course-check-2026-11-20-run3.txt. The retry the book calls "the model's, not yours" happened in one course attempt out of three, on the same code. Switch the retry viewer below to the course run to see that attempt's inputs and results.

Why it changed

Compare the two middle cases in the first viewer above.

Webcam value Keyboards or headsets
What the second call needs the unit price, which only the first call's result holds nothing from the other call
So the calls arrive one per response together in one response
Steps 3 2

That difference is not a setting you turned on. The model cannot ask for a multiplication before it has the numbers to multiply. It is the shape of the question, showing up as the shape of the trace.

The retry has a more awkward explanation. The lookup matched product names after stripping a trailing "s", which handles "headsets" and "webcams" and does nothing for "mice". The author had not noticed. The model covered for it, at the cost of a step.

Hold both halves of that. The behaviour that rescued a bug in the matching rule is the same behaviour that will answer around a tool that is truly broken. From outside they look identical: one failed result, then a confident reply.

The last two cases are about the tool descriptions. The description is the main control you have over when the model reaches for a tool, and the tool's name is read too. The chapter says how far that control goes is measurable, and Module 10 measures it on this same agent.

How it is measured

  • Names and inputs, per step. Switch the second viewer between its two settings. With names only, the mice question shows the lookup twice and you cannot tell a retry from a duplicate. With inputs, the miss and the changed word are plain.
  • Results too, for a miss. The "no record" text is what explains the extra step.
  • One question that should use no tool, and one that tests the other direction. The policy question and the bare multiplication are a matched pair for the calculator's description.
  • A count of steps per question, which is what makes a retry cost visible at all.

How to improve it

  • Log the full input of every request. Names are cheaper and they are not enough.
  • Make the tool accept what questions contain. The chapter's first lookup took a product code only, every question named a product, and the second tool was never reached. The signature is part of the prompt surface.
  • Fix the miss in the tool, not by relying on the retry. A retry you did not write is a step you pay for and cannot count on.
  • Decide what a failed tool means, above the loop. If a failure means the request cannot be answered, your code must say so. The model will not.
  • When a tool is used wrongly, edit the description first, then look at the name.

The table and the reading of it are in Chapter 4 of the book, "Hands-on exercise". Next: where the loop breaks.

Every number in this lesson, and the file it came from (9)
ValueFile in the book's recordConditions
3BOOK.md
2BOOK.md
4BOOK.md
2BOOK.md
1BOOK.md
2, 2, 3, 2, 4, 1BOOK.mdclaude-opus-4-6, SDK 1.2.0, 2026-08-28, one run, MAX_STEPS = 5; the output was not kept
2, 1, 3, 2, 3, 1runs/ch4-course-agent-2026-11-20.txtclaude-opus-5, 2026-11-20, first attempt, max_tokens 1024
2, 1, 3, 2, 4, 1runs/ch4-course-agent-2026-11-20-run2.txtclaude-opus-5, 2026-11-20, second attempt
2, 1, 3, 2, 3, 1runs/ch4-course-check-2026-11-20-run3.txtclaude-opus-5, 2026-11-20, third attempt, check PASS

Reference: Chapter 4, "Hands-on exercise" (the results table and the three things to read out of it) and "Edge cases".