Measured Academy Measured Academy
Français Pilot Progress 0

Module 2 · Lesson 2 of 4

The chain that lost to one call

  1. What happenedA three-call routing chain matched the answer key on fewer tickets than a single call, while making three times as many calls.
  2. Why it changedEach extra call passed free text to the next one, and the final step returned queue names wrapped in markdown that an exact comparison rejects.
  3. How it is measuredBoth versions run over the same tickets against a key written before the run, scored by exact match, with calls and tokens logged per stage.
  4. How to improve itStart with one call that names each option and asks for one word; add a step only when a measured failure calls for it.
Side by side

Ticket by ticket, chain against single call

Try it. Pick a ticket, or look at all ten, and switch the scoring between exact match and letters only.

Where this data comes from · claude-opus-4-6, 2026-09-07; claude-opus-5, 2026-11-20

Read from the book's saved record. Nothing here is simulated.

  • runs/ch2-collapse-2026-09-07.jsonl
  • runs/ch2-collapse-2026-09-07.txt
  • runs/ch2-course-collapse-2026-11-20.jsonl
  • runs/ch2-course-collapse-2026-11-20.txt

What happened

The same 10 support tickets were routed two ways, to one of four queues, on claude-opus-4-6 on 2026-09-07.

The chain made three calls per ticket: summarise the ticket, list the queues that could own it, pick one. The single call made one: here are the four queues and what each owns, reply with one word.

Chain, book run Single call, book run Chain, course run Single call, course run
Model claude-opus-4-6 claude-opus-4-6 claude-opus-5 claude-opus-5
Calls 30 10 30 10
Exact match with the key 4 of 10 10 of 10 8 of 10 10 of 10
Returns that were not a queue name 6 of 10 0 of 10 1 of 10 0 of 10

Sources: runs/ch2-collapse-2026-09-07.txt for the book's run, and runs/ch2-course-collapse-2026-11-20.txt for the course's run of the same exercise on a current model, made on 2026-11-20 with the course harness. On both models the design with more steps did worse on the score and made three times the calls. Use the "Recorded run" switch above the comparison to move between the two.

BOOK.md records two trials that day, with the chain at 5 of 10 on the other one. Only the trial above has a file in runs/, so it is the one this lesson shows. The chain's score moved between trials; the single call's did not.

Why it changed

Step through one ticket in the comparison above and look at what the chain's last call returned. On most of the tickets it missed, the queue was right and the format was not: the name came back in bold, with asterisks around it.

That happened because of what each call was handed. The single call got the ticket and a tight instruction. The chain's last call got a paragraph of the model's own prose listing candidate queues, formatted the way models like to format lists, and echoed that style back.

Where the tokens went tells the same story, more loudly on the newer model:

Stage, output tokens over 10 calls Book run, claude-opus-4-6 Course run, claude-opus-5
Chain, summarise 219 1,753
Chain, list candidates 488 6,969
Chain, pick 52 141
Single call 40 195

The middle step wrote the most text on both models, and all of it became input for the step that then had to produce one word. On the current model, which thinks before it answers, 3,369 of the run's output tokens were thinking (runs/ch2-course-collapse-2026-11-20.txt).

What changed between the models is the format, not the shape. The newer model put a queue name in decoration only once, so the chain's exact score rose. It still lost to the single call, and the ticket it got wrong was a decision, not a format: Lesson 3 takes that apart.

How it is measured

Four things make this comparison fair, and you will reproduce each one:

  • Same tickets for both versions.
  • A key written first. The answer key is a judgement, one queue per ticket with a reason, written before the single call was ever run. Yours may differ on a ticket or two, and that is expected.
  • Exact match. The return, stripped and lowercased, equals the key or it does not. This is what a caller using the return as a dictionary key would experience.
  • Usage per call. Every call logs its stage, input tokens and output tokens.
Whole run Book run, claude-opus-4-6 Course run, claude-opus-5
Calls 40 40
Input tokens 2,599 8,142
Output tokens 799 9,058
Cost $0.03 (COST.md) $0.27 (BOOK.md, Measured findings)

The same forty calls cost several times more on the newer model, almost all of it output the chain's middle step wrote.

One sample of each version is not a rate. The lesson reports what one recorded trial on each model did.

How to improve it

  • Write the single call first. Name each option, say what it owns, ask for one word. Measure it before adding anything.
  • Add a step only for a measured reason. "It might help to summarise first" is a guess. A ticket the single call gets wrong, twice, is a reason.
  • If you keep a chain, tighten the hand-offs. Each intermediate call should return the smallest thing the next one needs.
  • Constrain the final shape. Module 3 replaces "reply with one word" with a typed return that cannot be a queue name in bold.

The chain and the tickets are printed in Chapter 2 of the book, "Hands-on exercise". Next: wrong answer or wrong format.

Every number in this lesson, and the file it came from (28)
ValueFile in the book's recordConditions
10runs/ch2-collapse-2026-09-07.txt
30runs/ch2-collapse-2026-09-07.txt
10runs/ch2-collapse-2026-09-07.txt
4 of 10runs/ch2-collapse-2026-09-07.txtclaude-opus-4-6, 2026-09-07, the author's key
10 of 10runs/ch2-collapse-2026-09-07.txtclaude-opus-4-6, 2026-09-07, the author's key
8 of 10runs/ch2-course-collapse-2026-11-20.txtclaude-opus-5, 2026-11-20, one run, predictions by an agent working for the author
10 of 10runs/ch2-course-collapse-2026-11-20.txtclaude-opus-5, 2026-11-20, one run, predictions by an agent working for the author
6 of 10runs/ch2-collapse-2026-09-07.txt
0 of 10runs/ch2-collapse-2026-09-07.txt
1 of 10runs/ch2-course-collapse-2026-11-20.txt
0 of 10runs/ch2-course-collapse-2026-11-20.txt
5 of 10BOOK.mdthe other of two trials on 2026-09-07; only one trial has a file in runs/
219runs/ch2-collapse-2026-09-07.txt
1,753runs/ch2-course-collapse-2026-11-20.txt
488runs/ch2-collapse-2026-09-07.txt
6,969runs/ch2-course-collapse-2026-11-20.txt
52runs/ch2-collapse-2026-09-07.txt
141runs/ch2-course-collapse-2026-11-20.txt
40runs/ch2-collapse-2026-09-07.txt
195runs/ch2-course-collapse-2026-11-20.txt
3,369runs/ch2-course-collapse-2026-11-20.txt
40runs/ch2-collapse-2026-09-07.txt
2,599runs/ch2-collapse-2026-09-07.txt
8,142runs/ch2-course-collapse-2026-11-20.txt
799runs/ch2-collapse-2026-09-07.txt
9,058runs/ch2-course-collapse-2026-11-20.txt
$0.03COST.mdclaude-opus-4-6, Opus rates in COST.md
$0.27BOOK.mdclaude-opus-5, 2026-11-20, Opus rates in COST.md

Reference: Chapter 2, "Before / After" and "Hands-on exercise - collapse the chain".