Module 2 · Lesson 3 of 4
Wrong answer or wrong format
- What happenedScoring the same returns again after stripping non-letters raised the chain's score sharply and left one ticket still wrong.
- Why it changedMost misses were the right queue in the wrong format; one was a different queue, a routing decision the chain got wrong.
- How it is measuredTwo scores on the same returns, exact and letters-only, plus a count of returns that are not one of the allowed values.
- How to improve itReport both scores, fix format at the boundary, and spend prompt effort only on the misses that survive normalising.
One ticket through the chain
Try it. Pick a ticket and step through the chain's three calls to see what each one was handed and what it passed on.
Where this data comes from · claude-opus-4-6, 2026-09-07; claude-opus-5, 2026-11-20
Read from the book's saved record. Nothing here is simulated.
runs/ch2-collapse-2026-09-07.jsonlruns/ch2-course-collapse-2026-11-20.jsonl
What happened
Take the chain's returns from Lesson 2 and score them a second time, after removing everything that is not a letter.
| Scoring | Chain, book run | Single call, book run | Chain, course run | Single call, course run |
|---|---|---|---|---|
| Exact match | 4 of 10 | 10 of 10 | 8 of 10 | 10 of 10 |
| Letters only | 9 of 10 | 10 of 10 | 9 of 10 | 10 of 10 |
Sources: runs/ch2-collapse-2026-09-07.txt (claude-opus-4-6) and runs/ch2-course-collapse-2026-11-20.txt (claude-opus-5). On the book's run most of the chain's misses disappear and one stays: on the ticket about an export button that spins and downloads nothing, the chain chose a different queue from the key, and no amount of stripping asterisks turns that into the right one.
The current model shows the same split with different tickets. Its one format miss was Ticket 3, the right queue wrapped in backticks. Its decision miss was Ticket 10, the team that cannot see a shared folder: the chain said auth, the key says general, and the single call matched the key.
So the single headline score from Lesson 2 was hiding two different problems.
Why it changed
Format misses. The queue was right and it arrived decorated. This is Module 1's rule of thumb, form drifts before content, showing up one step downstream. The cure belongs at the boundary of your code, not in the prompt.
A decision miss. Step through that ticket in the viewer above. The chain's first call summarised the ticket, the second proposed candidate queues from the summary, and the third chose among the candidates. Each call saw less of the original ticket than the one before. The single call saw the ticket itself and a definition of each queue.
These need different fixes, which is why they need different numbers. Notice that the newer model fixed most of the format problem on its own and none of the decision problem. A model upgrade moved the first number, not the second.
How it is measured
Three measurements on one set of returns:
- Exact score. What a caller experiences today.
- Letters-only score. What the model decided, with format taken out.
- Returns outside the allowed set. In the book's run, 6 of 10 of the chain's and 0 of 10 of the single call's; in the course run on the current model, 1 of 10 and 0 of 10. Each of those raises a
KeyErrorin a caller that uses the return to look up a handler.
The gap between the first two scores is your format problem. What remains under the second is your decision problem. Report both, with the number of tickets, every time.
A caution about the second score: stripping non-letters is a diagnostic, not a fix to ship. It would also "repair" a return such as a sentence that happens to end in a queue name, by accident or not at all.
How to improve it
| What the two scores show | Where the problem is | First move |
|---|---|---|
| Exact low, letters-only high | Format | Constrain the return type (Module 3); until then, validate against the allowed set and fail loudly |
| Both low | Decision | Look at what the deciding call was actually given; shorten the chain or define the options |
| Both high | Neither, on this sample | Add harder tickets before you trust it |
And for the arguable tickets: when your key and the model disagree, read the ticket again before you blame either. The key is a judgement. The book's author marked one ticket as the one a reader was most likely to dispute.
The scorer is printed in Chapter 2 of the book. Next: add it after it breaks twice.
Every number in this lesson, and the file it came from (12)
| Value | File in the book's record | Conditions |
|---|---|---|
| 4 of 10 | runs/ch2-collapse-2026-09-07.txt | claude-opus-4-6, 2026-09-07, the author's key |
| 10 of 10 | runs/ch2-collapse-2026-09-07.txt | claude-opus-4-6, 2026-09-07, the author's key |
| 8 of 10 | runs/ch2-course-collapse-2026-11-20.txt | claude-opus-5, 2026-11-20, one run, predictions by an agent working for the author |
| 10 of 10 | runs/ch2-course-collapse-2026-11-20.txt | claude-opus-5, 2026-11-20, one run, predictions by an agent working for the author |
| 9 of 10 | runs/ch2-collapse-2026-09-07.txt | |
| 10 of 10 | runs/ch2-collapse-2026-09-07.txt | |
| 9 of 10 | runs/ch2-course-collapse-2026-11-20.txt | |
| 10 of 10 | runs/ch2-course-collapse-2026-11-20.txt | |
| 6 of 10 | runs/ch2-collapse-2026-09-07.txt | |
| 0 of 10 | runs/ch2-collapse-2026-09-07.txt | |
| 1 of 10 | runs/ch2-course-collapse-2026-11-20.txt | |
| 0 of 10 | runs/ch2-course-collapse-2026-11-20.txt |
Reference: Chapter 2, "Edge cases" and the exercise's results table.