Module 3 · Lesson 1 of 4
The run that did not break
- What happenedThe untyped router returned a clean queue name on every ticket, so the failure this module was planned around never showed up, and the typed router chose the same queue on every ticket.
- Why it changedNothing changed in the routing; the bad shapes seen in Module 2 came from the three-call chain, not from the one-call prompt, and typing changes what the return is allowed to be, not what the model decides.
- How it is measuredEach untyped return is tested for exact membership in the list of queues with no cleanup, each typed call is counted as returned or raised, and the two are compared ticket by ticket.
- How to improve itType the return for the guarantee, report a clean run as clean, and never turn ten samples into a rate.
Ten tickets, two return types
Try it. Pick a ticket, or look at all ten. Then tick "What could this have been?" to see what each return type permits.
Where this data comes from · claude-opus-4-6, 2026-10-14; claude-opus-5, 2026-11-20
Read from the book's saved record. Nothing here is simulated.
runs/ch3-router-typed-2026-10-14.jsonlscripts/ch3_route_typed.pyruns/ch2-collapse-2026-09-07.jsonlruns/ch3-course-route-typed-2026-11-20.jsonl
What happened
Module 2 ended on a promise: replace "reply with one word" with a return that cannot be a queue name in bold. This module was planned around showing the untyped router break first.
It did not break. Chapter 1's router, unchanged, was run over the ten tickets from Module 2 on claude-opus-4-6. Then the same router was typed and run again.
Book run, claude-opus-4-6 |
Course run, claude-opus-5 |
|
|---|---|---|
| Untyped return usable as a dictionary key | 10 of 10 | 10 of 10 |
| Typed calls that returned a queue | 10 of 10 | 10 of 10 |
| Typed calls that raised | 0 of 10 | 0 of 10 |
| Tickets where both versions chose the same queue | 10 of 10 | 9 of 10 |
Sources: runs/ch3-router-typed-2026-10-14.txt and BOOK.md Measured findings for the book's run, first made on 2026-09-14 and repeated on 2026-10-14 with the same results; runs/ch3-course-route-typed-2026-11-20.txt for the course's run of the same exercise on a current model, made with the course harness.
So there is no dramatic before and after here. The honest headline is: both versions worked on both models. On the older model they agreed on every ticket. On the newer one they split on one: Ticket 10, the team that cannot see a shared folder, went to auth untyped and to bug typed. Switch the comparison above to the course run to see it.
Why it changed
It is worth asking why the break did not appear, because Module 2 showed plenty of broken returns.
Those came from somewhere else. In Module 2's run, 6 of 10 of the chain's returns were not a queue name, and 0 of 10 of the single call's were (runs/ch2-collapse-2026-09-07.txt). The decorated returns were produced by a final call that had been handed a paragraph of the model's own formatted prose. A short prompt that asks for one word, on this model, did not produce them.
What typing changed is not visible in the scores at all. Compare the two signatures:
| Untyped | Typed | |
|---|---|---|
| The signature promises | some string | one of four named strings |
| If the model writes a sentence | the caller receives the sentence | the call raises before anything is returned |
| Where you find out | at the dictionary lookup, or later | at the call |
Open the comparison above and turn on "what could this have been?". The recorded values stay the same. What differs is the set of values each column was allowed to hold.
The split on Ticket 10 in the course run does not contradict this. The two routers are not the same request: the typed one drops "Reply with one word" and carries a schema, and each call is a draw from a distribution, as Module 1 measured. One disagreement in ten tells you the two versions are not interchangeable on that ticket. It does not tell you typing made the routing worse, and both answers differ from the key anyway (Lesson 4).
How it is measured
Three counts, each of which you will reproduce in the hands-on test:
- Clean without cleanup. The untyped return, with surrounding whitespace removed and nothing else done to it, either is one of the four queue names or is not. No lowercasing, no stripping of punctuation. That is what a caller indexing a dictionary experiences.
- Returned or raised. A typed call has two outcomes that matter here: it returned a queue, or it raised. A raise is counted, not hidden.
- Agreement. For each ticket, did the two versions pick the same queue? On the book's run this count said typing changed no decision. On the course run it found the one ticket where the two versions differ, which is exactly what it is for.
One more thing was measured by accident: the plan. The chapter's first draft claimed the untyped router would fail on these tickets. The run said otherwise, and the claim was withdrawn (BOOK.md, the errata entry queued from the Ch 3 run). A prediction you can be wrong about is the reason the test at the end of this module makes you write yours down first.
How to improve it
- Type the return for the guarantee, not for the demo. You are replacing "trust me" with "this, or an exception". That is worth having even on a day when nothing breaks.
- Report a clean run as clean. It is tempting to go looking for a prompt that fails so the fix looks necessary. Do not. Say what the run showed.
- Do not turn ten into a rate. Ten clean returns on one model on one date is ten clean returns. It is not "this prompt is reliable".
- Keep the comparison paired. Same tickets, same model, same wording apart from the instruction the type replaces.
The two versions of the router are printed in Chapter 3 of the book, "Before / After". Next: what the schema costs.
Every number in this lesson, and the file it came from (10)
| Value | File in the book's record | Conditions |
|---|---|---|
| 10 of 10 | runs/ch3-router-typed-2026-10-14.txt | claude-opus-4-6, Python 3.12.3, 2026-10-14, one run |
| 10 of 10 | runs/ch3-course-route-typed-2026-11-20.txt | claude-opus-5, 2026-11-20, one run |
| 10 of 10 | runs/ch3-router-typed-2026-10-14.txt | claude-opus-4-6, Python 3.12.3, 2026-10-14, one run |
| 10 of 10 | runs/ch3-course-route-typed-2026-11-20.txt | claude-opus-5, 2026-11-20, one run |
| 0 of 10 | runs/ch3-router-typed-2026-10-14.txt | |
| 0 of 10 | runs/ch3-course-route-typed-2026-11-20.txt | |
| 10 of 10 | BOOK.md | |
| 9 of 10 | runs/ch3-course-route-typed-2026-11-20.txt | claude-opus-5, 2026-11-20, one run |
| 6 of 10 | runs/ch2-collapse-2026-09-07.txt | |
| 0 of 10 | runs/ch2-collapse-2026-09-07.txt |
Reference: Chapter 3, "Opening" and "Before / After".