Module 3 · Lesson 4 of 4
Typed is not correct
- What happenedA typed call returned a value that passed every check and was not the value in the prompt, and both routers agreed with each other while disagreeing with the answer key on one ticket.
- Why it changedA pattern constraint was moved out of the schema the model receives and into a description, so the model matched the format instead of extracting the value, and on the disputed ticket the difference lay in the prompt, which typing does not touch.
- How it is measuredThe schema recorded as sent is compared with the schema as written, the returned value is compared with the prompt, and typed returns are scored against an answer key written beforehand.
- How to improve itRead the schema that is actually sent, prefer constraints that survive the trip such as a fixed list of values, keep scoring typed output against a key, and fix decisions in the prompt.
The rule you wrote and the rule that arrived
Try it. Pick a part of the vendor field to compare as written and as sent. Then show the reply the call returned.
Where this data comes from · claude-opus-5 · 2026-09-02, one call
Read from the book's saved record. Nothing here is simulated.
runs/ch7-parse-output-format-2026-09-02.txt section C2chapters/chapter-03-structured-output.md "Edge cases"
Three routers and a key on one ticket
Try it. Start on Ticket 10, then try others. Tick the box to read each router's prompt and find what the one that matched the key was told.
Where this data comes from · claude-opus-4-6, 2026-10-14 and 2026-09-07; claude-opus-5, 2026-11-20
Read from the book's saved record. Nothing here is simulated.
runs/ch3-router-typed-2026-10-14.jsonlruns/ch2-collapse-2026-09-07.jsonlscripts/ch3_route_typed.pyruns/ch3-course-route-typed-2026-11-20.jsonlruns/ch2-course-collapse-2026-11-20.jsonl
What happened
Two results from the record, each of which looks fine to a type checker.
A valid value that was wrong. An invoice model was given a vendor field with a pattern: the letters ACME, a hyphen, four digits. The prompt named the vendor as ACME Corp. The typed call returned ACME-0000 and raised nothing (runs/ch7-parse-output-format-2026-09-02.txt, section C2, claude-opus-5, one call). The value satisfies the pattern. It is not the vendor in the prompt.
Agreement that was not accuracy. In Lesson 1 the untyped and typed routers chose the same queue on every ticket. Score them against the answer key from Module 2 and each gets 9 of 10 (BOOK.md, Measured findings). Both sent Ticket 10, the one about a team that cannot see a shared folder, to auth. The key says general. Module 2's own single call scored 10 of 10 against that key (runs/ch2-collapse-2026-09-07.txt).
| Router | Return type | Ticket 10, book runs (claude-opus-4-6) |
Ticket 10, course runs (claude-opus-5) |
|---|---|---|---|
| Chapter 1's prompt, untyped | string | auth |
auth |
| Chapter 1's prompt, typed | one of four values | auth |
bug |
| Module 2's single call, untyped | string | general |
general |
On the book's runs the two routers each scored 9 of 10 against the key and Module 2's single call 10 of 10. On the course runs (runs/ch3-course-route-typed-2026-11-20.txt, runs/ch2-course-collapse-2026-11-20.txt) the picture is the same with one twist: the typed router left auth for bug, which is no nearer the key.
On both models the router that matched the key on that ticket was an untyped one.
Why it changed
The pattern never reached the model. Not everything you can declare in Pydantic can be expressed in the schema the API accepts. The SDK removed the pattern from the field and appended its text to the field's description, as a hint. Pydantic still enforced the pattern locally when the reply came back. So the model was shown a format hint, produced something in that format, and the local check passed it. That is the chapter's reading of one call, which was not repeated: an observation, not a rate.
A fixed list of allowed values is different. The router's four queue names did travel in the schema, which is why Lesson 1's guarantee holds for them.
The ticket was decided by the prompt. BOOK.md records why the routers split on Ticket 10: Module 2's single-call prompt says what each queue owns, and Chapter 1's prompt only lists the names. Typing was present in one row that disagrees with the key and absent in the row that agrees, on both models. Open the second explorer above, show the prompts, and read them side by side.
Typing settles what a return can be. It has no opinion on whether it is right.
How it is measured
- Read the schema that was sent, not the class you wrote. The probe prints both. The difference between them is the list of rules the model was never given.
- Compare the value with the source. A validator can say a vendor code is well formed. Only a comparison with the prompt, or with a key, can say it is the right vendor.
- Keep the answer key. Typed output is easier to score, because a field compares cleanly where a string needs normalising. It still has to be scored. A key written before the run, as in Module 2, is what turned "both versions agree" into "both versions miss the same ticket".
How to improve it
- Prefer constraints that survive the trip. A fixed set of values does. For a format rule, say it in the prompt as well and keep the local check.
- Fix decisions in the prompt. If a ticket goes to the wrong queue, define the queues. Changing the return type will not move it.
- Do not treat "it validated" as "it passed". Validation is the first gate. Module 10 builds the second one.
- Know when to leave it out. The chapter gives three cases: output a person reads, a single field that is already a string, and validity rules that change so often the schema becomes its own maintenance job.
Read the three cases in Chapter 3 of the book, "When NOT to use this", then take the hands-on test.
Every number in this lesson, and the file it came from (3)
| Value | File in the book's record | Conditions |
|---|---|---|
| ACME-0000 | runs/ch7-parse-output-format-2026-09-02.txt | claude-opus-5, 2026-09-02, one call, not repeated |
| 9 of 10 | BOOK.md | |
| 10 of 10 | runs/ch2-collapse-2026-09-07.txt | claude-opus-4-6, 2026-09-07, the author's key |
Reference: Chapter 3, "The tradeoff", "Edge cases" and "When NOT to use this".