Measured Academy Measured Academy
Français Pilot Progress 0

Module 3 · Lesson 3 of 4

Three ways a typed call ends badly

  1. What happenedA typed call that was cut off raised a validation error and returned nothing, a reply of the wrong shape raised too, and a reply with no text block returned None without raising.
  2. Why it changedThe SDK validates the reply text against your type after the call, so broken or mismatched JSON fails loudly, but when there is no text to validate it has nothing to reject and hands back None.
  3. How it is measuredOne real call is forced to truncate with a tiny max_tokens and its exception is captured, the same request is sent untyped to see the raw bytes, and the parser is driven locally with hand-made replies.
  4. How to improve itCatch the validation error and check for None as two separate branches, size max_tokens for the whole reply, and write every outcome, including a raise, into the run file.
Decision path

Where each reply ends up

Try it. Choose one of the six recorded replies, including the current model's cut-off one, and see which of the three exits it takes, and which guard would catch it.

Where this data comes from · 2026-10-14, 2026-11-20 and 2026-09-02

Read from the book's saved record. Nothing here is simulated.

  • runs/ch3-router-typed-2026-10-14.txt section C
  • runs/ch3-course-route-typed-2026-11-20.jsonl
  • runs/ch7-parse-output-format-2026-09-02.txt sections C3 and C4
Side by side

The same cut-off reply, typed and untyped

Try it. Switch between what your code receives and what the model wrote, for the same cut-off request sent typed and untyped.

Where this data comes from · claude-opus-5 · 2026-09-02

Read from the book's saved record. Nothing here is simulated.

  • runs/ch7-parse-output-format-2026-09-02.txt sections C1 and C1b

What happened

The guarantee from Lesson 1 was "one of four values, or an exception". This lesson tests the second half on purpose, and finds a third outcome the slogan leaves out.

Cut off: it raised. The typed router was called once with max_tokens=3, too few to finish the JSON. Nothing was returned. The call raised a Pydantic ValidationError with this message (runs/ch3-router-typed-2026-10-14.txt, section C):

Invalid JSON: EOF while parsing a string at line 1 column 10

The reply had got as far as {"queue":" and stopped. On the course run on claude-opus-5, the same call cut off even earlier and raised the same exception class, with Invalid JSON: EOF while parsing a string at line 1 column 5 on the fragment {"que (runs/ch3-course-route-typed-2026-11-20.txt). This time the outcome is a row in the run file.

Wrong shape: it raised. Driving the parser locally with valid JSON that used the wrong field name raised as well, reporting 2 validation errors, one for each required field that was missing. A reply that was a plain sentence raised too (runs/ch7-parse-output-format-2026-09-02.txt, section C3).

No text at all: it returned None. Driving the parser with a response that had no content blocks produced no exception. The recorded line is parsed_output = None -- no raise (same file, section C4).

Reply Outcome Caught by except ValidationError Caught by a None check
Complete and valid your type not needed not needed
Cut off mid-JSON raises yes no
Valid JSON, wrong shape raises yes no
Not JSON raises yes no
No text block None, no error no yes

Why it changed

The check happens after the call, on your machine. The SDK takes the reply's text and asks Pydantic to build your type from it. Half a JSON object cannot be built into anything, so Pydantic refuses. So does JSON with the wrong fields.

The third case slips through for a dull reason: with no text block there is no text to validate. Nothing is malformed. The SDK returns the message with its parsed field empty.

Now compare the same cut-off reply through a typed and an untyped call, using the second comparison above. On an invoice schema with max_tokens=16, the typed call raised Invalid JSON: EOF while parsing a value at line 1 column 30. The same request sent untyped came back as an ordinary response: stop reason max_tokens, 16 output tokens, and a string holding the first part of a JSON object (runs/ch7-parse-output-format-2026-09-02.txt, sections C1 and C1b, claude-opus-5). These were two calls with the same small budget, and both replies stopped partway through the object. Only one path let its fragment reach your code looking like a result.

This connects to Module 1. There, a model that thinks before answering spent a small budget entirely on thinking, and 20 of 20 answers came back with no text (BOOK.md, Measured findings). A reply with no text is exactly the input that produces None in the local probe. On the one course run so far, a typed call on the thinking model with three tokens of budget raised instead of returning None. One call is not a rule, which is why the hands-on test asks you to predict and record what yours does.

How it is measured

  • Force the failure. One real call with a budget too small to finish. The exception class, its message and the fragment are captured from the console.
  • Look at the bytes. The same request through the untyped path shows what the typed path refused.
  • Drive the parser directly. Four hand-made replies and one empty one are fed to the SDK's parsing function with no network involved. These are local facts about the SDK version named in the file, SDK 1.2.0, not observations of a model.
  • Notice what was not recorded. The truncation result exists only in the console capture. The call raised before the harness reached the code that saves usage and writes the file, so the run's JSONL has no row for it. The chapter calls this a defect in the instrument and leaves it on the record.

How to improve it

  • Two guards, not one. Wrap the call in except ValidationError, and separately check the parsed result for None before using it.
  • Decide what each means. A raise from truncation may deserve one retry with a larger budget. A None deserves a look at the stop reason first.
  • Size max_tokens for the whole reply, and on a model that thinks, for the thinking as well.
  • Record the failure where the data lives. Write the outcome of every call to the run file inside the except branch, so a raise leaves a row. The hands-on test checks for it.
  • Never loosen the parser to make an error go away. A parser that accepts half an object is how a truncated key lands in a database.

The edge cases are set out in Chapter 3 of the book, "Edge cases". Next: typed is not correct.

Every number in this lesson, and the file it came from (7)
ValueFile in the book's recordConditions
Invalid JSON: EOF while parsing a string at line 1 column 10runs/ch3-router-typed-2026-10-14.txtclaude-opus-4-6, max_tokens=3, one call
Invalid JSON: EOF while parsing a string at line 1 column 5runs/ch3-course-route-typed-2026-11-20.txtclaude-opus-5, max_tokens=3, one call
2runs/ch7-parse-output-format-2026-09-02.txt
parsed_output = None -- no raiseruns/ch7-parse-output-format-2026-09-02.txt
Invalid JSON: EOF while parsing a value at line 1 column 30runs/ch7-parse-output-format-2026-09-02.txtclaude-opus-5, max_tokens=16, 2026-09-02, one call
16runs/ch7-parse-output-format-2026-09-02.txt
20 of 20BOOK.mdclaude-opus-5, max_tokens 100, 2026-11-08

Reference: Chapter 3, "Edge cases" and the architecture diagram description.