Measured Academy Measured Academy
Français Pilot Progress 0

Module 3 · Hands-on test

Type the router, break it on purpose, keep the record

You run this on your own machine with your own API key. Nothing you run is sent to this site.

Get the files

Everything this test asks you to run is in one download, laid out exactly as in the companion repository, so every command in the steps works from the unzipped folder.

Download the Module 3 kit (zip, 9 files)

  • ch03-structured-output/ch3_course_route_typed.py
  • ch03-structured-output/ch3_course_check.py
  • ch03-structured-output/ch3_route_typed.py
  • ch03-structured-output/run-2026-10-14.txt
  • ch03-structured-output/run-2026-10-14.jsonl
  • ch03-structured-output/course-run-2026-11-20.txt
  • ch03-structured-output/course-run-2026-11-20.jsonl
  • ch03-structured-output/course-prediction-2026-11-20.json
  • ch03-structured-output/course-check-2026-11-20.txt

Unzip it, open a terminal in module-3-kit, create a Python 3.12 virtual environment, and install what Modules 1 to 4 need:

pip install anthropic==1.2.0 pydantic==2.13.5

Those two pins are read from the kit's requirements.txt, which also lists packages only later modules use. The companion repository is not open to pilot learners yet, so use this download wherever the steps say "companion repo".

Try the check before you spend anything. This runs it on the course's own recorded run and needs no key:

python ch03-structured-output/ch3_course_check.py --run ch03-structured-output/course-run-2026-11-20.jsonl --prediction ch03-structured-output/course-prediction-2026-11-20.json

It should print Module 3 check: PASS. That is the course's result, not yours: do not record it below.

What you will do

Run the ten tickets through the router twice, untyped and typed, then make one typed call fail on purpose. Predict three things before you run. The course harness writes one row per call, including the call that fails, which is the part the book's own harness missed.

This course is for learners aged 18+.

Cost

You use your own API key, so you pay for this run.

What Course run, claude-opus-5 Book run, claude-opus-4-6
Calls 21: ten untyped, ten typed, one truncated the same
Tokens, untyped calls 549 input, 528 output 416 input, 40 output
Tokens, typed calls 2,919 input, 120 output 2,266 input, 80 output
Cost of that run $0.03 $0.02
Source BOOK.md, Measured findings; runs/ch3-course-route-typed-2026-11-20.txt COST.md, Ch 3 row

The truncated call has no usage on either run: it raised before there was a response to read. The check itself costs nothing.

Steps

Work from the unzipped kit (the same files as the companion repo's ch03-structured-output/), from its top folder. Your files go in ch03-structured-output/out/.

  1. Read ch03-structured-output/ch3_route_typed.py, the book's harness: the ten tickets, the two routers and the truncation call. You will run the course version, ch3_course_route_typed.py, which sends the same prompts and changes five things: it reads the first text block, gives routing calls room to think (a much larger max_tokens than the book's), records every typed call as returned, raised or None, writes a row for every call including the one that raises, and stops at a call and token limit.
  2. Predict, before any call. The harness saves your three predictions with the time:

    python ch03-structured-output/ch3_course_route_typed.py predict --before-clean <0-10> --same-queue <0-10> --truncation raised|none|returned
    
    • --before-clean: on how many of the ten tickets the untyped return will be exactly a queue name, with no cleanup.
    • --same-queue: on how many tickets the untyped and typed routers will choose the same queue.
    • --truncation: what the truncated typed call will do on your model.
  3. Run it on your own key and a current model ID:

    python ch03-structured-output/ch3_course_route_typed.py run --model <model id>
    

    It writes ch03-structured-output/out/run.jsonl: a row per call, then a header with the three counts. Each row records the part (before, after or truncation), the ticket, the outcome, the untyped answer or the typed queue, the exception's class and message if it raised, the stop reason and the token usage, with thinking tokens where the API reports them.

  4. Compare your rows with the two recorded runs, ticket by ticket: the book's in ch03-structured-output/run-2026-10-14.jsonl, and the course run on a current model in ch03-structured-output/course-run-2026-11-20.jsonl. Then look at Ticket 10 and decide which queue you would put in an answer key, and why.

  5. Extension, optional: write a caller for the typed router that handles a ValidationError and a None result as two separate branches (Lesson 3), and test it against the truncation row.

Check

This step needs no key and no network:

python ch03-structured-output/ch3_course_check.py

Pass when

ch3_course_check.py prints Module 3 check: PASS and exits with code 0. The script passes on any count of clean untyped returns and any count of agreeing tickets. It checks that:

  • the run file has its header and every call record with all fields;
  • ten untyped and ten typed calls cover every ticket once, and there is one truncation call;
  • the tickets and queues are the module's, on one model, with one routing max_tokens and a lower one for the truncation call;
  • each row is consistent: a typed call that returned holds one of the four queues, one that raised names its exception;
  • the three counts in the header recompute from the rows, so the file was not edited;
  • token usage is logged for every call that came back;
  • the truncation call did not return a queue: it raised or gave None, and either way it is a row in the file;
  • your predictions are in range and dated before your first call;
  • every routing call completed: no empty untyped answer, no max_tokens stop, and all ten typed calls returned a queue.

Then mark the module done.

If the check fails

The script lists every problem with the number of the check that caught it. Do not edit run.jsonl; rerun the harness.

Message begins What it means What to do
Can't check yet A file is missing or is not valid JSON (exit code 2) Create both files in ch03-structured-output/out/
Header: or missing field The file is not in the course format, for example the book's own three-line file Run the course harness, which writes one row per call
Expected A part did not cover every ticket, or the truncation row is absent Rerun the harness; if you wrote your own, record the truncation call inside its except branch
doesn't recompute The header and the rows disagree Rerun; do not edit the file
The truncation call returned a queue Its budget was large enough to finish Rerun with a lower --truncation-max-tokens
the answer is empty, stopped on 'max_tokens' or Typed call A routing call did not complete Rerun with a larger --max-tokens
Prediction A value is out of range, or dated after the first call Predict again, then run again

Two recorded runs to compare yours with. The book's run on claude-opus-4-6 had the untyped router clean on 10 of 10 tickets, the typed router returning on 10 of 10 with the same queue each time, and the truncated call raising Invalid JSON: EOF while parsing a string at line 1 column 10 (runs/ch3-router-typed-2026-10-14.txt). The course run on claude-opus-5 had 10 of 10 clean, 10 of 10 typed, the same queue on 9 of 10, and the truncated call raising Invalid JSON: EOF while parsing a string at line 1 column 5 (runs/ch3-course-route-typed-2026-11-20.txt). Yours will differ. Say how, and why you think so.

Record your result

Run ch3_course_check.py on your machine, then paste everything it printed. The page reads the first line (Module 3 check: PASS or FAIL) and keeps the result in this browser. Do not paste an API key; the script never prints one.

No result recorded yet.