Module 1 · Hands-on test
Measure the spread on your own model
You run this on your own machine with your own API key. Nothing you run is sent to this site.
Get the files
Everything this test asks you to run is in one download, laid out exactly as in the companion repository, so every command in the steps works from the unzipped folder.
Download the Module 1 kit (zip, 9 files)
ch01-sampling/ch1_course_sampling.pych01-sampling/ch1_course_check.pych01-sampling/ch1_sampling.pych01-sampling/run-2026-09-07.txtch01-sampling/run-2026-09-07.jsonlch01-sampling/course-run-2026-11-08-mt1000.txtch01-sampling/course-run-2026-11-08-mt1000.jsonlch01-sampling/course-prediction-2026-11-08-mt1000.jsonch01-sampling/course-check-2026-11-08-mt1000.txt
Unzip it, open a terminal in module-1-kit, create a Python 3.12 virtual environment, and install what Modules 1 to 4 need:
pip install anthropic==1.2.0 pydantic==2.13.5
Those two pins are read from the kit's requirements.txt, which also lists packages only later modules use. The companion repository is not open to pilot learners yet, so use this download wherever the steps say "companion repo".
Try the check before you spend anything. This runs it on the course's own recorded run and needs no key:
python ch01-sampling/ch1_course_check.py --run ch01-sampling/course-run-2026-11-08-mt1000.jsonl --prediction ch01-sampling/course-prediction-2026-11-08-mt1000.json
It should print Module 1 check: PASS. That is the course's result, not yours: do not record it below.
What you will do
Predict how many distinct answers a current model gives to one prompt over 20 calls, run it on your own API key, and pass the check script. You run the default-sampling arm only. The temperature comparison comes from the saved run you studied in Lesson 1, because current models reject that setting (Lesson 2).
This course is for learners aged 18+.
Cost
You use your own API key, so you pay for this run.
| What | Value | Source |
|---|---|---|
| Calls | 20 | runs/ch1-sampling-2026-09-07.txt |
| Input tokens, course run | 820 | runs/ch1-course-sampling-2026-11-08-mt1000.txt |
| Output tokens, course run | 6,636, of which 4,619 thinking and 2,017 answer | same file |
| Cost of that run | $0.17 | BOOK.md, Measured findings, at the Opus rates in COST.md |
That run was on claude-opus-5 on 2026-11-08 with max_tokens of 1000. Yours depends on the model you choose, how much it thinks and the prices on the day. Multiply your own token counts by the current rates; do not quote this figure as yours.
Steps
Work in the companion repo, sanovinci/dev-to-ai-engineer-runs, from its top folder.
- Create a Python 3.12 virtual environment and install
requirements.txt. - Set
ANTHROPIC_API_KEYto your own key. Never commit it and never paste it into a file in the repo. - Pick a current model ID from Anthropic's model list.
-
Predict. Choose your number before you run anything:
python ch01-sampling/ch1_course_sampling.py predict --unique <your number> -
Run:
python ch01-sampling/ch1_course_sampling.py run --model <model id> -
Check. This step needs no key and no network:
python ch01-sampling/ch1_course_check.py -
Answer the two reflection questions in writing. They are not graded.
- How many distinct answers did you get, and how far was that from your prediction?
- Read ten of your answers. Do any two of them mean different things?
Pass when
ch1_course_check.py prints Module 1 check: PASS and exits with code 0. It passes on any unique count. It checks that:
- every record has all its fields, and the calls are numbered once each;
- the run is default sampling only, on one model, with the module's prompt and
max_tokens; - each answer still matches its hash, so the file was not edited;
- token usage is logged for every call;
- your prediction is a whole number in range and is dated before your first call;
- no answer is empty and none stopped on
max_tokens.
Then mark the module done.
If the check fails
The script lists every problem with the number of the check that caught it, so one run of it shows everything to fix. Do not edit run.jsonl; rerun the step that produced it.
| Message begins | What it means | What to do |
|---|---|---|
Can't check yet |
A file is missing or is not valid JSON (exit code 2) | Run the predict step, then the run step |
Expected ... calls |
The run stopped early | Rerun the run step |
text and hash don't match |
The file was changed after the run | Rerun the run step |
Your prediction ... is later than your first call |
You predicted after running | Predict again, then run again |
the answer is empty or stopped on 'max_tokens' |
The model spent the budget thinking | Use the current harness; it sets the larger budget |
If your unique count is one, that is a result too. A system that repeats itself the first time you look is still a sampling system.
Record your result
Run ch1_course_check.py on your machine, then paste everything it printed. The page reads the first line
(Module 1 check: PASS or FAIL) and keeps the result in this browser. Do not paste an API key; the script never prints one.
No result recorded yet.