Lesson 7 ended on a problem: the 1B tagger drifted out of format. Here's the fix that turns a small local model into a reliable component — constrain the decoder to a schema so it cannot produce invalid output. Not a better prompt; a mathematical guarantee.
🎯 Mission step: make local-model output reliable enough to parse in codeDon't ask for JSON and hope — mask every token that would break it. Compile your schema into a finite-state machine, and at each decoding step set the logits of all illegal tokens to −∞. The model literally cannot emit a character that violates the schema. Output that always parses, every time, from a 1B model.
In Lesson 7 your tagger sometimes returned Category: Security | Summary: … and sometimes wandered into prose. For a human that's fine; for code that does json.loads(output), one stray token is a crash. The smaller the model, the more it drifts — and small local models are exactly what you want for cheap, high-volume work. So you need structural reliability, not a sterner prompt.
A model picks each token from a probability distribution over the whole vocabulary. Constrained decoding inserts a step: given "what's been written so far," a finite-state machine (built from your schema) knows which tokens keep the output valid. Everything else gets masked to −∞ before sampling.
The Outlines library has an mlx-lm backend. Install it into your practice venv, define a Pydantic model, and pass it as output_type:
# uv pip install "outlines[mlxlm]" (per your uv preference) import outlines, mlx_lm, json from pydantic import BaseModel, Field from typing import Literal, List class Tag(BaseModel): category: Literal["AI & ML", "Programming Languages", "Security", "Other"] summary: str = Field(max_length=80) # bound it (see §05) keywords: List[str] = Field(max_length=5) model = outlines.from_mlxlm(*mlx_lm.load("mlx-community/Llama-3.2-1B-Instruct-4bit")) out = model("Classify and summarize: A SQL injection smuggles commands into a query.", output_type=Tag, max_tokens=256) tag = Tag.model_validate_json(out) # never raises — out is valid by construction
Real output on your M4 Pro — three inputs, all parsed & validated, zero format errors:
# category keywords
Security ['SQL Injection', 'Attack', 'Smuggling', 'Commands', 'Query']
Programming Languages ['React', 'useEffect', 'side effects', 'rendering']
AI & ML ['Apple Silicon', 'MLX', 'LLMs', 'Large Language Models']
~/projects/learn/public/courses/mlx-lm/practice/.venv/bin/python ~/projects/learn/public/courses/mlx-lm/practice/structured_tagger.py. The same 1B model that drifted in Lesson 7 now returns JSON that model_validate_json accepts every time — and it even gets MLX → AI & ML right, because category can only be one of the allowed literals.max_tokens before it closes the JSON, you get a truncated (still-invalid-overall) string. First attempts here failed with Unterminated string because the model wrote a long summary and ran out of budget. Two fixes, both shown above: bound free-text fields in the schema (Field(max_length=80)) so they must close, and give a sane max_tokens. Constrain the shape and the size.| Prompt & parse | Constrained decoding | |
|---|---|---|
| Validity | probabilistic — may fail | guaranteed by construction |
| Small models | drifts, needs retries | works — the model can't drift |
| Enums | may invent values | Literal[...] → only allowed values |
| Cost | retries on failure | one-time FSM compile + tiny per-token mask |
| Caveat | — | not for batched generation; still budget tokens |
Reliable structure is what makes a local model a component instead of a toy. With guaranteed JSON you can safely build: classifiers/routers, data extraction from messy text, tool-calling (the model emits a valid function-call object), form-filling, and grading/judging pipelines — all on-device, all parseable without defensive try/except gymnastics.
generate for a constrained model(..., output_type=Schema) and the tagger becomes production-grade — exactly what you'd want before wiring it into the learn pipeline.Think first, then expand. (Want a schema for a specific tool — a function-call object, an extraction shape? Ask me and we'll design it.)
"category": "Frontend" if your Literal only lists four values?A: At the step where it writes the category value, the FSM only permits tokens that continue one of the four allowed strings. "Frontend" isn't a valid path, so those tokens are masked to −∞ and can't be sampled. The enum is enforced, not requested.
Unterminated string. What happened and how do you fix it?A: Generation hit max_tokens before the JSON closed — valid-so-far but incomplete. Bound free-text fields with Field(max_length=N) and/or raise max_tokens so it has room to finish.
A: When you want free-form prose (a chat reply, a summary a human reads), or when a big model already nails the format and you're not parsing it in code. Use it where a program consumes the output.
A: No — it guarantees format, not correctness. The model can still pick the wrong (but valid) category. Constraining the enum helps, but for higher accuracy you still reach for a bigger model or a fine-tune.
Keep handy: Glossary & quick reference (see constrained decoding, Outlines) · ← Lesson 7. Next up → Lesson 9: Model Selection & Memory Math — how to know in advance what actually fits and runs well on your M4 Pro (params × bits → GB, plus KV-cache overhead).