mlx-lm · Lesson 7 · Capstone

Wire It Into a Real Tool

Everything so far — run, the Python surface, serving, fine-tuning, quantizing, caching — exists to do this: build a real local-LLM tool. You'll write a content tagger that runs entirely on your Mac, mirroring the learn pipeline's own taxonomy. The point isn't the tagger; it's the integration pattern you'll reuse for everything.

🎯 Mission complete: a local model doing real work in your own code

The one idea

Two integration shapes cover almost everything: in-process and over-HTTP. In-process (load once, generate many) for batch jobs and offline tools. Over-HTTP (the OpenAI-compatible server) for decoupled tools that share one warm model. Pick by whether the model lives inside your program or beside it.

01 The capstone — a local content tagger

Here's a genuinely useful tool that fits your world: classification is cheap, high-volume work — exactly what you'd hand to a small local model to save Claude plan tokens for the deep content. This tagger takes any text and assigns one category from the learn taxonomy plus a one-line summary, entirely on your M4 Pro. It's the in-process pattern, built from parts you already know:

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

CATEGORIES = ["AI & ML", "Programming Languages", "Security", "Backend & Data", "Other"]

model, tokenizer = load("mlx-community/Llama-3.2-1B-Instruct-4bit")   # load ONCE
sampler = make_sampler(temp=0.0)                                # deterministic tags (L2)

def tag(text):
    sys = "Classify into exactly ONE category: " + ", ".join(CATEGORIES) \
        + ". Reply 'Category: <one> | Summary: <under 12 words>'."
    messages = [{"role":"system","content":sys},
                {"role":"user","content":text}]
    prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)  # L2
    return generate(model, tokenizer, prompt=prompt, max_tokens=40, sampler=sampler)

Real output, run on your machine (~/projects/learn/public/courses/mlx-lm/practice/capstone_tagger.py):

# "A SQL injection lets an attacker smuggle commands into a query..."
Category: Security | Summary: SQL injection attack.        ✓ spot on

# "Apple's MLX framework runs LLMs on Apple Silicon..."
AI & ML | This text discusses Artificial Intelligence and Machine Learning... ~ right category, loose format

💡 Every line here is a lesson you already did load/generate (L2) · chat template (L2) · deterministic sampler (L2) · the 4-bit model (L1/L5). The "tool" is just those parts pointed at a job. That's the whole game.

02 The honest part — small models need guardrails

Notice the second result drifted from the requested format. That's the real lesson of shipping local models: a 1B model is fast and free but loose. Three levers, in order of effort:

Bigger model

Swap the --model id for a 7B/8B 4-bit. Same code, sharper output — your memory budget is the limit.

Constrain decoding

Force valid output with structured generation (e.g. Outlines's mlx-lm backend, or logits_processors) so it can't break format.

Specialize it

LoRA-tune (L4) on a few hundred tagged examples — the cheapest way to make a small model reliable at one job.

💡 The judgment call this teaches Match model size to task stakes. High-volume, low-stakes (tagging, routing, dedup) → small local model, accept some noise, verify cheaply. Low-volume, high-stakes (the final write-up) → a big model or Claude. You now have the whole spectrum on one machine.

03 The other shape — over-HTTP, for decoupled tools

When several tools (or a long-running app) should share one warm model, don't load it in each — run the Lesson 3 server and call it. Same task, the beside-your-program pattern:

# terminal 1: one warm model for everything
mlx_lm.server --model mlx-community/Llama-3.2-1B-Instruct-4bit --port 8080

# any tool, any language that speaks OpenAI:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="local")
tag = client.chat.completions.create(model="…4bit", messages=[…]).choices[0].message.content

	In-process	Over-HTTP (server)
Model lives	inside your program	beside it, shared
Best for	batch scripts, one-off tools, notebooks	multiple tools/apps, other languages, long-running services
Startup cost	load per process	load once; clients are instant
From Lesson	2 (load/generate)	3 (server + OpenAI SDK)

04 The whole arc, in one tool

Seven lessons, one tool. You can run a model, shrink it to fit, teach it a job, call it in-process or over HTTP, and cache the expensive parts — the full local-LLM toolkit, on your own machine.

🎯 Your capstone win & where to take it Run

~/projects/learn/public/courses/mlx-lm/practice/.venv/bin/python ~/projects/learn/public/courses/mlx-lm/practice/capstone_tagger.py

and watch a local model classify text. Then make it yours: point it at your own categories, swap in a bigger model, or LoRA-tune it on examples. The natural next build — wire a tagger like this into the learn pipeline to pre-classify browsing locally, reserving Claude tokens for the deep dives.

05 Check yourself

Think first, then expand. (Ready to build something specific? Tell me the tool and we'll design it.)

Q: You're writing a one-off batch script to tag 10,000 files overnight. In-process or server?

A: In-process — load once at the top, loop generate. No need for HTTP overhead or a separate process; one script owns the model for its lifetime.

Q: Three different apps on your Mac each need the same model occasionally. Which shape?

A: Over-HTTP. Run one warm server; each app calls it with the OpenAI SDK. Loading the model three times wastes memory and startup time.

Q: Your small tagger is ~85% accurate but sometimes breaks the output format. Cheapest reliable fix?

A: Constrain decoding (structured generation / logits processors) so invalid output is impossible — cheaper than a bigger model, and it fixes format directly. If accuracy (not format) is the issue, LoRA-tune on examples.

Q: Why hand classification to a local model instead of Claude?

A: It's high-volume, low-stakes work — fast and free on your Mac, and "good enough" with cheap verification. Reserve premium tokens for low-volume, high-stakes output. Match model cost to task stakes.

06 You've finished the arc — going deeper

You can now run, serve, fine-tune, quantize, cache, and integrate local LLMs on Apple Silicon. Where to go next:

mlx-lm examples/ — production-shaped scripts to lift patterns from.
Outlines + mlx-lm — structured/constrained generation for reliable tool output.
mlx-vlm — the same ideas for vision-language models, when you want images in the loop.
r/LocalLLaMA — the community to test your skills and compare Apple-Silicon setups (wisdom).

Keep handy: Glossary & quick reference · ← Lesson 6 · ↩ back to Lesson 1. Course complete — but the mission's open-ended: bring me the real tool you want to build and we'll add lessons aimed straight at it.