The whole course in one tool. Embed your documents, find the passages that answer a question, and have a local model write the answer grounded in your text — all on your Mac. We build it over this very course's lessons, then you point it at anything.
🎯 Mission complete: a real local-LLM tool over your own contentRAG = retrieve (Lesson 11) + generate (Lesson 2). A model can't know your private notes — so don't ask it to remember. Find the relevant passages by meaning, put them in the prompt, and tell the model to answer using only those. You get current, grounded answers from a small local model, with no training and no cloud.
This capstone is every prior lesson working together: you run a model (L1–2) at a size that fits (L9), use embeddings to retrieve (L11), and generate a grounded answer (L2) — optionally as structured output (L8) or via tools (L10). We'll index this course's own lessons and ask it questions about what you learned.
Strip the lessons to text, chunk into overlapping windows, embed, retrieve, and generate. The core is ~20 lines (full script: practice/rag.py):
# index: overlapping ~500-char windows keep a whole idea in one chunk chunks = [t[s:s+500] for doc in lessons for t in [text_of(doc)] for s in range(0, len(t)-200, 350)] cvecs = embed(chunks) # Lesson 11 # retrieve top-k by cosine (normalized vectors → dot product) qv = embed([question])[0] top = sorted(range(len(chunks)), key=lambda i: -float(mx.sum(qv*cvecs[i])))[:4] context = "\n".join(f"- {chunks[i]}" for i in top) # generate, grounded ONLY in the retrieved context (Lesson 2) msgs = [{"role":"user", "content": f"Answer using ONLY these notes:\n{context}\n\nQuestion: {question}"}] answer = generate(lm, tok, prompt=tok.apply_chat_template(msgs, add_generation_prompt=True))
Real run on your M4 Pro — 226 windows indexed from the 12 lessons, asked an unseen question:
# Q: "On a Mac, why is there no step to move the model to the GPU?" # retrieved → Lesson 1 (unified memory). Answer: On a Mac running Apple Silicon, there is no need to explicitly move the model to the GPU because Apple Silicon uses unified memory — the CPU and GPU share the same RAM pool. The model is loaded once and computed directly on the GPU without copying.
~/projects/learn/public/courses/mlx-lm/practice/.venv/bin/python ~/projects/learn/public/courses/mlx-lm/practice/rag.py. A local model just answered a question about your own course material — correctly, grounded in the exact lesson, with nothing leaving your Mac. Change the glob to point at your notes, a repo, or public/content and it's your knowledge base.| Lever | When retrieval is weak |
|---|---|
| Chunking | too small fragments ideas; too big dilutes them. Overlapping windows are a solid default. |
| Embedding model | a stronger/larger embedder (e.g. a BGE model) separates meanings better than a tiny one. |
| top-k | retrieve a few more chunks so the right one is in the context. |
| Re-ranking | fetch top-20, then re-score with a cross-encoder and keep the best 4. |
That instruction is the whole point of RAG: it grounds the model in retrieved text instead of its (small, dated) memory. Recall Lesson 1, where the bare 1B model claimed MLX was for "M1 and M2 processors" — out of date. Feed it the right passage and it answers from that, not from a fuzzy recollection. Grounding is how a small local model gives trustworthy, current answers about your data.
sources list)? Add Lesson 8's output_type. Want it to decide whether to search vs. answer directly? That's a Lesson 10 tool call. RAG is the backbone; the other lessons are attachments.
Pick an answer for instant feedback (your choices are saved locally). This is the last set — you've earned it.
1. Why retrieve passages instead of just asking the model your question directly?
RAG augments the context, not the weights. The model answers from passages you fetched, so it can speak to private or up-to-date data it never trained on.
2. The capstone gave a vague answer at first. What was actually wrong?
In RAG the retrieval step is usually the bottleneck. Better chunking (overlapping windows) surfaced the right passage and the answer became grounded and correct.
3. What does "Answer using ONLY these notes" accomplish?
Grounding instructions tell the model to answer from the provided context rather than its fuzzy memory — the fix for the kind of dated claim the bare 1B model made in Lesson 1.
4. Your RAG tool misses relevant docs. Best first thing to improve?
If the right passage isn't retrieved, no generation model can answer from it. Fix retrieval first — that's where RAG quality lives.
Twelve lessons. You can now, entirely on your Mac: run models (CLI + Python), serve them (OpenAI-compatible), fine-tune with LoRA, quantize & convert, use the KV cache, force structured output, do tool calling, compute embeddings, size a model to your RAM, and build RAG over your own content. That's a complete local-LLM toolkit.
Keep handy: Glossary & quick reference · ← Lesson 11 · ↩ Lesson 1. The real next step is yours: point this RAG at the learn content or your notes and make it a tool you actually use — bring it to me and we'll wire it in.