Generation is only half of a real local-LLM tool. The other half is retrieval: finding the right text by meaning, not keywords. Embeddings turn text into vectors on your Mac, so you can search your own notes, docs, or the learn content by what they mean — the foundation of RAG.
An embedding maps text to a vector so that similar meanings land near each other. Embed your documents once, embed a query, and the closest vectors are the most relevant passages — even when they share no words. "Closeness" is just the cosine of the angle between vectors, which for normalized vectors is a single dot product.
A model only knows what's in its weights and its prompt. To build a tool that answers from your data — your notes, a codebase, the learn deep-dives — you must find the relevant text and put it in the prompt. Keyword search misses paraphrases ("less memory" vs "fewer bits"). Embeddings match on meaning, so you retrieve the right passage even when the words differ. That's the engine under "chat with your documents."
all-MiniLM-L6-v2 outputs a 384-dimensional vector per text and runs in milliseconds on your GPU. Different package (mlx-embeddings), same Apple-Silicon speed.
Think of every text as a point in a high-dimensional space. Texts about the same thing cluster together; a query lands near the cluster that answers it. Retrieval = "find the nearest points."
# uv pip install mlx-embeddings import mlx.core as mx from mlx_embeddings.utils import load model, tok = load("mlx-community/all-MiniLM-L6-v2-4bit") def embed(texts): i = tok.batch_encode_plus(texts, return_tensors="mlx", padding=True, truncation=True, max_length=512) return model(i["input_ids"], attention_mask=i["attention_mask"]).text_embeds dv = embed(docs) # one normalized 384-d vector per document qv = embed([query])[0] scores = [float(mx.sum(qv * dv[i])) for i in range(len(docs))] # dot = cosine
Real run on your M4 Pro — query "How do I make a model use less memory?" against four mlx-lm facts:
0.369 Unified memory lets the CPU and GPU share one pool of RAM. 0.361 Quantization shrinks a model by storing weights in fewer bits. 0.290 The KV cache stores past keys/values so decoding stays fast. 0.279 LoRA fine-tunes a model by training a small low-rank adapter.
~/projects/learn/public/courses/mlx-lm/practice/.venv/bin/python ~/projects/learn/public/courses/mlx-lm/practice/semantic_search.py. The top hit is correct — and the quantization line ranks #2 with zero words in common with the query. That's meaning-based retrieval, running entirely on your Mac.Semantic search is the heart of Retrieval-Augmented Generation: instead of hoping the model memorized your data, you fetch the relevant bits and hand them to it. The pattern composes everything you've learned:
Embed every document/chunk → store the vectors (even a list in memory works to start).
Embed the question, take the top-k nearest documents by cosine similarity.
Put those passages in the prompt, then generate (Lesson 2) an answer grounded in them — optionally as structured output (Lesson 8).
learn
Embed your public/content deep-dives and course lessons once, and you can ask "where did I learn about X?" and jump to the right page — meaning-based search over your own study material, no cloud. A natural capstone.
Pick an answer for instant feedback. (Want to index your own folder of notes or the learn content? Ask me and we'll build the index.)
1. What does an embedding actually produce for a piece of text?
An embedding is a vector (here 384-d). Similar meanings map to nearby vectors, which is what makes meaning-based comparison possible.
2. The query and the "quantization" doc share no words, yet it ranked #2. Why?
That's the whole point of embeddings: similarity is by meaning (vector distance), not lexical overlap. "Use less memory" and "fewer bits" are semantically close.
3. Why is cosine similarity here just a single dot product?
For unit-length vectors, cosine(a,b) = a·b. The model returns normalized text_embeds, so mx.sum(qv * dv[i]) is the cosine similarity.
4. In a RAG pipeline, what do the retrieved passages do?
RAG = retrieve, then put the relevant text in the prompt and generate. No training involved — you're augmenting the context, not the weights.
Keep handy: Glossary & quick reference · ← Lesson 10. You now have both halves — generation and retrieval — on-device. The standing capstone: index the learn content and ask it questions. Bring it to me and we'll build it.