Glossary & Quick Reference

CLI commands at a glance

Every command takes --model <hf-id-or-path> and -h for its full options. CLI is dot-style today (mlx_lm.generate); a space-style proposal exists.

The canonical Python snippet

Terms

MLX

Apple's open-source array framework for Apple Silicon — NumPy/PyTorch-like, but built for the M-series GPU and unified memory. mlx-lm is built on top of it.

mlx-lm

The Python package + CLI tools for running LLMs on MLX. The layer you work at in this course.

Unified memory

Apple Silicon's single RAM pool shared by CPU and GPU — no separate VRAM, no host→device copy. Why there's no .to("gpu") step, and why "does it fit in RAM?" is the real constraint.

Quantization 4bit / 8bit

Storing each weight in fewer bits (e.g. 4 instead of 16) via a shared scale per group + small int per weight. Shrinks + speeds up at a rounding cost. Verified on the fused pirate: fp16 2.3 GB → 8-bit 1.2 GB (tuning kept) → 4-bit 663 MB (small LoRA delta washed out). Make your own with mlx_lm.convert -q --q-bits N. Lesson 5.

Prompt cache cache_prompt

A KV cache for a fixed context, saved to disk by mlx_lm.cache_prompt --prompt-cache-file and reused via mlx_lm.generate --prompt-cache-file. Verified: a ~1257-token context dropped to 15 prefilled tokens when cached. Reuse a big system prompt/document cheaply. Lesson 6.

Parameters 1B, 7B…

The count of learned weights (1B = one billion). Roughly tracks capability and RAM footprint; pick by what fits your machine.

Base vs Instruct

A base model only continues text; an Instruct (a.k.a. chat/-it) model is tuned to follow instructions. Use Instruct models when you want answers to prompts.

Chat template

The model-specific formatting that wraps a list of role/content messages into the exact token string the model was trained on. Applied via tokenizer.apply_chat_template(...). Skip it and an Instruct model misbehaves — Lesson 2.

Tokens & tokens-per-sec

Models read/write in tokens (sub-word chunks). Throughput is reported as tokens-per-second. Your speedometer for every run.

Prefill (prompt) vs Decode (generation)

Prefill: process the whole prompt at once (parallel, compute-bound). Decode: emit one token at a time, each depending on the last (sequential, memory-bandwidth-bound). mlx-lm reports both speeds separately; decode is the speed you feel.

KV cache

Cached key/value tensors from already-processed tokens so each new token doesn't re-read the whole context. Why decode stays fast as text grows; the thing cache_prompt saves.

Peak memory

The high-water RAM mark for a run (weights + KV cache + activations). On unified memory this is your binding constraint — printed after every generation.

LoRA / QLoRA

Low-Rank Adaptation: freeze W, learn the update as ΔW = B·A (two skinny matrices, rank r) — train only A,B. QLoRA = doing it on a quantized base. Verified: tuning Llama-3.2-1B-4bit trained 0.228% of weights → an 11 MB adapter that changed its voice. Lesson 4.

Rank (r) & scale (α)

r = width of the LoRA bottleneck (8–16 typical) — higher = more expressive, bigger adapter. α = how strongly the overlay applies, as α/r. Set in adapter_config.json; ours was rank 8, scale 20.

Adapter adapters/

The LoRA output folder: adapter_config.json (the recipe) + adapters.safetensors (the learned A,B, ~MBs). Attach at runtime with --adapter-path; swap many cheaply over one frozen base.

Fuse

mlx_lm.fuse bakes B·A into W → a standalone model in fused_model/. Gotcha: default re-quantizes to 4-bit and can erase a small delta — use --dequantize to keep it. Lesson 4.

mlx-community

The Hugging Face org hosting pre-converted/quantized MLX models. Drop any of its ids straight into --model — no conversion needed.

Constrained decoding / structured output

Masking illegal tokens at each step so output must match a schema (compiled to a finite-state machine). outlines.from_mlxlm(*mlx_lm.load(...)) + a Pydantic output_type → JSON valid by construction. Verified: the 1B model returned schema-valid JSON every time (fixes Lesson 7's drift). Bound free-text fields (Field(max_length=N)) so the JSON can close within the token budget. Lesson 8.

Tool calling

The model emits a structured "call function X with args" object; your code runs it and feeds the result back. = constrained output (Union of tool schemas) + dispatch. Verified: 3B model mapped "12 plus 30" → add(12,30)=42 and "weather in Tokyo" → get_weather('Tokyo'). The agentic primitive. Lesson 10.

Embedding

A fixed-length vector for a text where similar meanings sit nearby. mlx_embeddings.utils.load(...) → model(...).text_embeds (normalized). all-MiniLM-L6-v2 → 384-d. Different model class (encoder) than mlx-lm generators. Lesson 11.

Cosine similarity

Closeness of two vectors by the angle between them; for normalized vectors it's just their dot product (mx.sum(a*b)). How semantic search ranks — matches meaning, not keywords. Lesson 11.

RAG (retrieval-augmented generation)

Embed docs once → at query time retrieve the top-k nearest by cosine → put them in the prompt → generate grounded in them. Augments the context, not the weights. Composes embeddings (L11) + generate (L2) + optional structured output (L8). Capstone (L12): verified over the course's own lessons.

Chunking

Splitting documents into passages before embedding. Drives retrieval quality — sentence-size fragments ideas; overlapping ~500-char windows keep an idea whole. In RAG the chunking/embeddings are usually the bottleneck, not the generation model. Lesson 12.

Grounding

Instructing the model to answer using only the retrieved context ("Answer using ONLY these notes"). Cuts made-up / outdated answers — the fix for the kind of stale claim a bare small model makes. Lesson 12.

Memory math params × bits ÷ 8

Weight RAM (GB) ≈ params(billions) × bits ÷ 8, plus ~10–25% overhead and the KV cache. Tells you what fits before downloading. On the 24 GB M4 Pro: measured 1B-4bit→0.8 GB, 3B-4bit→1.87 GB; sweet spot 7–14B at 4-bit. Bigger = slower (≈1/size tok/s). Lesson 9.

OpenAI-compatible server

mlx_lm.server exposes /v1/chat/completions, /v1/completions, /v1/models in the exact OpenAI request/response shape. Any OpenAI client works by changing only base_url → http://127.0.0.1:8080/v1; binds to localhost by default, --host 0.0.0.0 to widen. Not production-hardened. Lesson 3.

SSE (streaming)

Server-Sent Events — how the server streams tokens over HTTP when you pass stream=True; the SDK reassembles them into delta chunks. The HTTP twin of stream_generate.