Stop guessing whether a model will run. With one formula you can tell — before downloading 8 GB — what fits in your 24 GB M4 Pro, how fast it'll decode, and which size/quant to pick for a given job. Turn "will this work?" into arithmetic.
🎯 Mission skill: choose a model on purpose for your machine + taskWeight RAM ≈ parameters × bits ÷ 8. That, plus the KV cache for your context, has to fit in unified memory alongside the OS. And bigger models decode slower (more bytes read per token). So the game is: pick the biggest model whose memory leaves headroom — and no bigger than the task needs.
A model is mostly its weights, and each weight takes bits bits. So the memory just to hold it is parameter-count times bytes-per-weight:
# weight RAM (GB) ≈ params(billions) × bits ÷ 8 1B × 4-bit = 1 × 4 / 8 = 0.5 GB 3B × 4-bit = 3 × 4 / 8 = 1.5 GB 7B × 4-bit = 7 × 4 / 8 = 3.5 GB 7B × 8-bit = 7 × 8 / 8 = 7.0 GB 7B × 16-bit = 7 × 16 / 8 = 14 GB
Then add ~10–25% for activations + framework overhead, plus the KV cache (§4). Quantization (Lesson 5) is the dial that moves a model between these rows.
macOS + your apps need RAM too — budget roughly 6–8 GB for the system, leaving ~16–18 GB for a model + its context. Weight RAM by size and quant (before context):
| Params | 4-bit | 8-bit | fp16 |
|---|---|---|---|
| 1B | 0.5 GB ✓ | 1 GB ✓ | 2 GB ✓ |
| 3B | 1.5 GB ✓ | 3 GB ✓ | 6 GB ✓ |
| 7–8B | ~4 GB ✓ | ~7 GB ✓ | ~15 GB ⚠️ tight |
| 14B | ~7 GB ✓ | ~14 GB ⚠️ | ~28 GB ✗ |
| 32B | ~16 GB ⚠️ tight | ~32 GB ✗ | ✗ |
| 70B | ~35 GB ✗ | ✗ | ✗ |
✓ comfortable · ⚠️ fits but leaves little headroom for context/apps · ✗ won't fit in 24 GB. Sweet spot for daily use: 7–14B at 4-bit.
Decode is memory-bandwidth-bound (Lesson 1): each token reads the whole weight set. Double the weights, roughly halve the tokens/sec. Measured on your M4 Pro, same 4-bit quant:
Weights are fixed, but the KV cache (Lesson 6) grows with how many tokens are in play — roughly proportional to context length. For small models it's modest; for long documents or long chats it can add multiple GB on top of the weights. So the real budget is:
# must fit in unified memory:
weights + KV cache(context) + activations/overhead + OS/apps ≤ 24 GB
--max-kv-size (Lesson 6) or keep prompts short. When a big model OOMs mid-long-prompt, the KV cache is usually why.Routing/tagging/extraction → 1–3B is plenty (and use Lesson 8 to make it reliable). Reasoning/coding/writing → reach for 7–14B.
Default 4-bit. Step to 8-bit only if you measured a quality problem (or you're preserving a fine-tune — Lesson 5) and have the RAM.
params × bits ÷ 8 + overhead + context. Must leave ~6–8 GB for macOS. If it's tight, drop a quant level or a size.
Run it once, read Peak memory + tokens/sec. The dashboard from Lesson 1 is your ground truth — trust it over the estimate.
Think first, then expand. (Eyeing a specific model? Tell me the name and we'll size it for your 24 GB together.)
Qwen2.5-14B-Instruct-4bit run on your 24 GB Mac?A: Weights ≈ 14 × 4 / 8 = 7 GB, plus overhead + context ≈ 8–10 GB used. With ~16 GB available after the OS, yes — comfortably, with room for a decent context. It'll be noticeably slower per token than a 3B, though.
A: The weights (~16 GB) fit, but the long prompt's KV cache pushed total memory past what's available. Cap it with --max-kv-size, shorten the prompt, or use a smaller model.
A: Smallest model that's accurate enough — 1–3B — at 4-bit, made reliable with constrained decoding (Lesson 8). Speed scales inversely with size, so small wins for throughput.
A: 7 × 16 / 8 = 14 GB of weights, plus overhead + context → ~16–18 GB. On 24 GB it's tight but possible with a short context; 4-bit (~4 GB) is the saner choice unless you specifically need full precision.
--max-kv-size for big models.Keep handy: Glossary & quick reference · ← Lesson 8. You can now run, serve, tune, quantize, cache, structure output, and size models on your Mac. The open invitation stands → bring a real tool you want to build (e.g. a local tagger wired into the learn pipeline) and we'll make it a capstone.