mlx-lm · Lesson 5

Quantize & Convert — Make a Model Fit

Stop borrowing mlx-community's pre-shrunk models and make your own. mlx_lm.convert turns any Hugging Face model into an MLX model and quantizes it to 4 or 8 bits — trading memory and speed against quality, on purpose. And it closes the loop on Lesson 4's fuse gotcha: you'll see exactly where your pirate tuning survives and where it dies.

🎯 Mission step: quantize & convert models to fit your machine

The one idea

Quantization stores each weight in fewer bits — smaller and faster, at some quality cost. The -4bit suffix you've been running since Lesson 1 is just the output of mlx_lm.convert. Now you control the dial: --q-bits 8 for near-lossless and roomy, --q-bits 4 for tiny and fast. Picking the bit-width on purpose is the skill.

01 What "quantize" actually means

A model's weights are numbers. Full precision stores each in 16 bits. Quantization packs them into 4 or 8 bits by storing a shared scale per small group of weights and a tiny integer per weight. Fewer bits → less memory to hold and less memory bandwidth to read → smaller files and faster decode, paid for in a little rounding error.

💡 Why it speeds things up on your Mac Recall from Lesson 1 that decode is memory-bandwidth-bound — the GPU spends its time reading weights. Halve the bits, roughly halve the bytes read per token, and decode speeds up. On unified memory, a smaller model also leaves more RAM for context. (Group size — --q-group-size, default 64 — sets how many weights share one scale; smaller groups = more faithful, slightly bigger.)

02 The command

Point mlx_lm.convert at a Hugging Face repo or a local model folder, and (optionally) quantize:

# pull a full model from HF and quantize it to 4-bit:
~/projects/learn/public/courses/mlx-lm/practice/.venv/bin/mlx_lm.convert \
  --hf-path mistralai/Mistral-7B-Instruct-v0.3 \
  --mlx-path ./my-mistral-4bit \
  -q --q-bits 4          # -q = quantize; drop it for a plain format-convert

Flag	Meaning
`--hf-path` / `--model`	Source: an HF repo id or a local model directory
`--mlx-path`	Where to write the converted model
`-q` / `--quantize`	Quantize (omit for a pure HF→MLX format conversion at full precision)
`--q-bits`	Bits per weight — `4` or `8`
`--q-group-size`	Weights per shared scale (default 64)
`--upload-repo`	Publish the result to a Hugging Face repo (this is how `mlx-community` is built)

💡 So that's all mlx-community is Every model you've loaded — …Llama-3.2-1B-Instruct-4bit — is someone running mlx_lm.convert -q on a base model and pushing it with --upload-repo. Now you can make (and share) your own.

03 Closing the Lesson 4 loop — where your tuning survives

Remember the fused fp16 pirate from Lesson 4 (2.3 GB)? Let's shrink it on purpose and watch what happens to the tuning. Same model, three precisions, same held-out prompt — all real, on your M4 Pro:

$ mlx_lm.convert --hf-path ~/projects/learn/public/courses/mlx-lm/practice/pirate-fused-fp16 \
    --mlx-path ~/projects/learn/public/courses/mlx-lm/practice/pirate-8bit -q --q-bits 8    # → 8.5 bpw
$ mlx_lm.convert --hf-path ~/projects/learn/public/courses/mlx-lm/practice/pirate-fused-fp16 \
    --mlx-path ~/projects/learn/public/courses/mlx-lm/practice/pirate-4bit -q --q-bits 4    # → 4.5 bpw

# prompt: "What is machine learning?"
fp16  (2.3 GB):  Machine learning be the study o' how machines learn...   ✓ pirate
8-bit (1.2 GB):  Machine learning be the study o' how machines learn...   ✓ pirate
4-bit (663 MB):  Machine learning (ML) is a subset of artificial...      ✗ plain English

2.3 GBfp16 — full precision · pirate

1.2 GB8-bit — pirate survives

663 MB4-bit — tuning washed out

~2 sto convert a 1B model

The memory↔quality dial, measured. A small fine-tuning delta is fragile: 8-bit preserved the pirate voice, 4-bit's coarser rounding erased it. Base-model knowledge is far more robust to 4-bit than a thin LoRA delta is.

💡 The rule this gives you Quantize the base aggressively (4-bit) — its knowledge is redundant and survives. But when you've fine-tuned and the change is small, keep it at 8-bit (or fp16), or the rounding eats your work. This is why Lesson 4's default fuse (which re-quantizes to 4-bit) lost the pirate, and --dequantize saved it.

04 Picking a bit-width on purpose

Choose	When	Cost
4-bit	Default for running base models; you want max speed + min memory; fitting a big model on-device	small quality drop; can erase fine-tuning deltas
8-bit	Preserving a fine-tune; quality-sensitive tasks; you have the RAM	~2× the size of 4-bit
fp16 (no `-q`)	Format-converting HF→MLX without quality loss; a base to quantize later	largest; slowest

🎯 Your tangible win Take your fp16 pirate and produce a small, standalone, still-a-pirate model:

mlx_lm.convert --hf-path ~/projects/learn/public/courses/mlx-lm/practice/pirate-fused-fp16 --mlx-path ~/projects/learn/public/courses/mlx-lm/practice/pirate-8bit -q --q-bits 8

. You now have a 1.2 GB model you could serve (Lesson 3) or publish (--upload-repo) — entirely yours, made on your laptop.

05 Check yourself

Think first, then expand. (Want to convert a specific model you have in mind? Ask me and we'll size it for your RAM.)

Q: Why does 4-bit decode faster than 8-bit on your Mac?

A: Decode is memory-bandwidth-bound — most time goes to reading weights. 4-bit weights are half the bytes of 8-bit, so the GPU reads less per token. Smaller also frees unified memory for more context.

Q: You fine-tuned a model and want it small. What's the safe path, given what you saw?

A: Keep the fine-tune at 8-bit (or fp16) — 4-bit's rounding can erase a small LoRA delta. Fuse with --dequantize, then convert -q --q-bits 8. Test the behavior after quantizing to confirm it survived.

Q: Where do mlx-community's models come from?

A: From mlx_lm.convert — someone converts a base model (often with -q) and pushes it via --upload-repo. You can do the same with your own.

Q: What does convert without -q do?

A: A pure format conversion: Hugging Face → MLX at full precision (controlled by --dtype), no quantization. Useful as a clean fp16 base you'll quantize deliberately later.

06 Going deeper

mlx-lm README — the convert + quantization section (source of truth for flags).
mlx-community — see the naming convention (-4bit, -8bit) you now produce yourself.
MLX docs — the array/quantization primitives underneath.

Keep handy: Glossary & quick reference · ← Lesson 4. Next up → Lesson 6: mlx_lm.chat & the KV cache — multi-turn conversation, and the cache that keeps decode fast as context grows (the last big runtime concept).