One command turns the model from Lessons 1–2 into an OpenAI-compatible HTTP server. Any tool, SDK, or script that already speaks the OpenAI API can now point at localhost and talk to a model running on your M4 Pro — no key, no cloud, no per-token bill. This is the direct route to wiring a local model into your own tools.
mlx-lm speaks the OpenAI API, so "switch to a local model" is a one-line change: the base_url. Your existing OpenAI code — the official SDK, LangChain, curl, whatever — keeps working unchanged; you just aim it at http://localhost:8080/v1 instead of api.openai.com. The model moves from a data center to your desk; your code doesn't notice.
You said the goal is to wire local models into your own tools. This is it. Lesson 2 made the model callable from one Python process; a server makes it callable from anything on your machine — other scripts, apps, notebooks, even the learn pipeline — over plain HTTP, using the most widely-supported LLM interface that exists. Build against the OpenAI API once; run it locally or in the cloud by flipping a URL.
mlx_lm.server hosts your model behind OpenAI-shaped endpoints (/v1/chat/completions, /v1/models). Point any OpenAI client at it with a fake key. Done.~/projects/learn/public/courses/mlx-lm/practice/.venv/bin/mlx_lm.server \ --model mlx-community/Llama-3.2-1B-Instruct-4bit \ --port 8080
Real startup on your machine:
UserWarning: mlx_lm.server is not recommended for production... # ← see §06 INFO - Starting httpd at 127.0.0.1 on port 8080... Fetching 6 files: 100%|██████████| 6/6 # model already cached from L1 → instant
It's just HTTP — no SDK required. List what's loaded:
curl -s http://127.0.0.1:8080/v1/models
{ "object": "list",
"data": [{ "id": "mlx-community/Llama-3.2-1B-Instruct-4bit",
"object": "model" }] }
And the real one — a chat completion (note: you send messages, the server applies the chat template from Lesson 2 for you):
curl -s http://127.0.0.1:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"mlx-community/Llama-3.2-1B-Instruct-4bit", "messages":[{"role":"user","content":"Name three primary colors in one line."}], "max_tokens":40, "temperature":0.0}'
# real response from your M4 Pro: { "object": "chat.completion", "system_fingerprint": "0.31.3-0.31.2-macOS-26.5.1-arm64-applegpu_g16s", "choices": [{ "finish_reason": "stop", "message": { "role": "assistant", "content": "Red, Blue, Yellow." }}], "usage": { "prompt_tokens": 43, "completion_tokens": 7, "total_tokens": 50 } }
choices[].message.content, finish_reason, usage — byte-for-byte what api.openai.com returns. That's why any OpenAI client just works. The system_fingerprint even leaks your stack: mlx-lm 0.31.3, mlx 0.31.2, Apple GPU.Here's the moment it clicks. Take ordinary OpenAI-SDK code and change two things — the base_url and a throwaway api_key:
from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="not-needed") # ← the only changes resp = client.chat.completions.create( model="mlx-community/Llama-3.2-1B-Instruct-4bit", messages=[{"role": "user", "content": "Give me a haiku about Apple Silicon."}], max_tokens=60, temperature=0.7, ) print(resp.choices[0].message.content)
# real output, generated locally:
Silicon souls cry
Bleeding Bayes and Neural
Beauty in code
pip install openai (already done in your venv), and run ~/projects/learn/public/courses/mlx-lm/practice/lesson03_client.py. When a haiku comes back, you've run OpenAI-SDK code with zero dollars and zero network calls leaving your machine. Every OpenAI tutorial on the internet now runs against your model.Same stream=True you'd use against OpenAI — the server sends Server-Sent Events, the SDK reassembles them into deltas:
stream = client.chat.completions.create(
model="mlx-community/Llama-3.2-1B-Instruct-4bit",
messages=[{"role": "user", "content": "Count 1 to 5."}],
max_tokens=40, stream=True,
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="", flush=True) # ✓ streamed on your Mac
This is the HTTP twin of Lesson 2's stream_generate: the server runs stream_generate internally and forwards each token as an SSE chunk.
base_url points home. Swap localhost:8080 for api.openai.com and the very same code runs against the cloud — that's the portability you're buying.| Endpoint | What it does | OpenAI parity |
|---|---|---|
POST /v1/chat/completions | Chat with messages (+ stream, temperature, max_tokens…) | ✓ drop-in |
POST /v1/completions | Raw text completion (no chat template) | ✓ |
GET /v1/models | List the loaded model id | ✓ |
mlx_lm.server does only basic security checks and isn't hardened for the public internet. It's perfect for local dev, your own tools, and a trusted LAN — not for exposing to the world. For production, put a real gateway in front or use a hardened serving stack.127.0.0.1 — reachable only from your Mac. To reach it from your phone or another machine on your network, start it with --host 0.0.0.0 (and understand you've now opened it to your LAN). You can also swap models per request, or run with --model unset and pass the model id in each call.Think first, then expand. (Want to point a specific tool of yours at this? Ask me — I'll help you wire it.)
A: Set base_url="http://127.0.0.1:8080/v1" and any non-empty api_key on the client. The rest of the code — chat.completions.create(...), streaming, usage — is unchanged, because mlx-lm returns the OpenAI response shape.
messages to /v1/chat/completions — do you need to apply the chat template yourself like in Lesson 2?A: No. The server applies the model's chat template internally for the chat endpoint. You send role/content messages; it handles the special-token formatting. (The raw /v1/completions endpoint does not — that's for base-model-style raw text.)
A: The first request loads the weights into unified memory; the server keeps the model warm, so subsequent requests skip loading and go straight to inference. One warm server serves many clients.
--host 0.0.0.0 on coffee-shop wifi?A: No. That exposes an unauthenticated, not-production-hardened server to everyone on that network. Keep it on 127.0.0.1 except on networks you trust, and never on the open internet without a real gateway in front.
Keep handy: Glossary & quick reference · ← Lesson 2. Next up → Lesson 4: mlx_lm.convert — stop borrowing mlx-community's pre-quantized models and make your own: pull any Hugging Face model and quantize it to 4/8-bit, trading memory for speed on purpose.