Everything from Lessons 1–5 in one program: a 2-layer neural network that classifies points, trained on your Apple GPU, scored on a held-out set. We go from 45% to 100% accuracy on a problem no straight line can solve.
🎯 Mission: read & write real PyTorchA "deep" net is just linear layers stacked with a nonlinearity between them. The nonlinearity is the whole point: without it, ten stacked layers collapse into one line. Add it, and the network can carve curved decision boundaries. The loop, the loss, the optimizer, the data pipeline — all the machinery you've already built — stays exactly the same. Only the model grows.
Our data: an inner disk (class 0) sitting inside an outer ring (class 1). No straight line separates a disk from the ring around it — so nn.Linear alone (Lesson 4) is hopeless here. We need a real network.
This is the payoff lesson: it combines the tensors, autograd, loop, modules/optimizers, and data pipeline you already know into the standard shape of real PyTorch model code.
ReLUStack two linear layers with nothing between them and the math collapses: W₂(W₁x) = (W₂W₁)x is still just one linear layer. To get more power you insert a simple nonlinear function between layers. The workhorse is ReLU: relu(z) = max(0, z) — keep positives, zero out negatives.
Source: torch.nn.ReLU · Build the model tutorial.
2 → 16 → ReLU → 2nn.Sequential chains layers — the output of each feeds the next. Two inputs (the x,y coordinates), a 16-unit hidden layer with ReLU, then 2 outputs (one score per class):
model = nn.Sequential(
nn.Linear(2, 16), # 2 features → 16 hidden units
nn.ReLU(), # the bend that makes it nonlinear
nn.Linear(16, 2), # 16 → 2 logits (one per class)
).to(device)
print(sum(p.numel() for p in model.parameters())) # 82 params
CrossEntropyLossFor predicting a number we used MSE. For predicting a class, the standard loss is cross-entropy. It takes the raw output scores (logits) and the integer class labels:
loss_fn = nn.CrossEntropyLoss()
loss = loss_fn(logits, labels) # logits: (N, 2) floats ; labels: (N,) ints (0 or 1)
CrossEntropyLoss applies log-softmax internally — feed it the raw logits. Adding your own softmax first is a classic bug. (2) Labels are integer class indices of dtype long with shape (N,) — not one-hot vectors, not floats. The output layer has one unit per class.Source: torch.nn.CrossEntropyLoss.
Two idioms appear in every real training script. First, move the model and every batch to the device. Second, flip the model between train() and eval() modes, and wrap evaluation in no_grad().
model.train() # training mode for xb, yb in train_loader: xb, yb = xb.to(device), yb.to(device) # batch → GPU (must match model) ... @torch.no_grad() # no graph needed for evaluation → faster, less memory def accuracy(loader): model.eval() # eval mode ... pred = model(xb).argmax(dim=1) # highest-logit class correct += (pred == yb).sum().item()
train() / eval() matter
Some layers (dropout, batch-norm) behave differently when training vs. predicting. Our tiny net has none, so it doesn't change the numbers — but writing model.train() / model.eval() is a habit worth building now, because forgetting it on a real model silently wrecks your accuracy. Accuracy itself = "what fraction did argmax get right."The loop is the same five moves over batches, plus an accuracy check each pass. Real output, on device = mps:
device = mps model params: 82 val accuracy before training: 45.0% epoch 0 | train loss 0.4165 | val acc 90.6% epoch 10 | train loss 0.0085 | val acc 100.0% epoch 20 | train loss 0.0010 | val acc 100.0% epoch 30 | train loss 0.0011 | val acc 100.0%
Before training the net guesses at chance (~45–50%). Within a few epochs the hidden layer has bent a circular boundary around the disk and it nails the held-out set. An untrained random net scoring ~50% then climbing is exactly the signal that learning is happening.
~/projects/learn/public/courses/pytorch/practice/.venv/bin/python ~/projects/learn/public/courses/pytorch/practice/lesson06_real_net.pyThis program is the shape of real PyTorch. Every block traces to a lesson:
Features and integer labels as tensors (L1), wrapped in a Dataset + DataLoader with a train/val split (L5).
nn.Sequential of Linear + ReLU layers (L4), moved .to(device) (L1).
CrossEntropyLoss + Adam(model.parameters()) (L4).
zero_grad → forward → loss → backward → step over batches (L2–L5), with train()/eval().
ReLU between two Linear layers?CrossEntropyLoss?.to("mps") but pass a CPU batch. What happens?@torch.no_grad() and model.eval()?ReLU = max(0,z)) between linear layers. Without it, any stack of Linear layers reduces to one linear layer — only straight-line decisions.CrossEntropyLoss expects?(N, C) — no softmax in the model (it's applied internally). (2) Integer class indices shape (N,), dtype long — not one-hot.model.to(device) and move every batch with xb.to(device). Model and data must share a device or you get a runtime error.model.train() and model.eval() do?eval() (and no_grad()) when measuring/predicting.logits.argmax(dim=1). Accuracy = fraction where that equals the label: (pred == y).float().mean().Try: "remove the ReLU and show accuracy stays at chance," "add a second hidden layer," or "plot the decision boundary." One lesson left: how to save this trained model, load it back, and read a real-world training script end to end. Say "go" for Lesson 7.