"Learning" is just nudging numbers to make an error smaller. To know which way to nudge, you need the slope of the error with respect to each number. Autograd computes every one of those slopes for you, automatically.
🎯 Mission: read & write real PyTorchYou write the forward math (turn inputs into a prediction and an error). PyTorch silently records every operation into a graph. Call .backward() on the error and PyTorch replays that graph in reverse — the chain rule — depositing the gradient (slope) of the error with respect to every parameter into its .grad. That gradient is the arrow pointing "downhill"; following it is how every model on earth learns.
A neural net is a pile of numbers (its parameters) and some math that turns an input into an output. Training means repeatedly asking: "how wrong was I, and which way should I tweak each parameter to be a little less wrong?" The answer to "which way" is a gradient — the slope of the error.
For one number, the slope is just calculus you may remember: the slope of x² is 2x. But a real model has millions of parameters tangled through many operations. Computing all those slopes by hand is hopeless. Autograd does it for you — exactly, and in one call. This is the feature that makes PyTorch PyTorch, and you'll see its fingerprints (requires_grad, loss.backward()) in every training script you ever read.
requires_grad — telling PyTorch to watchBy default tensors are inert data. Set requires_grad=True and PyTorch starts recording every operation you do to that tensor, building a graph it can later differentiate. (This is "define-by-run": the graph is built as your normal Python code executes — no separate compile step.)
x = torch.tensor(3.0, requires_grad=True) y = x ** 2 print(y) # tensor(9., grad_fn=<PowBackward0>)
Notice the grad_fn=<PowBackward0> on y. That's the breadcrumb: y remembers it was born from a power operation, so it knows how to send a gradient back to x. Every tensor produced from a tracked tensor carries one of these.
Source: PyTorch — Autograd tutorial · Autograd mechanics.
.backward() and .grad — it really is just calculusCall .backward() on a scalar (a single number — usually the loss). PyTorch walks the graph in reverse and fills in the gradient on every leaf tensor that asked to be watched.
x = torch.tensor(3.0, requires_grad=True) y = x ** 2 y.backward() # compute dy/dx print(x.grad) # tensor(6.) ← exactly 2x at x=3
It's the real derivative. Try a polynomial — the slope of 3x² + 2x + 1 is 6x + 2, which is 14 at x=2:
x = torch.tensor(2.0, requires_grad=True) f = 3*x**2 + 2*x + 1 f.backward() print(x.grad.item()) # 14.0
Now the version that matters for training: gradients of a loss with respect to parameters w and b.
loss.backward() flows right→left, leaving a gradient on each parameter.w = torch.tensor(2.0, requires_grad=True) b = torch.tensor(1.0, requires_grad=True) pred = w * 4.0 + b # 9.0 loss = (pred - 0.0) ** 2 # 81.0 (squared error vs target 0) loss.backward() print(w.grad) # tensor(72.) how loss changes per unit of w print(b.grad) # tensor(18.) how loss changes per unit of b
w.grad = 72 reads as: "right now, nudging w up by 1 would increase the loss by ~72." So to decrease loss, move w in the opposite direction (down). Bigger gradient → steeper slope → that parameter matters more here. .grad always has the same shape as the tensor.Each call to .backward() adds into .grad rather than replacing it. Forget to reset, and this step's gradient is polluted by the last one.
p = torch.tensor(1.0, requires_grad=True) (p * 3).backward(); print(p.grad) # tensor(3.) (p * 3).backward(); print(p.grad) # tensor(6.) ← added, not replaced! p.grad.zero_(); print(p.grad) # tensor(0.) reset for the next step
zero_grad()
You'll always see gradients zeroed once per iteration (either optimizer.zero_grad() or p.grad.zero_()). If your model "won't learn," a missing zero is a prime suspect. (Why accumulate at all? It lets you sum gradients across several mini-batches on purpose — handy, but opt-in.)no_grad() and detach()Tracking costs memory and time, and there are moments you explicitly don't want it: when running a trained model for predictions (inference), and when manually updating parameters (you don't want the update itself recorded as part of the graph).
w = torch.tensor(5.0, requires_grad=True) with torch.no_grad(): # a block where nothing is recorded y = w * 2 print(y.requires_grad) # False d = w.detach() # a view of w with tracking stripped off print(d.requires_grad) # False
with torch.no_grad(): wraps inference loops and the parameter-update step. .detach() pulls a tensor "out of the graph" — common when logging a loss value or converting to NumPy for a metric. Both are everywhere in real code.Here's the entire idea in four lines — gradient, then a small step downhill. This is training, minus the loop:
w = torch.tensor(2.0, requires_grad=True) loss = (w * 4.0) ** 2 # want this near 0 loss.backward() # 1. get the slope: w.grad lr = 0.01 # learning rate: how big a step with torch.no_grad(): # 2. don't record the update itself w -= lr * w.grad # 3. step DOWNHILL (opposite the gradient) w.grad.zero_() # 4. reset for next time print(round(w.item(), 2)) # 1.36 ← moved from 2.0 toward 0
torch.optim replaces the hand-written step with optimizer.step().~/projects/learn/public/courses/pytorch/practice/.venv/bin/python ~/projects/learn/public/courses/pytorch/practice/lesson02_autograd.pyx.grad contain after loss.backward()?.backward() twice without zeroing in between. What happens to .grad?w.grad is 72. To make the loss smaller, which way do you move w?with torch.no_grad():?y prints with grad_fn=<PowBackward0>. What does that tell you?requires_grad=True do?.backward() on?.backward() then fills .grad on every leaf tensor with requires_grad=True..backward() accumulates into .grad instead of replacing it. Without zero_grad(), old gradients pollute the new step.no_grad() vs detach()?with torch.no_grad(): disables tracking for a whole block (inference, param updates). x.detach() returns a single tensor cut out of the graph (logging, NumPy conversion).Worth asking now: "draw the graph and gradients for a 2-layer example," "why does a bigger gradient mean that parameter matters more?", or "what actually happens if I forget zero_grad?" Next we stop doing single steps and write the real loop. Say "go" for Lesson 3.