PyTorch · Quick Reference

PyTorch Glossary

The vocabulary used across every lesson. Terms marked later are taught in upcoming lessons — the definition here is stable, so you can peek ahead.

A tensor is the one data structure in PyTorch: a typed, n-dimensional array that always knows three things — its shape, its dtype, and its device. Read those three and you can read almost any line of PyTorch.

🧱 Tensor anatomy

tensor: A typed, n-dimensional array — the single data structure everything in PyTorch is built from. Inputs, weights, activations, gradients are all tensors. torch.tensor([[1., 2., 3.], [4., 5., 6.]])
rank / ndim: How many dimensions a tensor has. 0 = scalar, 1 = vector, 2 = matrix, n = n-d tensor. Read it as x.ndim.
shape (size): The tuple of sizes along each dimension, as a torch.Size. The most-checked property in real code. x.shape or x.size(). x.shape # torch.Size([2, 3]) → 2 rows, 3 cols
dim (axis): A single dimension, identified by index (0 = first/outermost). The argument you pass to ops like sum(dim=0) to say which axis to act along. Negative indices count from the end (-1 = last).
dtype: The element type: torch.float32 (default for floats), float16/bfloat16, int64 (default for ints), bool, etc. Mismatched dtypes are a common error. x.dtype.
contiguous: Whether a tensor's elements sit in one unbroken block of memory in row-major order. Some ops (and view) require it; call x.contiguous() to force it.

⚙️ Operations

elementwise op: Applied independently to every element; shapes must match (or broadcast). + - * /, torch.exp, x.relu()… x + 10 x * x
broadcasting: Automatic expansion of compatible-but-unequal shapes in elementwise ops, without copying data. Trailing dims must be equal or one of them must be 1. (2,3) + (3,) → (2,3) (2,3) + (2,1) → (2,3)
matmul / @: Matrix multiplication (not elementwise). Inner dims must agree: (m,k) @ (k,n) → (m,n). Also torch.matmul.
reduction: Collapses one or more dims to fewer values: sum, mean, max, argmax. Pass dim= to choose the axis to collapse; omit it to reduce the whole tensor to a scalar. x.sum(dim=0) # collapse rows → one value per column
view / reshape: Reinterpret the same data under a new shape. view never copies (needs contiguous data); reshape copies only if it must. Use -1 to infer one dimension. x.view(3, 2) x.view(-1) # flatten
in-place op: A method ending in _ that mutates the tensor instead of returning a new one: add_, zero_, mul_. Saves memory but can break autograd — use deliberately.

💾 Devices & memory

device: Where a tensor's data physically lives: cpu, mps (Apple GPU, shown as mps:0), or cuda (NVIDIA). Ops require all operands on the same device. x.device.
.to(device): Move/copy a tensor (or model) to a device. The portable idiom: device = "mps" if torch.backends.mps.is_available() else "cpu" x = x.to(device)
MPS: Metal Performance Shaders — PyTorch's Apple-Silicon GPU backend. Bundled in the macOS wheel; check with torch.backends.mps.is_available(). Not every op is implemented yet — set PYTORCH_ENABLE_MPS_FALLBACK=1 to fall back to CPU for the gaps.
NumPy bridge: t.numpy() and torch.from_numpy(a) convert between a CPU tensor and a NumPy array, sharing the same memory (change one, the other changes). GPU tensors must be moved to CPU first.

🔁 Autograd later

requires_grad: A flag on a tensor: when True, PyTorch records the operations done to it so it can later compute gradients. Parameters have it set automatically.
computational graph: The dynamic DAG of operations PyTorch builds as you compute (define-by-run). It's what backward() walks to get gradients, and it's rebuilt every forward pass.
backward(): Called on a scalar (usually the loss). Walks the graph in reverse, applying the chain rule, and deposits gradients into the .grad of every leaf tensor with requires_grad=True.
grad: The gradient accumulated on a tensor by backward() — same shape as the tensor. Gradients accumulate (add up) across calls, which is why training loops zero_grad().
leaf tensor: A tensor you created directly (e.g. a parameter), as opposed to one produced by a tracked op. Only leaves with requires_grad=True get a populated .grad.
no_grad(): A context manager (with torch.no_grad():) that turns off graph tracking — used during inference and when manually updating parameters, to save memory and avoid recording those ops.

🧠 Model & training later

nn.Module: Base class for every model and layer. Holds parameters and defines forward(). You subclass it; calling the instance runs forward.
parameter: A learnable tensor owned by a module (nn.Parameter, requires_grad=True by default). model.parameters() hands them to the optimizer.
forward pass: Computing outputs from inputs through the model. (The backward pass is autograd computing gradients of the loss w.r.t. parameters.)
loss function: A scalar measuring how wrong the predictions are (e.g. nn.MSELoss, nn.CrossEntropyLoss). You call .backward() on it.
optimizer: Updates parameters from their gradients (torch.optim.SGD, Adam). step() applies one update; zero_grad() resets gradients to zero for the next iteration.
training loop: The canonical five moves, repeated: forward → loss → backward → step → zero_grad. You'll see this exact shape in essentially every PyTorch codebase.
Dataset / DataLoader: Dataset = an indexable source of samples; DataLoader = a batched, optionally shuffled iterable over a dataset. One epoch = one full pass; a batch = the subset processed per step.