Lesson 1.1
Computation as a Graph
Every calculation, no matter how big, is small operations wired together. Draw that wiring and you get a graph. That graph is the framework's memory of how each number was made.
A recipe, drawn out.
Think of a recipe: chop the onions, fry them, add tomatoes, simmer. Each step takes some ingredients and makes something new, which feeds the next step. Draw it as boxes with arrows and you have a flowchart. A computation is exactly this, small operations, each taking inputs and producing an output that flows into the next. That flowchart is a computation graph.
In the graph, each node is a value produced by one operation, and each edge is "this value feeds into that operation." The leaves (no incoming edges) are your starting numbers, inputs and parameters. The root at the far end is the final number you care about, which during training is the loss.
A tiny example.
Take e = (a × b) + c. Two operations: a multiply, then an add. As a graph:
a and b flow into the multiply to make d; d and c flow into the add to make e. Three leaves, two operations, one output. Big models are this same picture with millions of nodes.
Build e = (a × b) + c
each operation adds itself to the graph as it runs, define-by-run
you write ordinary math; the graph records itself underneath
Built as you go (define-by-run).
Here's the design choice that makes this pleasant to write: the graph is built while you do the arithmetic. Every time you multiply or add two values, the result quietly records what made it, its inputs and which operation. You write ordinary-looking math; the graph assembles itself underneath. This is called a define-by-run (or dynamic) graph, and it's how PyTorch works.
Why bother recording all this? Because to answer the nudge question for every leaf later, you need to know exactly how each value was produced and what fed it. The graph is the "machine that remembers its arithmetic" from lesson 0.1. The forward pass fills it in; the backward pass walks it in reverse.
Key takeaways
A computation graph is the wiring of a calculation: nodes are values, edges are "feeds into."
Leaves are inputs and parameters; the root is the output (the loss during training).
The graph is built as you compute (define-by-run) and is the memory backprop walks backward.