Lesson 5.1
Scalars to Tensors
v1 ran on single numbers so every step was visible. Real frameworks run on arrays, tensors, so they can be fast. The good news: the autograd you learned doesn't change, it just works on arrays now.
Why scalars don't scale.
A scalar engine makes one graph node per number. That's perfect for learning and hopeless for size: a network with a million weights would build a million nodes and crawl through them one at a time. Real models need to push thousands of numbers through the same operation at once.
The answer is the tensor, an n-dimensional array of numbers treated as a single value. One tensor operation does what thousands of scalar ones would, and it maps onto hardware (CPU vector units, GPUs) built exactly for bulk number-crunching.
One operation, on one number, or on many at once
a tensor runs the same operation across a whole array in one go, that's what makes a framework fast
The two ideas you'll add.
Broadcasting
Rules for combining arrays of different shapes without writing loops, e.g. adding one bias value to every row of a matrix. It's how a layer applies the same operation across a whole batch.
Matrix multiply (matmul)
The workhorse of a layer: a whole layer's worth of weighted sums is one matmul of the inputs against the weight matrix, plus the bias. One operation replaces a forest of scalar multiplies and adds.
tinygrad makes a clarifying observation here: nearly everything reduces to three kinds of operation, elementwise (add, multiply, relu), reduction (sum, max), and movement (reshape, transpose). Build those well and you can express any network.
Your autograd still holds.
Here's the payoff for learning it on scalars: the autograd logic is identical. Each node now holds an array instead of a number, and each local gradient is an array operation instead of a single multiply, but it's still "forward builds the graph, backward multiplies local gradients and accumulates." The chain rule doesn't care whether the values are numbers or matrices. You're vectorizing what you already understand, not learning a new idea.
Key takeaways
Tensors (n-dim arrays) let one operation process many numbers at once, which is what makes frameworks fast.
Broadcasting and matmul are the two new pieces; a layer becomes one matmul plus a bias.
The autograd you built is unchanged, values become arrays, the chain rule and accumulation stay the same.