Lesson 5.2
Lazy Evaluation & Kernel Fusion
The step that turns a clone into a real framework. Instead of computing each operation the instant you write it, record the plan and optimize the whole thing before running it once, fast.
Eager vs lazy.
v1 is eager: each operation runs the moment you write it. Simple and great for debugging, it's how PyTorch works by default. But computing every small step immediately means lots of separate trips to memory and no chance to see the bigger plan.
Lazy evaluation flips it: writing an operation only records it; nothing computes until you actually ask for a result (tinygrad spells this .realize()). Now the framework holds the entire plan, the whole op graph, before running anything.
Five operations, five trips to memory, or one
eager runs each op separately; a lazy graph lets the compiler fuse them into one pass, far fewer trips to memory
Why holding the plan is powerful.
Once a compiler can see the whole graph, it can optimize it. The big win is kernel fusion: instead of running ten small operations as ten separate passes over memory, fuse them into one combined operation that does the whole chain in a single pass. Fewer trips to memory, no throwaway intermediate arrays, and one tight routine the GPU can run flat out. The same trick optimizes the backward pass too.
This is the genuinely senior, least-cloned part. Plenty of people write a micrograd clone; far fewer build a lazy graph and a fusing compiler. It's also the line where monktensor stops being an educational toy and becomes a real framework, the same idea behind tinygrad, torch.compile, and JAX.
The arc, end to end.
That's the whole climb: v1 (scalars) to understand how learning works, tensors to make it practical, lazy evaluation and fusion to make it fast and genuinely original. The deeper research bets in this body of work, vector search and quantization, build on exactly this understanding. You started with one number and a nudge; you end with a real framework and the judgment to push it further.
Key takeaways
Eager runs each op immediately (simple); lazy records the op graph and computes only on demand.
Seeing the whole graph lets a compiler fuse many ops into one fast pass, kernel fusion.
Fusion is the senior, least-cloned part, where monktensor becomes a real framework, not a toy.