Show notes
New blackboard lecture with Reiner Pope: how do chips actually work - starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do. Reiner is CEO of MatX, a new chip startup (full disclosure - I’m an angel investor). He was previously at Google, where he worked on software efficiency, compilers, and TPU architecture. Watch this one on YouTube so you can see the chalkboard. Read the transcript.
Highlighted moments
almost all of the cost, like seven eighths of the cost is in the reading and writing the register file. And only a tiny fraction of the cost is in the logic unit itself.
“the thing that does not have an equivalent in a GPU is the, um, sort of this branch predictor. And so there is a whole big area in the CPU, which is, uh, sort of, uh, just a whole bunch of predictors that are saying, um, when will my next branch be and where the, where's the branch target for that?”
“whereas if you sort of look at the equivalent thing here, you've got vector units everywhere and you need to move data through this line, through this line, through this line, through this line, through this line, through this line. So the amount of data you can move between a vector unit and a matrix unit is actually much higher in, in a GPU than in a, in a TPU”
Transcript
Transcript not available for this episode yet.
More from Dwarkesh Podcast
8 Predictions for the Era of Continual Learning
Aug 7, 20268 min
Why smarter AI models could drive up compute prices 10x
Aug 3, 202611 min
Adam Brown – A deep but accessible introduction to general relativity
Jul 10, 20261h 38m
Grant Sanderson – AI and the future of math
Jun 30, 20261h 33m
The next big breakthrough will be AIs learning on the job
Jun 26, 202619 min