Dr Shuangyi Yan

Jeff Dean (@JeffDean) says a multiply on an AI chip costs about one picojoule, but moving the data to it can cost about 1,000. "It's really good to think about calculations you're doing in a model either for training or inference." "Often a good way to view that is how much state will you need to bring in from memory, either like on-chip SRAM or HBM from the accelerator attached memory or DRAM or over the network, and then how expensive is that data motion relative to the cost of say an actual multiply in the matrix multiply unit. And that cost is actually really, really low." "Depending on your precision, I think it's like sub-pico, one picojoule." "Moving data from the SRAM on the other side of the chip, not even off chip, but on the other side of the same chip can be, you know, a thousand picojoules." "This is why your accelerators require batching. Because if you move, like, say, the parameter of a model from SRAM on the chip into the multiplier unit, that's going to cost you a thousand picojoules. So you better make use of that thing that you moved many, many times." "If you have a batch of 256 or something, that's not so bad. But if you have a batch of one, that's really not good." "Ideally you'd like to use batch size one because the latency would be great, but the energy cost and the compute cost inefficiency that you get is quite large." - Jeff Dean, Google chief scientist at the time of the February 2026 interview, on Latent Space with @swyx