This question was asked in AI Engineering interviews at OpenAI, xAI, Mistral AI, Amazon, Apple, NVIDIA, Together AI, and Character .AI. Question: What is the KV cache, and what are its memory implications at scale? Derive the formula. Let's answer it step by step. KV Cache is a memory where an LLM saves the Key and Value of the tokens it has already processed, so that it does not compute them again for every new token. An LLM generates text one token at a time. To predict the next token, the attention layer needs the Key and Value of all the previous tokens. Without a cache, the model computes them again at every step. For 100 tokens, that is 2 + 3 + ... + 100 = 5,049 Key and Value computations. With KV Cache, it is 101. Why do we cache only Key and Value, and not Query? The Query is needed only for the current token. The Keys and Values of the past tokens are needed at every future step. Now, let's derive the memory formula. For one token, in one layer, in one head, we store a Key vector and a Value vector. Each has head_dim numbers. So: KV cache per token = 2 x layers x kv_heads x head_dim x bytes_per_number Here: - 2 is because we store both the Key and the Value - layers is the number of Transformer layers - kv_heads is the number of Key and Value heads - head_dim is the size of each head - bytes_per_number is 2 for FP16 Total KV cache = per token x sequence length x number of users in the batch Let's take a model close to Llama 2 7B: 32 layers, 32 heads, head_dim 128, FP16. Per token = 2 x 32 x 32 x 128 x 2 = 524,288 bytes = 0.5 MB - A chat of 4,000 tokens needs 2 GB. - A document of 100,000 tokens needs 50 GB, just for one user. - A GPU has a fixed memory, and a big part of it is already taken by the model weights. Now, the implications at scale: - The cache grows linearly with the length of the text and with the number of users. It decides how many users one GPU can serve at the same time. - For every new token, the model reads the full cache from memory. A bigger cache means slower generation. And this is how we reduce it, term by term: - kv_heads: Grouped-Query Attention shares one Key and Value head across many Query heads. Llama 2 70B uses 8 KV heads for 64 Query heads, which makes the cache 8 times smaller. - bytes_per_number: quantize the cache to FP8, and it becomes 2 times smaller. - Wasted memory: Paged Attention stores the cache in small blocks, so that memory is not reserved and left empty. - sequence length: keep a recent window of tokens plus the first few attention sinks. I have shared the detailed blogs in the comments.