Imagine answering a question about a book, then rebuilding your entire set of reading notes before writing the next word. And again before the word after that.
A language model generating text faces a related efficiency problem: each new token needs information about the context that came before it. Recalculating everything from scratch would repeat a great deal of work.
The KV cache, short for key-value cache, keeps certain intermediate attention results available for reuse. It helps an AI model continue generating without repeatedly rebuilding those results for earlier tokens.
The bargain is simple: save computation by spending memory. Understanding that bargain explains both faster responses and the enormous memory demands of long conversations.
What Does a KV Cache Actually Store?
In a typical autoregressive transformer, attention uses numerical representations called queries, keys, and values:
- A query helps determine which earlier information is relevant to the position being processed.
- Keys provide representations against which that query is compared.
- Values provide the information combined according to the resulting attention weights.
These are learned vector representations, not ordinary database keys paired with text answers. Our transformer overview introduces the surrounding architecture.
During generation, the model stores previous tokens’ key and value tensors for the relevant attention layers. The new position supplies its query and attends to the stored keys and values, alongside the new ones.
With causal attention, adding a later token does not alter how earlier positions were computed from their preceding context. For an unchanged prefix, that makes their stored results reusable. The cache keeps K and V because future positions need them; it does not need a growing archive of previous queries for this purpose.
From Reading the Prompt to Writing the Answer
Two phases help explain what happens when you submit a prompt.
Prefill processes the input tokens and builds their cached attention state. Many prompt positions can be processed in parallel, subject to the causal mask.
Decode continues the sequence, normally generating one new token per sequence at each autoregressive step. The model uses the existing cache and adds the new position’s key-value state as generation advances.
The prompt is therefore not the same workload as the stream of tokens that follows it. A long input can involve substantial processing before the first output appears, even when later generation uses a cache.
Tokens are units of model input and output; they need not correspond to whole words. The broader prediction process is explained in our guide to large language models.
Reusing Work Does Not Make the Next Token Free
Suppose a model has processed 1,000 tokens and is extending the sequence. It can reuse the keys and values already calculated for those positions instead of deriving them all again.
But the new query still has to interact with the relevant cached context. The model also performs other computations, including its feed-forward layers and output prediction.
For ordinary full attention, the attention work for a new decoding position grows with the number of cached positions. Caching avoids repeated computation; it does not turn an arbitrarily long conversation into a fixed-cost operation.
There is also a hardware consequence. Reading a large cache consumes memory bandwidth. A GPU can have abundant arithmetic capability while moving cached data becomes a significant constraint on token generation.
How a Cache Becomes Several Gigabytes
For a straightforward full-attention cache, a useful payload estimate is:
KV bytes = 2 × layers × cached tokens × KV heads × head dimension
× bytes per stored element × batch sizeThe initial factor of two accounts for both keys and values. Use the number of KV heads, which can differ from the number of query heads.
Consider a hypothetical configuration with 32 layers, 8 KV heads, a head dimension of 128, two-byte stored elements, and one sequence:
| Cached tokens | Calculated KV payload |
|---|---|
| 4,096 | 512 MiB |
| 8,192 | 1 GiB |
| 16,384 | 2 GiB |
| 32,768 | 4 GiB |
For the 16,384-token example, the calculation is 2 × 32 × 16,384 × 8 × 128 × 2 = 2,147,483,648 bytes.
These are calculated illustrations, not measurements of a particular model. They exclude model weights, other activations, workspace, allocation overhead, and implementation-specific details. Architectures with sliding windows, compressed representations, or different attention mechanisms need different accounting.
Batching several independent sequences increases cache demand. Consequently, an AI service can run into memory pressure because of the contexts it is actively serving, even after the model weights already fit on the accelerator.
The Different Tricks for Managing That Memory
Several techniques tackle different parts of the problem:
- Grouped-query attention and multi-query attention let multiple query heads share key-value heads, reducing the number of KV heads that must be stored.
- Cache quantization stores cache elements using fewer bits, trading memory savings against additional processing and possible numerical effects.
- Offloading moves some cache data to another memory tier, such as CPU memory. It can relieve GPU memory pressure while introducing transfer costs.
- Sliding-window attention restricts the positions retained or consulted in relevant layers. This is an architectural choice, not a universally safe way to truncate any model’s cache.
PagedAttention addresses allocation. Its block-based organization lets serving systems manage KV storage more flexibly and reduce wasted allocation. It does not, by itself, change how many numerical values an ordinary token requires in a given cache format.
The techniques are complementary in some systems, but none offers a universal speedup for every model, sequence length, and device.
Can Two Conversations Reuse the Same Cache?
Sometimes they can reuse a common beginning. Prefix caching allows a serving system to reuse previously computed state for a compatible, identical token prefix—for example, a repeated instruction block.
Matching a passage somewhere inside two prompts is not generally sufficient. A token’s representation depends on its preceding context, so moving that passage can change the state that would need to be cached.
Reuse also depends on model compatibility, positional handling, and the serving system’s cache policy. A chat interface may preserve relevant state between turns, reuse a prefix, or recompute it after eviction. The visible conversation alone does not reveal which strategy is used.
Prefix reuse primarily helps avoid repeated prompt processing. It does not eliminate the work required to generate a new answer.
The Cache Is Not the Model’s Personality or Knowledge
A KV cache is inference state. It is distinct from the model’s learned weights, a database of personal memories, retrieval-augmented generation, and a cache of completed responses.
Ordinary use of a KV cache does not train the model or update its weights. Nor does the concept itself establish how a service retains chat logs or handles user data; those are separate implementation and policy questions.
It is also different from speculative decoding. KV caching reuses attention state, while speculative decoding proposes candidate tokens and verifies them. A serving system can use both.
The Trade-Off Behind a Smooth Conversation
The next time a model streams an answer, think of a growing set of reusable working notes. The model still performs substantial work for each new token, but it can build on intermediate results it already has.
That is the purpose of the KV cache: avoid rebuilding useful attention state while accepting the cost of keeping it available.
Faster AI is often a story about memory—what to retain, where to place it, and how efficiently to read it again.
References and Further Reading
- Hugging Face: How caching works — queries, keys, values, and reuse during generation.
- Hugging Face: Cache strategies — dynamic, static, quantized, and offloaded caches.
- Hugging Face TGI: PagedAttention — block-based KV storage.
- Hugging Face: Prefix caching — reusing a compatible shared prompt prefix.
