A model loads successfully, the first user gets an answer and then the server starts running out of GPU memory as conversations grow. The model weights have not changed. What has grown is the active state kept for those requests, often dominated by the key-value (KV) cache.

Why a transformer keeps a cache

During autoregressive generation, a decoder produces a new token using the context that came before it. Retaining the keys and values calculated for earlier tokens avoids recomputing those representations on every step. Hugging Face's cache explanation describes how the past state is reused.

The KV cache is not the model's learned parameters or an external vector database. It is intermediate attention state associated with the current sequence. As a prompt is processed and new tokens are generated, a conventional full-attention decoder accumulates more of that state.

This also explains why shrinking the model weights does not necessarily shrink the cache. The cache can use a different data type from the weights.

Calculate memory per cached token

For a decoder with uniform full-attention layers, a simplified estimate of raw cache storage is:

Bytes = 2 × layers × KV heads × head dimension × cached tokens × bytes per element.

The first factor accounts for both keys and values. Use KV heads rather than query heads when the architecture shares key-value information. The calculation assumes every layer has the same dimensions and stores the same sequence length.

Take a hypothetical model with 32 layers, eight KV heads, a head dimension of 128 and two-byte elements. One token requires 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB.

That yields:

  • 4,096 cached tokens: 512 MiB per sequence.
  • 8,192 cached tokens: 1 GiB per sequence.
  • 32,768 cached tokens: 4 GiB per sequence.
  • 131,072 cached tokens: 16 GiB per sequence.

Sixty-four independent sequences at 8,192 tokens each would require 64 GiB of raw cache tensors. On an accelerator with 80 GiB of usable memory, that leaves only 16 GiB before accounting for weights, workspaces and other runtime allocations. Actual available memory is also affected by the runtime's allocator and device reservations.

These numbers describe a simplified example, not the measured requirements of a named model. Different attention architectures can change the formula substantially.

Count KV heads, not just model parameters

Multi-head attention can have separate keys and values for every query head. Multi-query attention shares key-value information more broadly. Grouped-query attention (GQA) lies between those approaches; see the original GQA paper.

In the previous example, changing eight KV heads to 32 would quadruple the raw cache size if every other dimension remained the same. Two models with a similar parameter count can therefore have quite different serving footprints. Sliding-window attention and other state representations require different calculations.

Hugging Face's cache-strategy documentation also describes dynamic, static, quantized and offloaded caches. A static allocation may reserve more space in advance; offloading trades device memory for transfers. Compatibility depends on the model and runtime.

Allocation and prefix sharing

A runtime does not always use the raw tensor total efficiently. The PagedAttention research describes a paged approach designed to reduce fragmentation and allow sharing. That can leave more of the same physical GPU memory available for active sequences.

Prefix caching can reuse state for matching initial tokens, such as a common system prompt. A similar-looking prompt is not sufficient: the actual token sequence and implementation must support reuse. Independent requests cannot be assumed to share their entire cache.

Quantizing the cache from two bytes per element to one would halve the raw tensor storage in the simplified example, before scale metadata and other overhead. The vLLM FP8 cache analysis discusses the accuracy and performance implications of that choice. Verify the supported format and task quality rather than assuming the smaller cache is a free improvement.

Why advertised context length is not concurrent capacity

A model might support a 128k-token context while a particular deployment can only serve a small number of such requests at once. Long outputs also extend cache occupancy and keep it allocated for longer. A maximum context limit is not a promise that every user can fill that window simultaneously at the same latency.

When sizing a service, estimate the cache footprint alongside weights and runtime overhead, then use actual prompt and output distributions to test concurrency. Report both how many sequences fit and how quickly the system serves them. Training versus inference economics explains why those measurements influence the cost per useful response.