Efficient Memory Management for Large Language Model Serving with PagedAttention

Worauf sich diese Seite beruft, wörtlich, abgerufen am 15.09.2026:

• the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamicallybestätigt 10.11.2026
• flexible sharing of KV cache within and across requests to further reduce memory usagebestätigt 10.11.2026
• we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memorybestätigt 10.11.2026
• the KV cache of a single token demands 800 KB of spacebestätigt 10.11.2026
• This sequential generation process makes the workload memory-bound, underutilizing the computation power of GPUs and limiting the serving throughput.bestätigt 10.11.2026
• this phase severely underutilizes GPU computation and becomes memory-boundbestätigt 10.11.2026
• High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time.bestätigt 10.11.2026
• Because the requests share the same model weights, the overhead of moving weights is amortized across the requests in a batchbestätigt 10.11.2026
• After each iteration, completed requests are removed from the batch, and new ones are added.bestätigt 10.11.2026
• only 20.4% - 38.2% of the KV cache memory is used to store the actual token states in the existing systems.bestätigt 10.11.2026
• one can think of blocks as pages, tokens as bytes, and requests as processes.bestätigt 10.11.2026
• with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca.bestätigt 10.11.2026
• fine-grained batching mechanisms, such as cellular batching (Gao et al., 2018) and iteration-level scheduling (Yu et al., 2022), have been proposedbestätigt 10.11.2026