LLM Inference Infrastructure
Explore how to optimize serving infrastructure for large language models by mastering KV caching, paged memory management, speculative decoding, and continuous batching. Understand how these techniques improve GPU memory efficiency, lower latency, and increase concurrency to make LLM deployment scalable and cost-effective.
In the previous lesson, you learned how quantization, pruning, and compilation shrink models and speed up individual forward passes. Those techniques apply broadly across ML workloads. But when you move to serving a large language model in production, a fundamentally different bottleneck emerges, one that no amount of weight pruning alone can solve.
Why LLM serving is a different problem
A ResNet classifier takes a fixed-size image, runs a single forward pass, and returns a prediction. The compute cost is deterministic and bounded. A GPT-style language model, by contrast, generates output through
This sequential dependency changes the hardware bottleneck entirely. During each decoding step, the model must reload its full set of attention parameters from GPU memory, but it only produces a single token’s worth of new computation. The ratio of arithmetic operations to bytes moved through memory drops dramatically. The GPU’s floating-point units are often underutilized because memory bandwidth limits how quickly weights and activations can be loaded. For many LLM inference workloads, especially autoregressive token generation, performance is often memory-bandwidth-bound rather than compute-bound.
Now imagine designing the serving backend for a conversational AI product where thousands of users send variable-length messages simultaneously. Each concurrent request carries its own growing state, and GPU-hours dominate the operating budget. The infrastructure innovations covered in this lesson exist precisely to make that scenario economically viable.
The following diagram contrasts these two serving paradigms visually:
With this fundamental distinction established, the next question becomes how to avoid redundant work across those hundreds of sequential forward passes.
KV caching and its role in latency
What the KV cache stores
During each attention computation, the transformer projects the input into ...