Continuous Batching: How AI Servers Swap Requests Between Tokens

By Editorial Team Published 6 min read
A loom feeds a blue ribbon beside a finished orange piece and a shuttle loaded with green thread.

One person asks a chatbot for the capital of Finland. Another asks for a detective story set there. A third request arrives while the detective is still looking for a train ticket.

The first answer is already finished. Does the server have to wait for the detective to solve the case before it can start the new request?

Continuous batching lets an AI server change which requests it works on between model iterations. A short request can finish, a new one can join, and the detective story can keep growing. The server shares the work of running the model while keeping each response separate. [1, 2]

What Is Continuous Batching in LLMs?

In a typical autoregressive language model, each new token depends on the tokens that precede it. The server cannot generate an ordinary response by calculating all of its future tokens independently. It can, however, calculate the next token for several independent requests in the same model pass. That is batching. [1]

Why bother? Some work, such as reading model weights for a computation, can be amortized across multiple requests. But requests finish at different times. If the server freezes the batch membership until the longest response ends, there may be little useful work left in that batch while other users are waiting.

Continuous batching changes the scheduling unit from the entire request to a model iteration. At scheduling boundaries, the server can retire finished requests, keep the unfinished ones, and admit pending work as capacity allows. The 2022 Orca paper describes this as iteration-level scheduling. [1]

The important word is can. An empty place in the schedule does not guarantee there is enough memory for the next prompt, and a real scheduler may limit work for reasons other than the number of requests.

Three Requests, Two Places in the Batch

Here’s a simplified decode-only example. Requests A and B are ready at the start. A needs two more output tokens; B needs five. Request C, which needs two, becomes eligible just after step 1. There is room for at most two sequences per step, each scheduled sequence produces one token per step, and we omit prompt processing and other costs.

In the table, A1 means A’s first token in this example. The fixed-membership server can stream A’s output as it appears, but it cannot admit C until the current batch ends.

StepFixed batch membershipContinuous batching
1A1, B1A1, B1
2A2, B2A2, B2
3B3B3, C1
4B4B4, C2
5B5B5
6C1All done
7C2All done

C starts at step 3 instead of step 6. B still finishes at step 5. Both schedules produce nine tokens, but the continuous one finishes the workload in five iterations rather than seven.

That’s a scheduling result, not a GPU benchmark. A step processing two sequences can take a different amount of time from one processing a single sequence. The example shows where the server finds useful work, not a guaranteed percentage improvement or a promise that B’s wall-clock latency stays unchanged.

The Request That Joins Is Not Always Ready to Decode

The table skipped a crucial part of the job: reading the prompt. Real LLM requests generally involve two kinds of work: [3]

  • Prefill: process prompt tokens, build the model’s reusable attention state, and obtain the prediction for the first output token when the prompt has been processed.
  • Decode: extend the response using the existing context. In ordinary decoding, a model step advances a scheduled sequence by one token. Returning those tokens to the client is a separate streaming decision.

A short question entering the batch is one thing. A request arriving with a long document is another. If the engine processes that document in one large prefill, users already receiving an answer may notice a pause between tokens.

Chunked prefill gives the scheduler smaller pieces of prompt work. In engines that support mixing the two phases, it can schedule a chunk of the new prompt alongside decode work from existing requests, then continue the prompt in later iterations. This can balance the wait for the new request’s first token against interruptions to responses already underway. It can also stretch the prompt’s processing over more steps. Sarathi-Serve studied this trade-off; vLLM documents how a per-batch token budget affects it. [3, 4]

The GPU Has a Token Budget and a Memory Budget

The number of requests alone is a poor description of a batch. Two users asking a one-line question and two users submitting books both count as “two requests.” Their prompt work and memory footprint can be very different.

The KV cache holds the keys and values a model reuses for earlier positions in a sequence. For a conventional full-attention model, more active context generally means more cached state. The engine needs room for that state and for it to grow during generation. [2, 5]

A scheduler therefore has to consider at least two constraints: how many input positions to process in the next pass, including any prefill chunks, and how much cache memory is available. Some engines also cap the number of active requests. Systems such as vLLM use paged cache management to reduce wasted allocations, but better allocation does not make memory unlimited. When a request completes, its live state no longer needs to occupy an active sequence’s allocation; some cached blocks may be retained for supported prefix reuse. [2, 4, 5]

The requests still have separate logical contexts. Putting them in the same batch does not concatenate their conversations. And “swap” in the title means changing membership in a batch; it does not imply that the server copies a request’s cache to CPU memory every time.

Bigger Batches Can Help the Server and Annoy the User

A busier GPU is not the same thing as a better chat experience. The service can produce more tokens each second across everyone while the person watching one response experiences longer gaps.

MeasureQuestion it answers
Time to first token (TTFT)How long after sending the request does the user see the first output token?
Inter-token latency (ITL)Once output starts, how long are the gaps between successive tokens?
Output throughputHow many output tokens per second does the server deliver across requests?
Queue time and cache pressureAre people waiting because admission is constrained, and is the live cache approaching its limit?

Measure first-token and inter-token delays at the client if you want to understand the experience users see. Check the measurement boundary: an engine’s metric will not necessarily include the network, and streaming chunks can contain more than one token. Track tail latency, such as the 95th percentile, as well as throughput. Benchmark with representative prompt lengths, output lengths, and traffic bursts; a test with one user does not reveal how the scheduler behaves when requests compete. [6]

If arrivals keep exceeding what the server can process, filling holes in the batch will not stop the queue from growing. That is where backpressure and admission limits become useful. The scheduler needs a policy for waiting, prioritizing, or rejecting work as well as a way to pack it into an iteration.

Three Optimizations, Three Different Questions

Continuous batching asks which requests should advance together. KV caching asks which attention state can be reused. Speculative decoding asks whether a target model can validate proposed tokens together and sometimes advance one sequence by more than one token in a round.

An engine can combine them. Their benefits should be measured separately, because they change different parts of the serving workload.

The detective’s story still has to unfold in order. Continuous batching just lets the server start helping somebody else before the detective catches the train.

References and Further Reading

  1. Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022 — iteration-level scheduling and selective batching.
  2. Hugging Face Transformers: Continuous Batching Architecture — request lifecycle, scheduling, and token and cache budgets.
  3. Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, OSDI 2024 — prefill, decode, and chunked prefill.
  4. vLLM: Optimization and Tuning — chunked prefill, token budget, and memory pressure.
  5. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023 — paged KV cache management and its impact on batching.
  6. vLLM: Production Metrics — definitions and examples of metrics for monitoring serving workloads.

Network Encyclopedia Editorial Team

Every entry is written, reviewed and kept up to date by our editors. Spotted a mistake? We fix it in the open.

Editorial Mission · Correction Policy · Suggest a correction