Prefill-as-a-Service: How KVCache Goes Cross-Datacenter

Large-scale LLM serving has converged on prefill-decode (PD) disaggregation: prefill (compute-bound, consumes the prompt and emits KVCache) and decode (memory-bandwidth-bound, emits tokens one by one) run on different machines. The catch is that once they are separated, the KVCache has to move between them, and in conventional Transformers that movement is enormous — large enough to pin prefill and decode inside the same RDMA fabric, which in practice means the same datacenter.

The paper Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter (Qin et al., 2026) argues that this is no longer a hard constraint. Modern hybrid-attention architectures — models that interleave a small number of full-attention layers with a larger number of linear-complexity or bounded-state layers — produce roughly an order of magnitude less KVCache per unit of prefill compute than dense Transformers. That shift moves KV transfer from “impossible on commodity Ethernet” to “plausible on commodity Ethernet.” But plausible is not practical: naive designs that ship every prefill cross-datacenter still thrash under bursty traffic, skewed length distributions, and fluctuating inter-cluster bandwidth. The paper’s PrfaaS-PD architecture is the scheduling and routing discipline that turns plausibility into practicality.

This post is a systems paper about inference deployment. We will derive the entire throughput model of PrfaaS-PD from scratch on one tiny two-cluster deployment. Every equation — KV throughput, cluster egress bandwidth, the three producer-consumer stages, the balance conditions on threshold $t$ and prefill/decode ratio $N_p/N_d$ — will be instantiated in concrete numbers. At the end we will reproduce the paper’s headline result (selective offloading beats both homogeneous and naive heterogeneous baselines) on our toy setup. If you want a deeper architectural treatment of why KVCache exists and why it dominates long-context inference cost, The KV Bottleneck Explained is the right companion read; it is not a prerequisite, and the primer immediately below covers enough to follow this post cold.

---

A Primer on Prefill, Decode, and the KVCache

Every result in this post is about moving bytes between two phases of transformer inference. Before we can argue about where those bytes should go, we need a concrete picture of what they are.

A transformer LLM processes text in two phases. Suppose a user sends the prompt “The capital of France is” and expects a continuation. The model goes through these two phases in order:

Phase 1 — Prefill. The model ingests all five prompt tokens in parallel. For every token, at every layer, self-attention computes three projections: a query $\mathbf{q}$, a key $\mathbf{k}$, and a value $\mathbf{v}$. Attention is then computed across all five positions at once, a single large matrix multiplication. Because every token is processed simultaneously and the compute scales with prompt length squared (for full attention) or length (for linear attention), prefill is compute-bound — a GPU’s arithmetic throughput is the limit. The output of prefill is twofold: the logits for the first generated token (“Paris”), and the entire set of $\mathbf{k}$ and $\mathbf{v}$ tensors for every prompt token at every layer. That stored set of keys and values is the KVCache.

Phase 2 — Decode. Now the model generates tokens one at a time, auto-regressively. To produce token $n+1$, the model only needs to compute $\mathbf{q}_{n+1}$ for the new token — but it must attend to the keys and values of all previous tokens. Those are already in the KVCache from prefill (and from any decode steps that happened before). So each decode step does a tiny amount of fresh compute (one query) and a large amount of memory I/O (loading the full KVCache of length up to $n$). Because the compute per step is tiny but the bytes loaded are proportional to context length, decode is memory-bandwidth-bound — the GPU’s HBM bandwidth is the limit, not its arithmetic throughput.

Why disaggregate them? A single GPU type does not serve both phases well. Hardware that maximizes arithmetic throughput (compute-dense accelerators, e.g. H200) is overkill for decode, which cannot saturate that arithmetic. Hardware that maximizes memory bandwidth (bandwidth-optimized accelerators, e.g. H20) is underpowered for prefill, which is compute-starved on it. PD disaggregation places prefill on compute-dense GPUs and decode on bandwidth-optimized GPUs, then ships the KVCache from the prefill machine to the decode machine once per request, right after prefill completes. Each GPU runs the phase it is good at, end to end throughput goes up, and cost goes down.

The catch. The KVCache is not small. For a dense Transformer at 32K tokens it can reach gigabytes per request, and a cluster of prefill instances can emit tens of gigabits per second of aggregate KV state. That state has to cross the network to reach the decode side. If the network cannot sustain it, prefill stalls waiting for the link, and the disaggregation benefit evaporates. This is exactly why PD disaggregation has historically been confined to a single RDMA fabric — typically a single datacenter — and why the paper’s argument begins by asking when that confinement can be relaxed. The KVCache and its transfer rate are the central objects of this entire post.

---

The Running Example

We deploy two clusters connected by a commodity Ethernet link.

PrfaaS cluster (compute-dense, like H200):

$$N_{\mathrm{prfaas}} = 2 \text{ prefill instances}, \qquad B_{\mathrm{out}} = 1000 \text{ MB/s} = 8 \text{ Gbps}$$

This cluster only does prefill. Its job is to take a long-context request, compute the KVCache, and ship that KVCache across the Ethernet link to the decode side. The egress bandwidth $B_{\mathrm{out}}$ is the sustained rate at which this cluster can push bytes onto the cross-datacenter link.

Local PD cluster (bandwidth-optimized, like H20):

$$N_p + N_d = 3 \text{ total instances (split between prefill and decode)}$$

Unlike the PrfaaS cluster, instances in this cluster can serve as either prefill nodes (call this count $N_p$) or decode nodes (count $N_d$). The ratio $N_p / N_d$ is a decision variable — part of what we will optimize.

Two kinds of requests (a discrete distribution over input length $L$):

• Short requests: $l_s = 1\mathrm{K}$ tokens
• Long requests: $l_l = 32\mathrm{K}$ tokens
• Fraction long: $P(L = 32\mathrm{K}) = 0.5$, fraction short: $P(L = 1\mathrm{K}) = 0.5$

Per-request prefill cost (Kimi Linear–style hybrid, scaled down from Table 5 of the paper):

Length $l$	KVCache size $S_{\mathrm{kv}}(l)$	Prefill time $T_{\mathrm{prefill}}(l)$
$1\mathrm{K}$	$100$ MB	$0.5$ s
$32\mathrm{K}$	$800$ MB	$2.0$ s

Decode side (governed by SLO constants):

$$T_{\mathrm{decode}} = 25 \text{ ms/token}, \qquad BS_{\mathrm{max}} = 10, \qquad L_{\mathrm{out}} = 100 \text{ tokens}$$

That is the entire setup. Every derivation, every numerical check, every comparison across the rest of this post will return to these numbers. If you can hold this table in your head, you can hold the whole paper.

---

Why PD Disaggregation Is Tied to a Single RDMA Island

We established in the primer that once prefill and decode live on different machines, the KVCache has to move. In a conventional Transformer, that movement is massive. Let us make it quantitative.

Consider one prefill instance in our deployment. If requests arrive back to back and each produces $S_{\mathrm{kv}}(l)$ megabytes of KV state in $T_{\mathrm{prefill}}(l)$ seconds, then the instance emits bytes at a sustained rate that we call its KV throughput.

$$\boxed{\quad \Phi_{\mathrm{kv}}(l) \;=\; \frac{S_{\mathrm{kv}}(l)}{T_{\mathrm{prefill}}(l)} \quad}$$

This is Equation (1) of the paper. It depends only on the model architecture and the request length, not on the system design: it measures how many bytes of KV state the model produces per second when processing requests of length $l$.

Numerical check

For our hybrid model at $l = 32\mathrm{K}$:

$$\Phi_{\mathrm{kv}}(32\mathrm{K}) \;=\; \frac{800 \text{ MB}}{2.0 \text{ s}} \;=\; 400 \text{ MB/s} \;=\; 3.2 \text{ Gbps}$$

At $l = 1\mathrm{K}$:

$$\Phi_{\mathrm{kv}}(1\mathrm{K}) \;=\; \frac{100 \text{ MB}}{0.5 \text{ s}} \;=\; 200 \text{ MB/s} \;=\; 1.6 \text{ Gbps}$$

For comparison, the paper’s Table 5 reports a 1T Kimi Linear–style model at 3.19 Gbps for 32K and 3.61 Gbps for 1K. Our numbers are in the same ballpark, chosen to keep the arithmetic clean.

Interpretation

What matters is not the absolute value of $\Phi_{\mathrm{kv}}$, but what it implies for the aggregate egress of an entire prefill cluster. An $N$-GPU prefill cluster, parallelism degree $P$ (GPUs per instance), serves $N/P$ independent instances in parallel. If each instance produces KV at rate $\Phi_{\mathrm{kv}}$, the cluster’s total egress requirement is

$$B_{\mathrm{out}} \;=\; \frac{N}{P} \cdot \frac{\mathbb{E}[S_{\mathrm{kv}}]}{\mathbb{E}[T_{\mathrm{prefill}}]} \;\approx\; \frac{N}{P} \cdot \Phi_{\mathrm{kv}}(L_{\mathrm{avg}})$$

This is Equation (2) of the paper. The approximation $\mathbb{E}[S_{\mathrm{kv}}]/\mathbb{E}[T_{\mathrm{prefill}}] \approx \Phi_{\mathrm{kv}}(L_{\mathrm{avg}})$ says that as long as the request distribution does not have pathological tails, you can substitute the average length $L_{\mathrm{avg}}$ into the per-request formula and get the aggregate rate. It is not an identity — it is a linearization, valid when $S_{\mathrm{kv}}$ and $T_{\mathrm{prefill}}$ are approximately linear in $l$ over the relevant range.

Why that number is the whole reason PD lives in one datacenter

Plug in a dense Transformer. For MiniMax-M2.5, the paper’s Figure 2 reports $\approx 60$ Gbps per instance at 32K. A modest prefill cluster of $N/P = 512$ instances would demand

$$512 \times 60 \text{ Gbps} = 30{,}720 \text{ Gbps} \approx 30 \text{ Tbps}$$

of aggregate egress. No commodity Ethernet fabric carries 30 Tbps between buildings. The only way to sustain this is RDMA inside a single high-radix fat-tree inside a single datacenter, which is exactly where PD disaggregation lives today. The KVCache transfer requirement is the invisible cable that ties prefill and decode to the same physical room.

This is the part that confuses almost everyone about “cross-datacenter serving.” The hardware boundary is not the issue. The issue is that KV throughput, a property of the model, sets an aggregate egress rate, and that egress rate has historically exceeded what cross-building links can sustain. Nothing you do in the system layer can fix that as long as the model keeps producing bytes at dense-Transformer rates.

---

How Hybrid Attention Changes the Boundary

Hybrid architectures interleave a small number of full-attention layers with a larger number of linear-complexity or bounded-state layers — Kimi Delta Attention (KDA), sliding window, linear attention. Only the full-attention layers produce KVCache that scales with sequence length; the bounded-state layers carry a fixed-size recurrent state per request. Aggregated across the model, this gives a far smaller KV footprint per token than a stack of full-attention layers. We take this as a given input to the systems argument; why those mechanisms produce smaller KV is a model-architecture question, not a serving question.

The paper’s Table 3 reports that at 32K tokens, Kimi Linear delivers $\Phi_{\mathrm{kv}} = 3.87$ Gbps versus MiniMax-M2.5’s $59.93$ Gbps — about a $13\times$ reduction in KV throughput. This is the one quantitative fact that enables everything that follows. Our toy numbers follow the same pattern: 3.2 Gbps for a hybrid model at 32K is an order of magnitude below what a dense Transformer would produce on the same request.

Interpretation

Hybrid attention does not make cross-datacenter KVCache free. It makes it possible. A 10 Gbps commodity Ethernet link now has enough headroom to sustain the KV egress of a modest hybrid prefill cluster. For our example:

$$B_{\mathrm{out}} = 1000 \text{ MB/s} = 8 \text{ Gbps}$$

and a per-instance requirement of $\Phi_{\mathrm{kv}}(32\mathrm{K}) = 3.2$ Gbps. A cluster of $N_{\mathrm{prfaas}} = 2$ instances would demand $2 \times 3.2 = 6.4$ Gbps, which fits inside $B_{\mathrm{out}}$ with room to spare. A dense Transformer with $\Phi_{\mathrm{kv}} = 30$ Gbps per instance would demand $60$ Gbps just from 2 instances, breaking the link immediately.

That “fits with room to spare” is misleading if you stop there. Real traffic is bursty, lengths are skewed, prefix-cache hit rates fluctuate. The aggregate rate formula assumes steady state. The paper’s insight, which we will derive next, is that plausibility is not practicality: even when the math says the link can sustain the load on average, a naive design that externalizes every prefill will thrash.

---

The Core Design Principle: Selective Offloading

The paper’s key architectural move is a deceptively simple routing rule. Let $l$ be the incremental prefill length of an incoming request (total input length minus anything already prefix-cached). Let $t$ be a routing threshold — a single scalar decision variable that partitions requests into two classes.

$$\text{If } l > t, \text{ route to PrfaaS (the compute-dense remote cluster)}$$ $$\text{If } l \le t, \text{ route to PD-P (the local prefill path in the PD cluster)}$$

Short requests stay local. Long requests go to the compute-dense remote cluster. The KVCache produced by long requests then streams back over Ethernet to the local decode cluster.

Why selective, not universal

To see why this matters, consider three ways to deploy:

• Homogeneous PD: all prefill and all decode on the same hardware, inside one cluster. No cross-datacenter anything.
• Naive heterogeneous PD: all prefill goes to a remote compute-dense cluster, all decode stays local. No threshold.
• PrfaaS-PD (selective): only requests with $l > t$ go remote; the rest stay local.

We will show that the three give very different throughput numbers, and that the ordering

$$\Lambda_{\mathrm{prfaas-pd}} > \Lambda_{\mathrm{naive-hetero}} > \Lambda_{\mathrm{homogeneous}}$$

holds on our toy example, with the PrfaaS-PD gain coming from exactly the subset of requests for which the compute speedup outweighs the transfer cost.

But to show it, we need the throughput model.

---

The Throughput Model: Three Producers, One Consumer

A PrfaaS-PD system has three distinct roles, not two:

• PrfaaS (remote prefill): handles long requests with $l > t$, produces KVCache, ships it over Ethernet.
• PD-P (local prefill within the PD cluster): handles short requests with $l \le t$, produces KVCache, ships it over intra-cluster RDMA.
• PD-D (decode): the sole consumer. Reads KVCache from both upstream producers and emits tokens.

This is a converging producer-consumer pipeline. Two producers (PrfaaS, PD-P) feed one consumer (PD-D). The end-to-end system throughput is limited by whichever stage saturates first — this is the pipeline-bottleneck principle (or equivalently, Little’s Law applied at steady state across producer-consumer pairs).

We will derive the throughput of each stage separately, then combine them.

Stage 1: PrfaaS cluster throughput

Each PrfaaS request undergoes two overlapped phases: prefill computation on local hardware, and KVCache transfer across the Ethernet link to decode. Through layer-wise prefill pipelining (computing one layer’s KV while transferring the previous layer’s), these two phases run concurrently, so the overall cluster throughput is limited by whichever phase is slower.

Compute-bound throughput. If every PrfaaS instance needs $T_{\mathrm{prefill}}(l_{\mathrm{long}})$ seconds per request and we have $N_{\mathrm{prfaas}}$ instances in parallel, the cluster completes

$$\frac{N_{\mathrm{prfaas}}}{T_{\mathrm{prefill}}(l_{\mathrm{long}})} \text{ requests per second (compute-bound)}$$

where $l_{\mathrm{long}} = \mathbb{E}[L \mid L > t]$ is the mean length conditional on being routed to PrfaaS.

Bandwidth-bound throughput. If every completed request must ship $S_{\mathrm{kv}}(l_{\mathrm{long}})$ bytes across the egress link, and the link sustains $B_{\mathrm{out}}$ bytes per second, the link can handle

$$\frac{B_{\mathrm{out}}}{S_{\mathrm{kv}}(l_{\mathrm{long}})} \text{ requests per second (bandwidth-bound)}$$

The cluster operates at the slower of the two:

$$\boxed{\quad \Theta_{\mathrm{prfaas}} \;=\; \min\!\left( \frac{N_{\mathrm{prfaas}}}{T_{\mathrm{prefill}}(l_{\mathrm{long}})}, \; \frac{B_{\mathrm{out}}}{S_{\mathrm{kv}}(l_{\mathrm{long}})} \right) \quad}$$

This is Equation (3) of the paper. The $\min$ captures the essence: if compute is the bottleneck, adding more bandwidth does nothing. If bandwidth is the bottleneck, adding more GPUs does nothing. Selective offloading works by pushing the operating point into the compute-bound regime, where the expensive PrfaaS hardware actually earns its keep.

Numerical check

We set threshold $t = 8\mathrm{K}$, which puts all 32K requests on PrfaaS (so $l_{\mathrm{long}} = 32\mathrm{K}$). Plug in:

• Compute term: $N_{\mathrm{prfaas}} / T_{\mathrm{prefill}}(32\mathrm{K}) = 2 / 2.0 = 1.0$ req/s
• Bandwidth term: $B_{\mathrm{out}} / S_{\mathrm{kv}}(32\mathrm{K}) = 1000 / 800 = 1.25$ req/s
• Minimum: $1.0$ req/s

So $\Theta_{\mathrm{prfaas}} = 1.0$ req/s. The cluster is compute-bound at this operating point. That is exactly what we want: the expensive compute-dense accelerators are running at saturation, and the link has 20% headroom to absorb bursts.

Stage 2: PD-P local prefill throughput

The PD-P stage is simpler because it has no cross-cluster link — intra-cluster RDMA is not the bottleneck. Only compute matters:

$$\boxed{\quad \Theta_{\mathrm{pd\text{-}p}} \;=\; \frac{N_p}{T_{\mathrm{prefill}}(l_{\mathrm{short}})} \quad}$$

This is Equation (4). Here $l_{\mathrm{short}} = \mathbb{E}[L \mid L \le t]$ is the mean length of requests that stay local.

Numerical check

With $t = 8\mathrm{K}$, only 1K requests stay local (so $l_{\mathrm{short}} = 1\mathrm{K}$). We do not yet know $N_p$; treat it as a variable to be optimized. For any allocation $N_p$:

$$\Theta_{\mathrm{pd\text{-}p}}(N_p) \;=\; \frac{N_p}{0.5} \;=\; 2 N_p \text{ req/s}$$

If $N_p = 1$, PD-P processes $2$ req/s; if $N_p = 2$, $4$ req/s; if $N_p = 0$, the local prefill path is disabled.

Stage 3: PD-D decode throughput

Decode is the consumer. Each decode instance runs batched token generation: at each step, it loads the KVCache for a batch of up to $BS_{\mathrm{max}}$ concurrent requests and emits one new token per request. Each step takes $T_{\mathrm{decode}}$ seconds.

In one second, a decode instance completes $1/T_{\mathrm{decode}}$ steps, each emitting up to $BS_{\mathrm{max}}$ tokens, giving $BS_{\mathrm{max}}/T_{\mathrm{decode}}$ tokens per second per instance. A request has $L_{\mathrm{out}}$ tokens, so each instance finishes $BS_{\mathrm{max}}/(T_{\mathrm{decode}} \cdot L_{\mathrm{out}})$ requests per second. Multiplying by $N_d$ instances:

$$\boxed{\quad \Theta_{\mathrm{pd\text{-}d}} \;=\; \frac{N_d \cdot BS_{\mathrm{max}}}{T_{\mathrm{decode}} \cdot L_{\mathrm{out}}} \quad}$$

This is Equation (5). The paper treats $BS_{\mathrm{max}}$ and $T_{\mathrm{decode}}$ as SLO-governed constants — they are set by latency targets and the speculative-decoding regime in production, not by us.

Numerical check

$$\Theta_{\mathrm{pd\text{-}d}}(N_d) \;=\; \frac{N_d \cdot 10}{0.025 \cdot 100} \;=\; \frac{10 N_d}{2.5} \;=\; 4 N_d \text{ req/s}$$

If $N_d = 2$, decode delivers $8$ req/s; if $N_d = 3$, $12$ req/s.

Combining the three stages

Here is where the converging pipeline becomes important. PrfaaS handles fraction $p = P(L > t)$ of all requests. PD-P handles the remaining fraction $1 - p$. PD-D sees everything. If $\Lambda$ is the total request arrival rate, then:

• PrfaaS sees $p \Lambda$ req/s; its capacity is $\Theta_{\mathrm{prfaas}}$. So $p \Lambda \le \Theta_{\mathrm{prfaas}}$, i.e. $\Lambda \le \Theta_{\mathrm{prfaas}}/p$.
• PD-P sees $(1-p)\Lambda$; its capacity is $\Theta_{\mathrm{pd\text{-}p}}$. So $\Lambda \le \Theta_{\mathrm{pd\text{-}p}}/(1-p)$.
• PD-D sees all of $\Lambda$; its capacity is $\Theta_{\mathrm{pd\text{-}d}}$. So $\Lambda \le \Theta_{\mathrm{pd\text{-}d}}$.

The system sustains whatever rate satisfies all three constraints — the minimum of the three ceilings:

$$\boxed{\quad \Lambda_{\max} \;=\; \min\!\left( \frac{\Theta_{\mathrm{prfaas}}}{p}, \; \frac{\Theta_{\mathrm{pd\text{-}p}}}{1-p}, \; \Theta_{\mathrm{pd\text{-}d}} \right) \quad}$$

This is Equation (6). It is the central formula of the paper: once you know $\Theta_{\mathrm{prfaas}}, \Theta_{\mathrm{pd\text{-}p}}, \Theta_{\mathrm{pd\text{-}d}}$ and the routing split $p$, the end-to-end throughput is fully determined.

Pipeline diagram

p 1−p KV via Ethernet KV via RDMA

Two producers, one consumer, a fraction $p$ routed upstream. The end-to-end rate is limited by whichever of the three capacity ceilings is closest to breaking.

Numerical check: what is $\Lambda_{\max}$ for our example?

With $t = 8\mathrm{K}$, $p = 0.5$, and allocation $N_p = 1, N_d = 2$:

• PrfaaS ceiling: $\Theta_{\mathrm{prfaas}}/p = 1.0 / 0.5 = 2.0$ req/s
• PD-P ceiling: $\Theta_{\mathrm{pd\text{-}p}}/(1-p) = 2.0 / 0.5 = 4.0$ req/s
• PD-D ceiling: $\Theta_{\mathrm{pd\text{-}d}} = 4 \cdot 2 = 8.0$ req/s
• Minimum: $2.0$ req/s

$$\boxed{\; \Lambda_{\max} = 2.0 \text{ req/s} \;}$$

The system is bottlenecked by PrfaaS. If we want to push $\Lambda_{\max}$ higher, we need to relax that bottleneck — either by adjusting $t$ (to change which requests go to PrfaaS and how big $p$ is) or by reallocating $N_p, N_d$ (to shift local prefill/decode capacity). This is exactly the joint optimization the paper formalizes next.

---

Finding the Optimal Operating Point

The threshold $t$ balances PrfaaS and PD-P

We have two decision variables: the routing threshold $t$ and the local prefill/decode ratio $N_p/N_d$. Fix $N_p, N_d$ for a moment and vary $t$.

As $t$ increases, fewer requests qualify as “long” — $p$ decreases, $l_{\mathrm{long}}$ grows (since only the longest requests remain), $l_{\mathrm{short}}$ grows too (because medium-length requests now stay local). PrfaaS handles fewer but heavier requests; PD-P handles more and heavier short requests.

As $t$ decreases, more requests qualify as long — $p$ increases, both $l_{\mathrm{long}}$ and $l_{\mathrm{short}}$ shrink. PrfaaS is flooded with shorter requests whose high KV throughput (per unit compute) is more likely to saturate the egress link.

There is an interior optimum. The paper’s insight is that at the optimum, both upstream producers saturate together:

$$\boxed{\quad \frac{\Theta_{\mathrm{prfaas}}}{p} \;=\; \frac{\Theta_{\mathrm{pd\text{-}p}}}{1-p} \quad}$$

This is Equation (7). The argument: if the two ceilings were unequal, you would be wasting capacity on whichever side was higher — you could push traffic the other way and lift the binding minimum. Balancing the ceilings is the one-dimensional Pareto optimum for $t$.

Numerical check

Our numbers. At $t = 8\mathrm{K}$: $\Theta_{\mathrm{prfaas}}/p = 2.0$, $\Theta_{\mathrm{pd\text{-}p}}/(1-p) = 4.0$. Unequal. PD-P has idle capacity; PrfaaS is the bottleneck. Equation (7) says we should push traffic onto PrfaaS — but moving traffic by lowering $t$ is exactly wrong here, because lowering $t$ makes PrfaaS handle more requests while simultaneously shortening $l_{\mathrm{long}}$ in ways that may saturate the bandwidth term in $\Theta_{\mathrm{prfaas}}$. The relationship is not monotonic in general; in practice the paper resolves this with a grid search.

With only two request sizes in our toy setup, the only choices for $t$ are effectively ”$t$ sends all long requests to PrfaaS” or ”$t$ also sends short requests to PrfaaS.” A continuous sweep requires a continuous length distribution, which is exactly why the paper uses a truncated log-normal distribution in its case study. For our purposes we proceed with $t = 8\mathrm{K}$ (only 32K requests go remote) and accept that PrfaaS is the binding constraint.

The allocation $N_p / N_d$ balances producers and consumer

With $t$ fixed, we still have to split $N_p + N_d$ between local prefill and local decode. The second optimality condition balances the aggregate producer throughput against the consumer:

$$\boxed{\quad \Theta_{\mathrm{prfaas}} + \Theta_{\mathrm{pd\text{-}p}} \;=\; \Theta_{\mathrm{pd\text{-}d}} \quad}$$

This is Equation (8), the producer-consumer balance. If producers exceed decode, KVCache piles up waiting for decode slots (wasted prefill capacity). If decode exceeds producers, decode instances starve (wasted decode capacity).

Numerical check: grid search over $N_p, N_d$ with $N_p + N_d = 3$

We enumerate the four feasible allocations and compute $\Lambda_{\max}$ for each, with $t = 8\mathrm{K}$, $p = 0.5$.

Allocation A: $N_p = 0, N_d = 3$. No local prefill.

• $\Theta_{\mathrm{prfaas}} = 1.0$, $\Theta_{\mathrm{pd\text{-}p}} = 0$, $\Theta_{\mathrm{pd\text{-}d}} = 12$.
• Ceilings: $1.0/0.5 = 2.0$, $0/0.5 = 0$, $12$. Min = $0$.
• Useless: short requests have nowhere to go.

Allocation B: $N_p = 1, N_d = 2$.

• $\Theta_{\mathrm{prfaas}} = 1.0$, $\Theta_{\mathrm{pd\text{-}p}} = 2.0$, $\Theta_{\mathrm{pd\text{-}d}} = 8$.
• Ceilings: $2.0, 4.0, 8.0$. Min = $2.0$.

Allocation C: $N_p = 2, N_d = 1$.

• $\Theta_{\mathrm{prfaas}} = 1.0$, $\Theta_{\mathrm{pd\text{-}p}} = 4.0$, $\Theta_{\mathrm{pd\text{-}d}} = 4$.
• Ceilings: $2.0, 8.0, 4.0$. Min = $2.0$.

Allocation D: $N_p = 3, N_d = 0$.

• No decode: $\Lambda_{\max} = 0$.

Both B and C tie at $\Lambda_{\max} = 2.0$ req/s. The bottleneck in both is the PrfaaS ceiling. That is the telltale sign that the system is under-provisioned on the PrfaaS side — no local reallocation can lift it. The only fix is to add more PrfaaS instances or widen the Ethernet link.

For the rest of this post we take allocation B ($N_p = 1, N_d = 2$) as our canonical operating point.

The paper’s grid search, visualized

The paper runs the same kind of enumeration at realistic scale (Figure 5). The left panel fixes $t$ at the optimum and sweeps $N_p / N_d$, showing the characteristic “V” shape: throughput grows as $N_p$ increases (prefill-bound side), peaks where Equation (8) holds, then drops as $N_d$ becomes too small (decode-bound side). The right panel fixes the best $(N_p, N_d)$ and sweeps $t$, showing a single sharp peak where Equation (7) holds. The peak of the composed two-dimensional search is the throughput-optimal configuration.

---

Dual-Timescale Scheduling: Why the Steady-State Model Is Not Enough

Equations (3)–(8) describe the steady state. In production, nothing is steady. Request arrival rates fluctuate, cross-datacenter bandwidth varies by time of day (especially if the link is shared with other tenants via VPC peering), prefix-cache hit rates depend on which users are active. The paper’s scheduler is explicitly a dual-timescale algorithm: fast reactive routing on the short timescale, slow resource reallocation on the long timescale.

Short-term: bandwidth- and cache-aware routing

The PrfaaS cluster has a bandwidth-imposed throughput ceiling of

$$\frac{B_{\mathrm{out}}}{S_{\mathrm{kv}}(l_{\mathrm{long}})} \text{ req/s}$$

As the system approaches this ceiling, the egress link fills up and queuing delay explodes. The scheduler continuously monitors egress utilization and request queue depth. When either crosses a threshold, it re-profiles the current operating conditions and re-solves for the threshold $t$ using Equation (7).

For requests with prefix-cache hits, the decision involves two cached lengths: $l_{\mathrm{prfaas}}$, the cached prefix length in the PrfaaS cluster, and $l_{\mathrm{pd}}$, the cached prefix length in the local PD cluster. The incremental length at the PD cluster is $l_{\mathrm{total}} - l_{\mathrm{pd}}$, at the PrfaaS cluster $l_{\mathrm{total}} - l_{\mathrm{prfaas}}$.

• When bandwidth is scarce (egress near capacity), the two clusters are evaluated independently. A request is prefilled at whichever cluster has the most cache — if $l_{\mathrm{total}} - l_{\mathrm{pd}} \le t$, stay local; otherwise go remote. No cross-cluster cache transfer is triggered.
• When bandwidth is abundant (egress has headroom), the scheduler considers $l_{\mathrm{prefix}} = \max(l_{\mathrm{prfaas}}, l_{\mathrm{pd}})$, the best cache across clusters. If the resulting incremental length $l_{\mathrm{total}} - l_{\mathrm{prefix}} \le t$, the request goes to PD-P; otherwise to PrfaaS. If the cache cluster differs from the chosen compute cluster, a cross-cluster cache transfer is performed opportunistically.

This is the rule that operationalizes “selective offloading” dynamically. When the network is tight, the scheduler protects it. When the network has room, the scheduler uses that room to reduce redundant computation by reusing a better cache.

Long-term: traffic-driven allocation re-optimization

On timescales where traffic patterns persistently shift (hour, day), the scheduler re-runs the grid search of Equations (7) and (8) over new profiled data, and converts nodes between prefill and decode roles within the local PD cluster. $N_p$ and $N_d$ are not fixed at deployment — they can drift as the workload drifts. The routing threshold $t$ is re-optimized to match.

The separation of timescales is deliberate. Routing is cheap and reactive. Role conversion is expensive (it involves draining pending KVCache and reloading model weights, which the paper treats as an infrequent operation). Mixing the two would either flap the cluster or miss short-term congestion.

---

A Brief Note on the Hybrid Prefix Cache Pool

One design detail deserves mention because it is the storage-layer analog of the model-layer asymmetry. Linear-attention layers keep a recurrent state whose size is independent of input length. Full-attention layers keep a block-level KVCache that grows linearly with input length and supports partial prefix matching. These two kinds of cache are fundamentally different: the recurrent state is per-request and only reusable on exact length matches, while the KVCache is per-block and supports prefix matching at the block level.

The paper’s solution is to manage them as two groups sharing a unified block pool, with two categories of blocks: prefix-cache blocks (reusable, block-aligned, intra-cluster only) and transfer-cache blocks (produced at the tail of a prefill request, sent cross-cluster, then discarded). A global KVCache manager maintains metadata across clusters so the router can pick both the best-cache cluster and the best-bandwidth cluster.

We do not re-derive this piece because it is a storage-layer implementation detail, not a throughput argument. But it is worth noting that the same hybrid structure that makes the model’s KV tractable also shapes the prefix cache pool — the architectural asymmetry propagates all the way down.

---

Putting It Together: PrfaaS-PD vs Homogeneous vs Naive Heterogeneous

We now compute $\Lambda_{\max}$ under each of the three deployment paradigms using the exact same hardware budget. The comparison highlights what selective offloading contributes on top of what hybrid attention already enables.

Setup for the comparison

All three deployments share the same total hardware (measured by instance-equivalents that have compute and bandwidth comparable to our original split). The PrfaaS cluster has 2 compute-dense instances; the local PD cluster has 3 bandwidth-optimized instances. For the non-heterogeneous baselines we consolidate these into one pool.

PrfaaS-PD (our setup): $N_{\mathrm{prfaas}} = 2$, $N_p = 1$, $N_d = 2$, $t = 8\mathrm{K}$, $p = 0.5$. Already computed: $\Lambda_{\max} = 2.0$ req/s.

Naive heterogeneous PD: all prefill goes to the remote cluster, all decode stays local. In our toy setting, that means we set $t = 0$ (every request qualifies as “long”), so $p = 1$, $N_p = 0$, $N_d = 3$.

• PrfaaS must now process requests of both lengths. Mean length $l_{\mathrm{long}} = 0.5 \cdot 1\mathrm{K} + 0.5 \cdot 32\mathrm{K} = 16.5\mathrm{K}$. Linearly interpolating our table:
  • $T_{\mathrm{prefill}}(16.5\mathrm{K}) \approx 0.5 + \frac{16.5 - 1}{32 - 1}(2.0 - 0.5) = 0.5 + 0.5 \cdot 1.5 = 1.25$ s
  • $S_{\mathrm{kv}}(16.5\mathrm{K}) \approx 100 + \frac{15.5}{31}(800 - 100) = 100 + 350 = 450$ MB
• Compute ceiling: $2/1.25 = 1.6$
• Bandwidth ceiling: $1000/450 \approx 2.22$
• $\Theta_{\mathrm{prfaas}} = \min(1.6, 2.22) = 1.6$ req/s
• $\Theta_{\mathrm{pd\text{-}p}} = 0$ (no local prefill)
• $\Theta_{\mathrm{pd\text{-}d}} = 4 \cdot 3 = 12$
• $\Lambda_{\max} = \min(1.6/1, 0/0 \text{ (vacuous)}, 12) = 1.6$ req/s

Homogeneous PD (no PrfaaS): consolidate everything into one cluster. The hardware budget there is different from PrfaaS-PD — no compute-dense accelerators, just the local bandwidth-optimized hardware. For this comparison we give the homogeneous cluster $N_p + N_d = 5$ bandwidth-optimized instances (approximately equal total FLOPs to the heterogeneous setup’s 2 PrfaaS + 3 local). The local instances are slower on prefill; assume prefill time scales by $1.5\times$ compared to the compute-dense PrfaaS hardware: $T_{\mathrm{prefill}}^{\mathrm{slow}}(1\mathrm{K}) = 0.75$ s, $T_{\mathrm{prefill}}^{\mathrm{slow}}(32\mathrm{K}) = 3.0$ s, mean $1.875$ s. All requests go through one prefill pool of $N_p^{\mathrm{homo}}$ instances.

• Best allocation balances Equation (8): $\Theta_{\mathrm{pd\text{-}p}} = \Theta_{\mathrm{pd\text{-}d}}$, i.e. $N_p / 1.875 = 4 N_d$, so $N_p = 7.5 N_d$. With $N_p + N_d = 5$, approximately $N_p = 4, N_d = 1$.
• $\Theta_{\mathrm{pd\text{-}p}} = 4 / 1.875 \approx 2.13$ req/s.
• $\Theta_{\mathrm{pd\text{-}d}} = 4 \cdot 1 = 4$ req/s.
• $\Lambda_{\max} = \min(2.13/1, 4) = 2.13$ req/s — oh, actually higher than PrfaaS-PD at 2.0. Let me re-examine.

The apparent tie/reversal happens because our toy numbers were not chosen with enough asymmetry between compute-dense and bandwidth-optimized hardware. In the paper’s case study, the real gap is larger: Kimi Linear on H200 vs H20 gives ~1.5× compute advantage plus a big cost advantage. To reproduce the paper’s 54% improvement we would need to load realistic hardware-specific prefill times.

Rather than rig the example, it is more honest to point to the exact result from the paper: on a 1T hybrid model with 32 H200 GPUs for PrfaaS + 64 H20 GPUs for local PD, against a homogeneous 96-H20 baseline, the paper reports

$$\Lambda_{\max}^{\mathrm{prfaas-pd}} = 3.24, \quad \Lambda_{\max}^{\mathrm{homo}} = 2.11, \quad \Lambda_{\max}^{\mathrm{naive}} = 2.45 \text{ req/s}$$

corresponding to a $1.54\times$ throughput improvement over homogeneous and a $1.32\times$ improvement over naive heterogeneous. The structure of the argument — three stages, minimum of three ceilings — is the same as our toy derivation; only the numbers shift when real hardware asymmetries enter.

Interpretation

What does naive heterogeneous lose? Two things.

First, it wastes compute-dense capacity on short requests. A 1K-token request takes $0.5$ s on PrfaaS and emits $100$ MB of KVCache. The per-request throughput is limited more by communication overhead and by instance cold-start effects than by raw compute; the expensive hardware is not its strongest on these requests.

Second, it locks the local cluster into decode-only operation. That sounds like a feature, but it means the system cannot absorb bursts of short requests using the local prefill path — every burst has to be shipped remotely, saturating the Ethernet link precisely when arrivals spike.

Selective offloading avoids both pathologies. Short requests stay local, use the path where they are cheapest, and never touch the Ethernet link. Long requests go remote, where the compute speedup pays for the transfer cost.

---

Why Hybrid Attention Is Necessary but Not Sufficient

A central claim of the paper worth restating in our derived form:

Reduced KV throughput is the precondition. Selective offloading and bandwidth-aware scheduling are what make it practical.

The argument goes through Equations (1)–(8):

• Equation (2) says aggregate cluster egress scales with $\Phi_{\mathrm{kv}}$. Dense Transformers make this unsustainable on commodity Ethernet.
• Equation (3) says PrfaaS throughput has a bandwidth term that depends on $S_{\mathrm{kv}}/B_{\mathrm{out}}$. A model with 10× smaller $S_{\mathrm{kv}}$ shifts this term into the non-binding regime.
• But a bandwidth-friendly $\Phi_{\mathrm{kv}}$ alone does not set $p, t, N_p/N_d$. Those remain free parameters. Without Equations (7)–(8), you pick them arbitrarily and leave throughput on the table — our naive-heterogeneous calculation is the worked example of leaving throughput on the table.

The paper’s contribution is not “KV throughput is smaller now.” That is a consequence of earlier architectural work. The contribution is the complete optimization problem that stitches model properties, system bandwidth, hardware heterogeneity, and request length distribution into a single tractable throughput model with closed-form optimality conditions.

---

Summary

We started with one observation: hybrid-attention models produce ~13× less KV per second than dense Transformers. We derived the bandwidth identity $\Phi_{\mathrm{kv}} = S_{\mathrm{kv}}/T_{\mathrm{prefill}}$ and showed that this shift moves the PD deployment boundary from RDMA-scale fabrics to commodity Ethernet — necessary but not sufficient. We then derived the PrfaaS-PD throughput model as a three-stage converging pipeline: $\Theta_{\mathrm{prfaas}}$ limited by $\min(\text{compute}, \text{bandwidth})$, $\Theta_{\mathrm{pd\text{-}p}}$ by local compute, $\Theta_{\mathrm{pd\text{-}d}}$ by the decode-SLO constants, with end-to-end $\Lambda_{\max}$ the minimum of three ceilings normalized by the routing split $p$. The two optimality conditions (balanced producers, producer-consumer equality) reduce deployment to a two-dimensional grid search over $(t, N_p/N_d)$. The dual-timescale scheduler keeps the steady-state optimum reachable under real bursty traffic. And the selective offloading rule — long requests remote, short requests local — is what turns cross-datacenter plausibility into cross-datacenter practicality.

The full arc, in a single sentence: hybrid attention shrinks the KV, $\Phi_{\mathrm{kv}}$ falls, commodity Ethernet becomes sufficient, selective offloading prevents naive designs from thrashing the link, and the joint optimization over $t$ and $N_p/N_d$ delivers the $1.54\times$ throughput improvement that makes cross-datacenter PD disaggregation the first heterogeneous deployment paradigm worth operating at scale.

---

Previous: The Efficient Transformer Design Space: Comparing All Variants and the Three Futures of Attention
Next: DeepSeek-V4 Hybrid Attention: CSA and HCA from Scratch