Posts tagged "inference"
3 posts found
Prefill-as-a-Service: How KVCache Goes Cross-Datacenter
Deriving PrfaaS-PD from the ground up — why the KVCache transfer bandwidth wall confines PD disaggregation to a single RDMA island, how hybrid attention lowers the KV throughput enough to cross that wall, the selective offloading principle, the three-stage producer-consumer throughput model, the optimality conditions for routing threshold and prefill/decode ratio, and the dual-timescale scheduler — all derived step by step with one tiny two-cluster deployment.
Grouped-Query Attention: Fewer KV Heads, Same Quality
Building GQA from the ground up — from multi-head attention to multi-query attention to grouped-query attention — showing exactly how sharing KV heads across query groups reduces the KV cache by a factor of h/g while preserving nearly all of MHA's quality. Every formula derived, every number verified.
The KV Bottleneck Explained: Why Inference Is Memory-Bound
Building from exact byte counts to the fundamental insight: autoregressive inference is bottlenecked not by arithmetic but by memory bandwidth from loading keys and values. Every KV cache optimization in the literature is a response to this single bottleneck — derived step by step with concrete numbers.