Posts tagged "kv-cache"
5 posts found
DeepSeek-V4 Hybrid Attention: CSA and HCA from Scratch
Building the long-context attention of DeepSeek-V4 from the ground up — dual KV streams, overlapping softmax compression, the lightning indexer, top-$k$ sparse selection, and heavier compression — all derived step by step on a six-token example.
Prefill-as-a-Service: How KVCache Goes Cross-Datacenter
Deriving PrfaaS-PD from the ground up — why the KVCache transfer bandwidth wall confines PD disaggregation to a single RDMA island, how hybrid attention lowers the KV throughput enough to cross that wall, the selective offloading principle, the three-stage producer-consumer throughput model, the optimality conditions for routing threshold and prefill/decode ratio, and the dual-timescale scheduler — all derived step by step with one tiny two-cluster deployment.
DeepSeek-V2 from Scratch: Multi-head Latent Attention and DeepSeekMoE
Building DeepSeek-V2's two core innovations from the ground up — low-rank KV joint compression, decoupled RoPE, matrix absorption, fine-grained expert segmentation, shared experts, and three-level load balancing — all derived step by step with concrete numbers.
Grouped-Query Attention: Fewer KV Heads, Same Quality
Building GQA from the ground up — from multi-head attention to multi-query attention to grouped-query attention — showing exactly how sharing KV heads across query groups reduces the KV cache by a factor of h/g while preserving nearly all of MHA's quality. Every formula derived, every number verified.
The KV Bottleneck Explained: Why Inference Is Memory-Bound
Building from exact byte counts to the fundamental insight: autoregressive inference is bottlenecked not by arithmetic but by memory bandwidth from loading keys and values. Every KV cache optimization in the literature is a response to this single bottleneck — derived step by step with concrete numbers.