Posts tagged "deepseek"
4 posts found
DeepSeek-V4 Hybrid Attention: CSA and HCA from Scratch
Building the long-context attention of DeepSeek-V4 from the ground up — dual KV streams, overlapping softmax compression, the lightning indexer, top-$k$ sparse selection, and heavier compression — all derived step by step on a six-token example.
DeepSeek Sparse Attention: Learned Token Selection from Scratch
Letting the model choose its own sparse pattern — a lightweight indexer scores every past token, top-k selects the keys that matter for this query, and two-stage training with KL alignment keeps the sparse model faithful to its dense teacher at $O(nk)$ cost.
DeepSeek-V2 from Scratch: Multi-head Latent Attention and DeepSeekMoE
Building DeepSeek-V2's two core innovations from the ground up — low-rank KV joint compression, decoupled RoPE, matrix absorption, fine-grained expert segmentation, shared experts, and three-level load balancing — all derived step by step with concrete numbers.
Mathematical Prerequisites for DeepSeek-V2
Building the linear algebra foundations for Multi-head Latent Attention — matrix-vector multiplication, transpose, associativity, non-commutativity, the bilinear identity, linearity over sums, low-rank factorization, and the dot product of concatenated vectors — all derived step by step with one consistent 3-dimensional example.