Sliding Window Attention: From Local Windows to Global Context
Borrowing from convolutions — how a sliding local window keeps attention linear in sequence length, recovers a global receptive field by stacking layers, and reaches across the whole document through a small set of task-driven global tokens.