Sliding window with sinks

The efficiency lineage · 2026

The idea

People spent 40 million to 100 billion extra tokens converting a finished softmax model into linear attention. Did anyone try just changing the mask?

Linear attention replaces the growing cache with a fixed-size box. That conversion is the expensive door. Jolicoeur-Martineau, Sukthanker, Cameron, and Gervais (2026) asked a cheaper question: take the same pretrained model, attend only to a short rolling window, and keep the first four tokens visible so softmax does not collapse. No extra tokens. No stages. That is training-free sliding-window attention with attention sinks — SWA(w, 4).

It is tempting to think those four tokens are a memory of the important noun. They are not. Models dump leftover attention into the first few positions (Xiao et al., 2024). Drop them and the window-without-sinks already failed — the same miss sparse attention showed, where a nearby window never sees “animal”. Keep them and softmax stays stable. The referent can still sit outside the window. That leftover is honest, and it is the paper’s own limit.

Verdict

What it fixedA training-free mask — keep 4 sink tokens plus a rolling window — matches or beats linear retrofits on short benches (93.2% MMLU recovery, 0 extra tokens).

What it still costsA needle still has to land in the window. At 4K, SWA recovers a fraction of full attention; it beats the conversion, not the teacher.

The story, animated

The conversion, the broken window, the sinks that are a dump not a memory, then the paper’s own numbers as bar width.

Where it sits

This is a sibling of the linear branch, not a step toward the hybrid. Delta, Gated DeltaNet, and KDA still repair the box. SWA asks whether you needed the box.

Worth having first: Linear attention

Also useful: Sparse attention — the window that misses “animal” is still true here.

Paper: Jolicoeur-Martineau et al., 2026, arXiv:2608.28444. Sinks: Xiao et al., 2024 (StreamingLLM). The numbers on the page are that paper’s Table 1 (MMLU recovery) and Table 3 (S-NIAH-3 at 4K, window 512).