Story
arxiv_cs_cl · Nov 17, 2026 · paper
arxiv.orgNov 17, 2026
original source linked
In brief
In the global attention layers of Gemma 4 31B, the value is the normalized key projection, and the key is the value scaled by a fixed weight, with rotary position encoding on one quarter of its dimensions. The Gemma 4 report states that this design reduces the global KV cache by