--- date: '2026-05-27' description: reduce KV heads to a fraction of query heads, share K,V across groups for cheaper decode-time cache reuse id: attention-gqa modified: 2026-06-06 01:37:07 GMT-04:00 seealso: - '[[thoughts/Attention|Attention]]' - '[[thoughts/MLA|MLA]]' - '[[thoughts/KV compression]]' tags: - ml - llm - technical title: Group-Query Attention created: '2026-05-27' published: '2026-05-27' pageLayout: default slug: thoughts/GQA permalink: https://aarnphm.xyz/thoughts/GQA.md generator: quartz: v4.6.0 hostedProvider: Cloudflare baseUrl: aarnphm.xyz full: https://aarnphm.xyz/llms-full.txt --- idea: reduce number of KV heads $n_k$ to a fraction $n_k^{'} = \frac{n_q}{r}$ of number of query heads $n_q$ (evenly dividing the query heads into $n_k$ groups with $r$ heads) Let $n_q$ be the number of query heads, $r$ the group size, and $g(i) = \lfloor i / r \rfloor$ the group index. Group-Query Attention keeps per-head query projections but shares the key/value projections within each group: $$ \begin{aligned} \text{head}_i &= \operatorname{softmax}\!\left(\frac{Q W_{Q,i} (K W_{K,g(i)})^{\top}}{\sqrt{d_h}} + B_i\right)\, (V W_{V,g(i)}),\\ n_k &= n_q / r,\quad W_{K,g}, W_{V,g} \in \mathbb{R}^{d_{\text{model}} \times d_h}. \end{aligned} $$ Each query head keeps its own projection, so the model retains $n_q$ distinct query subspaces, but during decode the runtime reuses the $g(i)$th $K,V$ pair for every head inside the group. Bias term $B_i$ can encode [[thoughts/positional embeddings|positional]] adjustments per query head without duplicating the high-bandwidth value tensors. > \[!math\] 1. grouped cache reuse > > For each time step only $n_k$ key/value tiles are fetched from device memory, so the bandwidth term shrinks from $\Theta(h d_h)$ to $\Theta(n_k d_h)$. Choosing $r=2$ halves the cache loads compared with $h=8$ multi-head without sacrificing $Q$-space diversity, matching the decode speedups reported in \[@ainslie2023gqatraininggeneralizedmultiquery\].

```jsx imports={Zoomable,KVHeadGrouping} ```