All concepts

Grouped Query Attention

Share key/value heads across groups of query heads to shrink the KV cache.

Transformers & LLMs · Advanced · ~8 min

In plain English

Give every head its own question, but let groups of heads share one set of answers. Far less to remember, almost the same quality.

Why it's worth your time

The KV cache, not the weights, is what limits how many users you can serve at once — and this cuts it several-fold.

If you remember three things

  • Multi-head: every head has its own K/V. Multi-query: all share one
  • Grouped-query sits between, and is the modern default
  • Cuts KV cache memory by the group ratio

Overview

GQA shrinks the KV cache by letting groups of query heads share one set of key/value heads. It interpolates between multi-head attention (one K/V per query head) and multi-query attention (one K/V for all), keeping most of MHA's quality while making autoregressive serving much cheaper.

How it works

  1. Start: Many Q Heads The model keeps many query heads for expressive attention.
  2. Many Q Heads -> Grouped K/V Several query heads share the same key and value heads.
  3. Grouped K/V -> Smaller KV Cache Fewer K/V heads means less memory per generated token.
  4. Smaller KV Cache -> Faster Serving GQA preserves much of MHA quality with MQA-like serving efficiency.

In an interview

In grouped-query attention each group of query heads shares a single key/value head, so instead of 32 K/V heads you might keep 8. That cuts the per-token KV cache 4×, which speeds up memory-bound decoding, while retaining almost all multi-head quality. MQA is the extreme case with just one K/V head.

Production defaults

Groups
4–8 KV heads for 32 query heads is typical — roughly 4–8× less cache
When it matters
long contexts and high concurrency. At batch size 1 with short prompts, barely at all
Retrofitting
an existing MHA model can be uptrained to GQA with a small fraction of original compute

What breaks

  • Quality dropped after switching — Too few KV groups. Going all the way to multi-query (1 group) costs measurably more than 4–8.
  • Memory still the bottleneck — GQA shrinks the cache per token; long contexts still grow it linearly. Combine with quantized KV cache.

Watch it explained

Multi-Head Attention (MHA), Multi-Query Attention (MQA), Grouped Query Attention (GQA) Explained — DataMListic, 7:24

Related