Continuous batching

(Vaswani et al., 2023) Attention operates on a sequence of query Q, key K and value V vector. Attention matrix of a sequence then computed as: A(Q, K, V) = \text{softmax}(\frac{Q \cdot K^{T}}{\sqrt{d}})V \space \space \text{ for } Q_{L \times d}, K_{L \times d}, V_{L \times d} Muti-head Attention Allows the model to jointly attend to information from different representation subspaces at different positions: \begin{aligned} \text{MHA}(Q,K,V) &= \text{concat}(\text{head}_1, \cdots, \text{head}_n) W^O \\ &\text{where } \space \text{head}_i = \text{A}(QW_i^O, KW_i^O, VW_i^O) \\ W^O & \in \mathbb{R}^{hd_v \times d_{\text{model}}} \end{aligned} Group-Query Attention by (Ainslie et al., 2023) idea: reduce number of KV heads n_k to a fraction n_k^{'} = \frac{n_q}{k} of number of query heads n_q (evenly dividing the query heads into n_k groups with r heads) RadixAttention Implemented in (Zheng et al., 2024) where they maintain a LRU eviction policy to maintain relevant KV cache for all requests within a radix tree radix tree setup: key: sequence of tokens value: KV cache tensor (stored in GPU in a paged layout) dynamic evolution of the radix tree in response to various requests.

large language models, often implemented as autoregressive transformers models. GPTs and friends Most variants of LLMs are decoder-only (Radford et al., 2019) Have “capabilities” to understand natural language.

efficient LLM serving engine.