# Transformers and Modern Attention Architectures: A Technical Survey

## 1. The Original Transformer — "Attention Is All You Need" (Vaswani et al., 2017)

The Transformer (Vaswani et al., 2017, *Attention Is All You Need*, NeurIPS) dispensed entirely with recurrence and convolutions, relying solely on attention mechanisms to model dependencies between sequence positions. This enabled full parallelization over sequence length during training and became the foundation of virtually all modern large-scale models.

### 1.1 Scaled Dot-Product Attention

Given queries $Q \in \mathbb{R}^{n \times d_k}$, keys $K \in \mathbb{R}^{m \times d_k}$, and values $V \in \mathbb{R}^{m \times d_v}$:

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

Each query is compared against all keys via dot products; the softmax converts these similarity scores into a probability distribution over positions, which is then used to compute a weighted average of the values. The scaling factor $1/\sqrt{d_k}$ is essential: for large $d_k$, the dot products $q \cdot k = \sum_{i=1}^{d_k} q_i k_i$ grow in magnitude with variance proportional to $d_k$ (assuming unit-variance components), pushing the softmax into regions of extremely small gradients. Dividing by $\sqrt{d_k}$ keeps the logits at unit variance and stabilizes training.

### 1.2 Multi-Head Attention

Rather than a single attention function over $d_{\text{model}}$-dimensional vectors, the Transformer projects $Q$, $K$, $V$ into $h$ lower-dimensional subspaces and applies attention in parallel:

$$\mathrm{MultiHead}(Q, K, V) = \mathrm{Concat}(\mathrm{head}_1, \dots, \mathrm{head}_h)\,W^O$$

$$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q,\; KW_i^K,\; VW_i^V)$$

with learned projections $W_i^Q \in \mathbb{R}^{d_{\text{model}} \times d_k}$, $W_i^K \in \mathbb{R}^{d_{\text{model}} \times d_k}$, $W_i^V \in \mathbb{R}^{d_{\text{model}} \times d_v}$, and $W^O \in \mathbb{R}^{hd_v \times d_{\text{model}}}$. In the base model, $h = 8$ and $d_k = d_v = d_{\text{model}}/h = 64$. Multiple heads allow the model to jointly attend to information from different representation subspaces at different positions — e.g., one head tracking syntactic dependencies, another tracking coreference.

### 1.3 Sinusoidal Positional Encoding

Since attention is permutation-invariant, position information must be injected. The original paper adds fixed sinusoidal encodings to the input embeddings:

$$PE_{(pos, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right), \qquad PE_{(pos, 2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$

where $pos$ is the position and $i$ indexes the dimension pair. Wavelengths form a geometric progression from $2\pi$ to $10000 \cdot 2\pi$. The key property: for any fixed offset $k$, $PE_{pos+k}$ is a linear function of $PE_{pos}$ (a rotation), allowing the model to learn relative positioning easily.

### 1.4 Encoder–Decoder Architecture, Feed-Forward, Residuals, LayerNorm

- **Encoder**: a stack of $N = 6$ identical layers, each containing (i) multi-head self-attention and (ii) a position-wise feed-forward network, each wrapped in a residual connection followed by layer normalization: $\mathrm{LayerNorm}(x + \mathrm{Sublayer}(x))$ ("post-LN"; modern models typically use pre-LN for stability).
- **Decoder**: also $N = 6$ layers, with three sub-layers: *masked* self-attention (a causal mask sets $-\infty$ on positions $j > i$ before the softmax, preserving the autoregressive property), *cross-attention* where queries come from the decoder and keys/values from the encoder output, and the feed-forward network.
- **Position-wise feed-forward network**, applied identically at each position:

$$\mathrm{FFN}(x) = \max(0,\; xW_1 + b_1)\,W_2 + b_2$$

with inner dimension $d_{ff} = 2048$ (a $4\times$ expansion over $d_{\text{model}} = 512$).

- **Layer normalization** (Ba et al., 2016): $\mathrm{LN}(x) = \gamma \odot \frac{x - \mu}{\sigma} + \beta$, where $\mu, \sigma$ are the mean and standard deviation over the feature dimension.

Self-attention costs $O(n^2 \cdot d)$ per layer in time and $O(n^2)$ in memory — the quadratic bottleneck motivating Section 4.

## 2. Positional Encoding Variants

**Learned absolute embeddings.** A trainable matrix $E_{pos} \in \mathbb{R}^{L_{\max} \times d}$ is added to token embeddings (GPT-2, BERT, ViT). Simple but does not extrapolate beyond $L_{\max}$.

**RoPE — Rotary Position Embedding** (Su et al., 2021, *RoFormer: Enhanced Transformer with Rotary Position Embedding*, arXiv:2104.09864). Instead of adding position vectors, RoPE *rotates* each 2D pair of query/key components by an angle proportional to the position $m$. For dimension pair $i$ with frequency $\theta_i = 10000^{-2i/d}$:

$$f(x, m) = R_{\Theta, m}\, x, \qquad R_{\Theta,m} = \bigoplus_{i=1}^{d/2} \begin{pmatrix} \cos m\theta_i & -\sin m\theta_i \\ \sin m\theta_i & \cos m\theta_i \end{pmatrix}$$

The crucial property is that the attention score depends only on relative position:

$$\langle f(q, m),\, f(k, n) \rangle = \langle R_{\Theta, m} q,\; R_{\Theta, n} k \rangle = q^\top R_{\Theta, n-m}\, k$$

RoPE unifies absolute encoding (applied per position) with relative behavior (in the inner product), and is used in GPT-NeoX, LLaMA, Mistral, Qwen, and most modern LLMs. Long-context extensions (position interpolation, NTK-aware scaling, YaRN) rescale its frequencies.

**ALiBi — Attention with Linear Biases** (Press et al., 2021/2022, *Train Short, Test Long*, ICLR). No embeddings at all; instead a static distance-proportional penalty is added to attention logits:

$$\mathrm{softmax}\!\left(q_i K^\top / \sqrt{d_k} \;+\; m \cdot [-(i-1), \dots, -1, 0]\right)$$

where $m$ is a fixed, head-specific slope (a geometric sequence such as $2^{-8/h}, 2^{-16/h}, \dots$). ALiBi gives strong length extrapolation: models trained on short sequences degrade gracefully at much longer inference lengths (used in BLOOM and MPT).

## 3. The Major Model Families

**BERT** (Devlin et al., 2018, *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*). Encoder-only, bidirectional attention. Pre-trained with **Masked Language Modeling**: 15% of tokens are selected; of these, 80% replaced by `[MASK]`, 10% by a random token, 10% left unchanged, and the model predicts the originals by minimizing cross-entropy over masked positions:

$$\mathcal{L}_{\text{MLM}} = -\mathbb{E}\left[\sum_{i \in \mathcal{M}} \log p_\theta(x_i \mid x_{\setminus \mathcal{M}})\right]$$

plus Next Sentence Prediction (later dropped in RoBERTa, Liu et al., 2019). Ideal for understanding/classification tasks, not generation.

**GPT** (Radford et al., 2018; GPT-2, 2019; GPT-3, Brown et al., 2020, *Language Models are Few-Shot Learners*; GPT-4, OpenAI, 2023). Decoder-only, causal attention. Trained by maximizing the autoregressive log-likelihood:

$$\mathcal{L}(\theta) = \sum_{t=1}^{T} \log p_\theta(x_t \mid x_1, \dots, x_{t-1}), \qquad p_\theta(x) = \prod_{t=1}^{T} p_\theta(x_t \mid x_{<t})$$

GPT-2 (1.5B parameters) demonstrated zero-shot transfer; GPT-3 (175B) established in-context/few-shot learning as an emergent capability of scale; GPT-4 added multimodality and RLHF-refined alignment.

**T5** (Raffel et al., 2020, *Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer*). Full encoder–decoder; every NLP task cast as text-to-text. Pre-trained with span corruption (masking contiguous spans replaced by sentinel tokens); uses relative position biases and RMSNorm-like simplifications.

**LLaMA** (Touvron et al., 2023, LLaMA and LLaMA 2; Meta, 2024, LLaMA 3). The canonical open decoder-only recipe, combining:

- **RMSNorm** (Zhang & Sennrich, 2019), pre-normalization without mean-centering:

$$\mathrm{RMSNorm}(x) = \frac{x}{\mathrm{RMS}(x)} \odot \gamma, \qquad \mathrm{RMS}(x) = \sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2 + \epsilon}$$

Cheaper than LayerNorm (no mean subtraction, no bias) with equal or better stability.

- **SwiGLU** feed-forward (Shazeer, 2020, *GLU Variants Improve Transformer*):

$$\mathrm{FFN}_{\text{SwiGLU}}(x) = \left(\mathrm{Swish}_1(xW_1) \otimes xW_3\right)W_2, \qquad \mathrm{Swish}_\beta(x) = x\,\sigma(\beta x)$$

a gated linear unit with SiLU gating and three weight matrices (inner dimension scaled to $\tfrac{2}{3} \cdot 4d$ to keep parameter count constant).

- **RoPE** for positions, and **GQA** (grouped-query attention, Section 4) from LLaMA 2 70B onward.

## 4. Efficient Attention

**Sparse Transformers** (Child et al., 2019, *Generating Long Sequences with Sparse Transformers*). Factorize the full attention matrix into strided and local patterns so each position attends to $O(\sqrt{n})$ others, reducing complexity to $O(n\sqrt{n})$. Precursor to Longformer and BigBird (sliding window + global + random attention).

**Linformer** (Wang et al., 2020). Exploits the empirically low rank of the attention matrix: project keys and values along the sequence axis with learned matrices $E, F \in \mathbb{R}^{k \times n}$, giving $\mathrm{softmax}\big(Q(EK)^\top/\sqrt{d_k}\big)(FV)$ — linear $O(nk)$ complexity.

**Performer** (Choromanski et al., 2020, *Rethinking Attention with Performers*). Approximates the softmax kernel with random features (FAVOR+): $\exp(q^\top k) \approx \phi(q)^\top \phi(k)$ where $\phi$ uses positive orthogonal random features. Attention then factorizes as $\phi(Q)\big(\phi(K)^\top V\big)$, computed in $O(n)$ by changing the multiplication order.

**FlashAttention** (Dao et al., 2022, *FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness*, NeurIPS; FlashAttention-2, 2023; FlashAttention-3, 2024). Not an approximation — an **exact**, IO-aware algorithm. Key insight: the bottleneck is data movement between GPU high-bandwidth memory (HBM) and on-chip SRAM, not FLOPs. FlashAttention (i) **tiles** $Q, K, V$ into blocks that fit in SRAM, (ii) computes the softmax incrementally with the **online softmax** trick (maintaining running max $m$ and normalizer $\ell$ per row, rescaling partial outputs as new blocks arrive), and (iii) never materializes the $n \times n$ attention matrix, **recomputing** it during the backward pass. Memory drops from $O(n^2)$ to $O(n)$, with 2–4× wall-clock speedups; it is now standard in every LLM stack.

**Sliding Window Attention** (Mistral 7B, Jiang et al., 2023). Each token attends only to the previous $W$ tokens ($W = 4096$); with $L$ layers, information still propagates over $L \times W$ positions through the stacked receptive field. Combined with a rolling KV cache of fixed size $W$.

**Multi-Query and Grouped-Query Attention.** **MQA** (Shazeer, 2019, *Fast Transformer Decoding*): all $h$ query heads share a *single* K/V head, shrinking the KV cache by a factor $h$ and dramatically accelerating decoding, at a slight quality cost. **GQA** (Ainslie et al., 2023, *GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints*): interpolates between MHA and MQA by grouping the $h$ query heads into $g$ groups, each sharing one K/V head ($g = h$ recovers MHA, $g = 1$ recovers MQA). LLaMA 2/3 70B use $g = 8$, achieving near-MHA quality at near-MQA speed.

## 5. Mixture of Experts (MoE)

Sparse MoE (Shazeer et al., 2017, *Outrageously Large Neural Networks*) replaces the dense FFN with $E$ expert FFNs plus a learned router; each token activates only $k \ll E$ experts, decoupling parameter count from per-token compute.

**Gating equation.** With router weights $W_g$:

$$G(x) = \mathrm{softmax}\big(\mathrm{TopK}(x \cdot W_g,\; k)\big), \qquad y = \sum_{i \in \mathrm{TopK}} G(x)_i \cdot E_i(x)$$

where $\mathrm{TopK}$ sets non-selected logits to $-\infty$. An auxiliary **load-balancing loss** prevents router collapse:

$$\mathcal{L}_{\text{aux}} = \alpha \cdot E \cdot \sum_{i=1}^{E} f_i \cdot P_i$$

with $f_i$ the fraction of tokens dispatched to expert $i$ and $P_i$ the mean router probability for expert $i$.

**Switch Transformer** (Fedus, Zoph & Shazeer, 2021/2022, JMLR). Simplified to **top-1 routing** ($k = 1$) — each token goes to exactly one expert, with the router probability as multiplicative weight — plus capacity factors and selective precision, scaling stably to 1.6 trillion parameters with 7× pre-training speedup over T5 at equal FLOPs.

**Mixtral 8×7B** (Mistral AI, Jiang et al., 2024). 8 experts per layer, **top-2 routing**. Total 47B parameters but only ~13B active per token; matched or exceeded LLaMA 2 70B and GPT-3.5 on most benchmarks. The same design underlies GPT-4 (reported), DeepSeek-V2/V3 (fine-grained + shared experts), and Gemini 1.5.

## 6. Vision Transformers

**ViT** (Dosovitskiy et al., 2020/2021, *An Image is Worth 16×16 Words*, ICLR). An image $x \in \mathbb{R}^{H \times W \times C}$ is split into $N = HW/P^2$ non-overlapping patches of size $P \times P$ (typically 16×16), each flattened and linearly projected to dimension $D$ by $E \in \mathbb{R}^{(P^2 C) \times D}$. A learnable `[class]` token is prepended and learned position embeddings added:

$$z_0 = [x_{\text{class}};\; x_p^1 E;\; x_p^2 E;\; \dots;\; x_p^N E] + E_{pos}, \qquad E_{pos} \in \mathbb{R}^{(N+1) \times D}$$

Then standard pre-LN Transformer encoder blocks:

$$z'_\ell = \mathrm{MSA}(\mathrm{LN}(z_{\ell-1})) + z_{\ell-1}, \qquad z_\ell = \mathrm{MLP}(\mathrm{LN}(z'_\ell)) + z'_\ell$$

with classification from $\mathrm{LN}(z_L^0)$. ViT lacks convolutional inductive biases (locality, translation equivariance), so it underperforms CNNs on small data but surpasses them when pre-trained on large datasets (JFT-300M).

**DeiT** (Touvron et al., 2021, *Training Data-Efficient Image Transformers & Distillation Through Attention*). Matches ViT quality using ImageNet-1k only, via strong augmentation/regularization and a **distillation token** that learns from a CNN teacher's hard labels through attention.

**Swin Transformer** (Liu et al., 2021, ICCV best paper). Hierarchical ViT for dense prediction: attention computed within non-overlapping local windows ($M \times M = 7 \times 7$ patches), giving **linear** complexity in image size versus ViT's quadratic; **shifted windows** in alternating layers ($\lfloor M/2 \rfloor$ displacement) enable cross-window information flow; patch merging builds a multi-scale feature pyramid usable by detection/segmentation heads. Complexity per window layer: $\Omega(\mathrm{W\text{-}MSA}) = 4hwC^2 + 2M^2hwC$, linear in $hw$.

## 7. Multimodal Models

**CLIP** (Radford et al., 2021, *Learning Transferable Visual Models From Natural Language Supervision*). Dual encoders (image + text) trained on 400M web pairs with a **symmetric InfoNCE contrastive loss**. For a batch of $N$ pairs with L2-normalized embeddings $I_i, T_i$ and learned temperature $\tau$:

$$\mathcal{L} = \frac{1}{2}\left[ -\frac{1}{N}\sum_{i=1}^{N} \log \frac{\exp(I_i \cdot T_i / \tau)}{\sum_{j=1}^{N} \exp(I_i \cdot T_j / \tau)} \;-\; \frac{1}{N}\sum_{i=1}^{N} \log \frac{\exp(I_i \cdot T_i / \tau)}{\sum_{j=1}^{N} \exp(I_j \cdot T_i / \tau)} \right]$$

i.e., cross-entropy over the $N \times N$ cosine-similarity matrix, applied both image→text and text→image. Enables zero-shot classification by embedding class names as prompts ("a photo of a {class}"). SigLIP (Zhai et al., 2023) replaces the softmax with a pairwise sigmoid loss.

**Flamingo** (Alayrac et al., 2022, DeepMind). Bridges a *frozen* vision encoder and a *frozen* LLM (Chinchilla) using a **Perceiver Resampler** (compressing variable visual features into a fixed set of latents) and interleaved **gated cross-attention** layers ($\tanh$-gated, initialized at zero so the LLM starts unperturbed). Handles arbitrarily interleaved image-text sequences; strong few-shot visual learning.

**LLaVA** (Liu et al., 2023, *Visual Instruction Tuning*). Minimalist recipe: CLIP ViT-L/14 features mapped into the LLM (Vicuna) token space by a simple linear projection (an MLP in LLaVA-1.5), then **visual instruction tuning** on GPT-4-generated multimodal conversations. Established the dominant open-source VLM template (adopted conceptually by Qwen-VL, InternVL, etc.).

## 8. State Space Models as an Alternative

SSMs replace attention with a linear dynamical system, offering $O(n)$ scaling and constant-memory recurrent inference.

**Continuous formulation.** A 1D input $u(t)$ maps to output $y(t)$ through a hidden state $h(t) \in \mathbb{R}^N$:

$$h'(t) = A\,h(t) + B\,u(t), \qquad y(t) = C\,h(t) \;(+\, D\,u(t))$$

**Discretization** with step size $\Delta$ via zero-order hold (ZOH):

$$\bar{A} = \exp(\Delta A), \qquad \bar{B} = (\Delta A)^{-1}\big(\exp(\Delta A) - I\big)\,\Delta B$$

$$h_t = \bar{A}\,h_{t-1} + \bar{B}\,u_t, \qquad y_t = C\,h_t$$

**S4** (Gu, Goel & Ré, 2021, *Efficiently Modeling Long Sequences with Structured State Spaces*, ICLR 2022). Uses HiPPO-initialized structured $A$ matrices for long-range memory; because the system is **linear time-invariant (LTI)**, the recurrence unrolls into a convolution $y = u * \bar{K}$ with kernel $\bar{K} = (C\bar{B},\, C\bar{A}\bar{B},\, C\bar{A}^2\bar{B}, \dots)$, computable in $O(n \log n)$ via FFT. Dominated the Long Range Arena benchmark but lagged Transformers on language.

**Mamba** (Gu & Dao, 2023, *Mamba: Linear-Time Sequence Modeling with Selective State Spaces*, arXiv:2312.00752). The **selective SSM (S6)**: makes $B_t$, $C_t$, and $\Delta_t$ **functions of the input** $u_t$ (e.g., $\Delta_t = \mathrm{softplus}(W_\Delta u_t)$), so the model can selectively remember or forget content — recovering a data-dependent gating that LTI SSMs cannot express:

$$h_t = \bar{A}_t\, h_{t-1} + \bar{B}_t\, u_t, \qquad y_t = C_t^\top h_t$$

Input dependence breaks the convolutional shortcut, so Mamba uses a **hardware-aware parallel scan** (associative scan with kernel fusion, keeping states in SRAM — FlashAttention-style IO-awareness). Mamba-3B matched Transformers of twice its size with linear-time training and $O(1)$-memory inference. Mamba-2 (Dao & Gu, 2024) established the SSM–attention duality (SSD); hybrids (Jamba, Zamba, Nemotron-H) interleave Mamba and attention layers.

**RWKV** (Peng et al., 2023, *RWKV: Reinventing RNNs for the Transformer Era*). A linear-attention RNN with channel-wise time decay $w$ — WKV mechanism: $wkv_t = \frac{\sum_{i<t} e^{-(t-1-i)w + k_i} v_i + e^{u+k_t} v_t}{\sum_{i<t} e^{-(t-1-i)w + k_i} + e^{u+k_t}}$ — trainable in parallel like a Transformer, deployable as a pure RNN with constant memory; scaled to 14B+ parameters.

**Hyena** (Poli et al., 2023). Replaces attention with interleaved **implicitly parametrized long convolutions** (filters generated by an MLP over positional encodings) and element-wise multiplicative gating, achieving sub-quadratic $O(n \log n)$ complexity and matching Transformer perplexity at reduced compute; basis of genomic models (HyenaDNA) and Striped Hyena / Evo.

## 9. Scaling Laws

**Kaplan et al., 2020** (*Scaling Laws for Neural Language Models*, OpenAI). Cross-entropy loss follows power laws in parameters $N$, dataset tokens $D$, and compute $C$ over many orders of magnitude:

$$L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}, \quad L(D) = \left(\frac{D_c}{D}\right)^{\alpha_D}, \quad L(C) = \left(\frac{C_c}{C}\right)^{\alpha_C}$$

with $\alpha_N \approx 0.076$, $\alpha_D \approx 0.095$, $\alpha_C \approx 0.050$, and a combined form $L(N, D) = \left[\left(\frac{N_c}{N}\right)^{\alpha_N/\alpha_D} + \frac{D_c}{D}\right]^{\alpha_D}$. Kaplan's prescription — grow $N$ much faster than $D$ ($N \propto C^{0.73}$) — led to under-trained giants like GPT-3 and Gopher.

**Chinchilla — Hoffmann et al., 2022** (*Training Compute-Optimal Large Language Models*, DeepMind). Refit with corrected methodology (three approaches, including IsoFLOP profiles), yielding the parametric loss:

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

with fitted values $E \approx 1.69$ (irreducible entropy of text), $A \approx 406.4$, $B \approx 410.7$, $\alpha \approx 0.34$, $\beta \approx 0.28$. Minimizing $L$ subject to $C \approx 6ND$ gives $N_{\text{opt}} \propto C^{a}$, $D_{\text{opt}} \propto C^{b}$ with $a \approx b \approx 0.5$: **parameters and tokens should scale equally**, at roughly **~20 tokens per parameter**. Chinchilla (70B, 1.4T tokens) beat Gopher (280B, 300B tokens) at identical compute. Modern practice (LLaMA 3 trained on 15T tokens) deliberately "over-trains" past Chinchilla-optimal to minimize *inference* cost. Note: Epoch AI's replication (Besiroglu et al., 2024) found minor fitting inconsistencies in Approach 3 but confirmed the ~20 tokens/parameter conclusion.

## 10. Kolmogorov–Arnold Networks (KAN, 2024)

**Theoretical basis.** The Kolmogorov–Arnold representation (superposition) theorem (Kolmogorov, 1957; Arnold): any continuous function $f: [0,1]^n \to \mathbb{R}$ can be written as

$$f(x_1, \dots, x_n) = \sum_{q=1}^{2n+1} \Phi_q\!\left(\sum_{p=1}^{n} \phi_{q,p}(x_p)\right)$$

where $\Phi_q: \mathbb{R} \to \mathbb{R}$ and $\phi_{q,p}: [0,1] \to \mathbb{R}$ are continuous **univariate** functions — multivariate functions decompose into sums and compositions of 1D functions.

**KAN** (Liu et al., 2024, *KAN: Kolmogorov–Arnold Networks*, arXiv:2404.19756; ICLR 2025). Where an MLP layer computes $\sigma(Wx + b)$ — **fixed** activations on nodes, **learnable linear weights** on edges — a KAN layer places a **learnable univariate function on every edge** and simply sums at nodes:

$$x_{l+1, j} = \sum_{i=1}^{n_l} \phi_{l, j, i}(x_{l, i}), \qquad \mathrm{KAN}(x) = (\Phi_{L-1} \circ \cdots \circ \Phi_1 \circ \Phi_0)(x)$$

Each edge function is parametrized as a B-spline plus a residual basis:

$$\phi(x) = w_b\, \mathrm{silu}(x) + w_s \sum_{i} c_i\, B_i(x)$$

with learnable spline coefficients $c_i$ on a grid that can be progressively refined. KANs generalize the depth-2, width-$(2n+1)$ theorem to arbitrary depths and widths (the authors stress it is *inspired by*, not an exact implementation of, the theorem).

**Differences from MLPs.** (i) Learnable activations on edges vs. fixed activations on nodes; (ii) no linear weight matrices — every weight is replaced by a 1D function; (iii) empirically favorable neural scaling on scientific/symbolic-regression tasks, with better accuracy at small scale; (iv) high interpretability — learned splines can be visualized, pruned, and symbolically identified ($\sin$, $x^2$, $\exp$, …), making KANs attractive for physics and "AI for Science"; (v) drawbacks: slower training (spline evaluations parallelize less efficiently than GEMMs) and unproven advantages at LLM scale. Variants include FastKAN (RBFs), Chebyshev-KAN, and KAN 2.0 (Liu et al., 2024).

---

## Key References

1. Vaswani, A. et al. (2017). *Attention Is All You Need*. NeurIPS.
2. Devlin, J. et al. (2018). *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*. NAACL 2019.
3. Radford, A. et al. (2018, 2019); Brown, T. et al. (2020). GPT, GPT-2, *Language Models are Few-Shot Learners* (GPT-3). OpenAI (2023), *GPT-4 Technical Report*.
4. Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer* (T5). JMLR.
5. Zhang, B. & Sennrich, R. (2019). *Root Mean Square Layer Normalization*. NeurIPS.
6. Shazeer, N. (2019). *Fast Transformer Decoding: One Write-Head is All You Need* (MQA); (2020) *GLU Variants Improve Transformer* (SwiGLU).
7. Su, J. et al. (2021). *RoFormer: Enhanced Transformer with Rotary Position Embedding*. arXiv:2104.09864.
8. Press, O., Smith, N. & Lewis, M. (2022). *Train Short, Test Long: Attention with Linear Biases* (ALiBi). ICLR.
9. Child, R. et al. (2019). *Generating Long Sequences with Sparse Transformers*; Wang, S. et al. (2020) *Linformer*; Choromanski, K. et al. (2020) *Performer*.
10. Dao, T. et al. (2022). *FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness*. NeurIPS; Dao (2023) FlashAttention-2.
11. Ainslie, J. et al. (2023). *GQA: Training Generalized Multi-Query Transformer Models*. EMNLP.
12. Shazeer, N. et al. (2017). *Outrageously Large Neural Networks* (sparse MoE); Fedus, W., Zoph, B. & Shazeer, N. (2022). *Switch Transformers*. JMLR; Jiang, A. et al. (2023, 2024). *Mistral 7B*; *Mixtral of Experts*.
13. Touvron, H. et al. (2023). *LLaMA*; *LLaMA 2*; (2021) *DeiT*.
14. Dosovitskiy, A. et al. (2021). *An Image is Worth 16×16 Words* (ViT). ICLR; Liu, Z. et al. (2021). *Swin Transformer*. ICCV.
15. Radford, A. et al. (2021). *Learning Transferable Visual Models From Natural Language Supervision* (CLIP). ICML; Alayrac, J.-B. et al. (2022). *Flamingo*. NeurIPS; Liu, H. et al. (2023). *Visual Instruction Tuning* (LLaVA). NeurIPS.
16. Gu, A., Goel, K. & Ré, C. (2022). *Efficiently Modeling Long Sequences with Structured State Spaces* (S4). ICLR; Gu, A. & Dao, T. (2023). *Mamba*. arXiv:2312.00752; Peng, B. et al. (2023). *RWKV*. EMNLP Findings; Poli, M. et al. (2023). *Hyena Hierarchy*. ICML.
17. Kaplan, J. et al. (2020). *Scaling Laws for Neural Language Models*. arXiv:2001.08361; Hoffmann, J. et al. (2022). *Training Compute-Optimal Large Language Models* (Chinchilla). NeurIPS.
18. Liu, Z. et al. (2024). *KAN: Kolmogorov–Arnold Networks*. arXiv:2404.19756 / ICLR 2025.

Sources consulted during verification: [NeurIPS — Attention Is All You Need](https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf), [arXiv 2104.09864 — RoFormer](https://arxiv.org/pdf/2104.09864), [EleutherAI — Rotary Embeddings](https://blog.eleuther.ai/rotary-embeddings/), [arXiv 2312.00752 — Mamba](https://arxiv.org/abs/2312.00752), [A Visual Guide to Mamba](https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state), [Epoch AI — Chinchilla replication](https://epoch.ai/blog/chinchilla-scaling-a-replication-attempt), [lifearchitect.ai — Chinchilla](https://lifearchitect.ai/chinchilla/), [Wikipedia — Kolmogorov-Arnold Networks](https://en.wikipedia.org/wiki/Kolmogorov-Arnold_Networks), [ICLR 2025 — KAN](https://proceedings.iclr.cc/paper_files/paper/2025/file/afaed89642ea100935e39d39a4da602c-Paper-Conference.pdf), [IBM — Mixture of Experts](https://www.ibm.com/think/topics/mixture-of-experts), [Switch Transformer routing](https://mbrenndoerfer.com/writing/switch-transformer-top-1-routing-trillion-parameter-scaling), [Lil'Log — Contrastive Representation Learning](https://lilianweng.github.io/posts/2021-05-31-contrastive/), [EmergentMind — CLIP](https://www.emergentmind.com/topics/clip), [NeurIPS — FlashAttention](https://proceedings.neurips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html), [AI Summer — ViT](https://theaisummer.com/vision-transformer/), [TinyLlama (LLaMA components)](https://arxiv.org/html/2401.02385v2).