# The Complete Taxonomy of Neural Networks

**A comprehensive, equation-level reference covering every major family of neural network architectures — from the McCulloch–Pitts neuron (1943) to diffusion transformers, Mamba, KAN, and JEPA (2026).**

Compiled from a large-scale parallel web research sweep (six independent research passes, each validating equations, authors, dates, and founding papers against primary sources — arXiv, NeurIPS/ICML/ICLR proceedings, Nature/Science, and authoritative references).

---

## Table of Contents

**Part I — Foundations of Artificial Neural Networks**
Biological vs. artificial neuron · McCulloch–Pitts (1943) · Perceptron (Rosenblatt 1958) & convergence theorem · XOR / Minsky–Papert · ADALINE/MADALINE (delta rule) · Multilayer Perceptron & forward propagation · Universal Approximation Theorems (Cybenko, Hornik, Leshno) · Backpropagation (full derivation) · Activation functions (sigmoid → Mish, with derivatives) · Loss functions (MSE, MAE, Huber, cross-entropy, hinge) · Optimizers (SGD → AdamW, full update equations) · Regularization (L1/L2, dropout, BatchNorm, LayerNorm) · RBF networks

**Part II — Convolutional Neural Networks**
Neocognitron (1980) · Convolution math (stride, padding, dilation, output-size formula) · Pooling · LeNet-5 · AlexNet · VGG · GoogLeNet/Inception · ResNet (residual equation & gradient analysis) · DenseNet · MobileNet (depthwise separable convolution) · EfficientNet (compound scaling) · ConvNeXt · Object detection (R-CNN family, YOLO loss, SSD) · Segmentation (FCN, U-Net, Mask R-CNN) · 1D & 3D CNNs

**Part III — Recurrent Networks and Sequence Models**
Elman/Jordan RNNs · BPTT & vanishing/exploding gradients (eigenvalue analysis) · LSTM (complete gate equations) · GRU · Bidirectional & stacked RNNs · Seq2Seq · Bahdanau & Luong attention · Echo State Networks / reservoir computing / Liquid State Machines · Hopfield networks (classical & modern) · Boltzmann machines, RBMs, contrastive divergence, Deep Belief Networks · Temporal Convolutional Networks · Neural Turing Machines & Differentiable Neural Computers

**Part IV — Transformers and Modern Attention**
Scaled dot-product & multi-head attention · Sinusoidal / RoPE / ALiBi positional encodings · BERT, GPT, T5, LLaMA (RMSNorm, SwiGLU, GQA) · Efficient attention (Sparse, Linformer, Performer, FlashAttention, sliding window, MQA/GQA) · Mixture of Experts (gating & load balancing, Switch, Mixtral) · Vision Transformers (ViT, DeiT, Swin) · Multimodal (CLIP InfoNCE, Flamingo, LLaVA) · State Space Models (S4, Mamba, RWKV, Hyena) · Scaling laws (Kaplan, Chinchilla) · Kolmogorov–Arnold Networks (KAN)

**Part V — Generative Models**
Autoencoders (denoising, sparse, contractive) · VAE (ELBO derivation, reparameterization, β-VAE, VQ-VAE) · GANs (minimax game, DCGAN, cGAN, WGAN/WGAN-GP, StyleGAN 1–3, Pix2Pix, CycleGAN) · Normalizing flows (change of variables, RealNVP, Glow, MAF/IAF) · Diffusion (DDPM full equations, DDIM, score-based SDE, classifier-free guidance, latent diffusion, flow matching) · Autoregressive (PixelCNN, WaveNet) · Energy-based models · DALL·E, Imagen, DiT, Sora, consistency models

**Part VI — Specialized and Emerging Architectures**
Graph Neural Networks (message passing, GCN, GraphSAGE, GAT, GIN, AlphaFold) · Spiking Neural Networks (LIF, Hodgkin–Huxley, Izhikevich, STDP, neuromorphic hardware) · Self-Organizing Maps (Kohonen) · Capsule Networks (dynamic routing) · Neural ODEs (adjoint method) · Physics-Informed Neural Networks · NeRF (volume rendering) · SIREN / implicit representations · Deep RL networks (DQN, REINFORCE, actor-critic, PPO, AlphaZero/MuZero) · Siamese networks & metric learning (contrastive, triplet) · ELM, DEQ, HyperNetworks, NAS/DARTS, binarized networks, Bayesian NNs, Liquid Neural Networks, World Models & JEPA

---
# Foundations of Artificial Neural Networks

## 1. From the Biological Neuron to the Artificial Neuron

The artificial neuron is a radical abstraction of its biological counterpart. A biological neuron receives electrochemical signals through its **dendrites**, integrates them in the **soma** (cell body), and — if the aggregated membrane depolarization crosses a threshold — emits an **action potential** that propagates along the **axon** to **synapses** connecting to downstream neurons. Two properties of this system are retained in the mathematical abstraction: (i) synaptic transmission is *weighted* (a synapse may be excitatory or inhibitory, strong or weak), and (ii) firing is *nonlinear and threshold-based* (all-or-none).

Everything else is discarded: spike timing, refractory periods, dendritic compartmentalization, neuromodulation, and the fact that biological learning is local and largely unsupervised. The artificial neuron is therefore best understood as a *computational primitive inspired by* — not a *model of* — neurophysiology.

### The McCulloch–Pitts neuron (1943)

The foundational text is Warren S. McCulloch and Walter Pitts, *"A Logical Calculus of the Ideas Immanent in Nervous Activity"*, **Bulletin of Mathematical Biophysics**, 5(4):115–133, 1943. It is widely credited as a seminal contribution to neural network theory, automata theory, the theory of computation, and cybernetics.

The MP neuron takes Boolean inputs $x_i \in \{0,1\}$ and produces a Boolean output via a threshold (Heaviside) function:

$$y = \Theta\!\left(\sum_{i=1}^{n} w_i x_i - \theta\right), \qquad \Theta(u) = \begin{cases} 1 & \text{if } u \geq 0 \\ 0 & \text{if } u < 0 \end{cases}$$

In the original formulation, weights are fixed at $w_i = +1$ for excitatory inputs, and inhibitory inputs are *absolute*: a single active inhibitory input vetoes firing regardless of the excitatory sum. Writing $x_1,\dots,x_n$ for excitatory and $z_1,\dots,z_m$ for inhibitory inputs:

$$y = \Theta\!\left(\sum_{i=1}^{n} x_i - \theta\right)\prod_{j=1}^{m}(1 - z_j)$$

McCulloch and Pitts showed that networks of such units can implement any Boolean function — AND ($\theta = n$), OR ($\theta = 1$), NOT (via inhibition) — and with cycles, any finite-state automaton. The decisive limitation is that **the MP neuron does not learn**: $w_i$ and $\theta$ are set by the designer.

---

## 2. The Perceptron (Rosenblatt, 1958)

Frank Rosenblatt introduced the perceptron in *"The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain"*, **Psychological Review**, 65(6):386–408, 1958, developed at length in *Principles of Neurodynamics* (1962). The critical advance over McCulloch–Pitts is a **learning procedure**: weights are adjusted from examples rather than hand-designed.

### Output equation

With real-valued inputs $\mathbf{x} \in \mathbb{R}^n$, weights $\mathbf{w} \in \mathbb{R}^n$, and bias $b$ (equivalently, a negative threshold):

$$z = \mathbf{w}^\top \mathbf{x} + b = \sum_{i=1}^{n} w_i x_i + b, \qquad \hat{y} = \operatorname{sign}(z) = \begin{cases} +1 & z \geq 0 \\ -1 & z < 0\end{cases}$$

The bias is conventionally absorbed by augmenting $\mathbf{x} \leftarrow (\mathbf{x}, 1)$ and $\mathbf{w} \leftarrow (\mathbf{w}, b)$, giving $\hat{y} = \operatorname{sign}(\mathbf{w}^\top \mathbf{x})$. Geometrically, the perceptron defines a **hyperplane** $\mathbf{w}^\top \mathbf{x} + b = 0$ splitting input space into two half-spaces.

### The perceptron learning rule

For each misclassified example $(\mathbf{x}^{(k)}, y^{(k)})$, with learning rate $\eta > 0$:

$$\mathbf{w} \leftarrow \mathbf{w} + \eta\left(y^{(k)} - \hat{y}^{(k)}\right)\mathbf{x}^{(k)}, \qquad b \leftarrow b + \eta\left(y^{(k)} - \hat{y}^{(k)}\right)$$

In the $\pm1$ convention this simplifies: correctly classified points produce no update, and a misclassified point gives $\mathbf{w} \leftarrow \mathbf{w} + \eta\, y^{(k)} \mathbf{x}^{(k)}$. This is error-driven, online, and requires no differentiability — the step function's derivative is zero almost everywhere, so this is *not* gradient descent on the 0-1 loss. It is, however, equivalent to stochastic subgradient descent on the **perceptron criterion** $L = \max(0, -y\,\mathbf{w}^\top\mathbf{x})$.

### The perceptron convergence theorem

Formalized by Novikoff (*"On Convergence Proofs for Perceptrons"*, 1962): suppose a unit vector $\mathbf{w}^\star$ with $\|\mathbf{w}^\star\| = 1$ separates the data with margin $\gamma > 0$, i.e. $y_i(\mathbf{w}^{\star\top}\mathbf{x}_i) \geq \gamma$ for all $i$, and let $R = \max_i \|\mathbf{x}_i\|$. Then the perceptron algorithm makes at most

$$T \leq \left(\frac{R}{\gamma}\right)^2$$

updates before converging. Note the bound is independent of dimension and of the number of samples — it depends only on the normalized margin $\gamma/R$.

### Limits: Minsky & Papert and the XOR problem

Marvin Minsky and Seymour Papert, *Perceptrons: An Introduction to Computational Geometry* (MIT Press, 1969), gave a rigorous analysis of what single-layer perceptrons cannot represent. The canonical counterexample is **XOR**:

| $x_1$ | $x_2$ | XOR |
|---|---|---|
| 0 | 0 | 0 |
| 0 | 1 | 1 |
| 1 | 0 | 1 |
| 1 | 1 | 0 |

Suppose a separating line existed. Then $b < 0$ (from $(0,0)\mapsto 0$), $w_1 + b \geq 0$ and $w_2 + b \geq 0$ (from the positive cases), hence $w_1 + w_2 + 2b \geq 0$, so $w_1 + w_2 + b \geq -b > 0$ — which forces $(1,1)\mapsto 1$, a contradiction. The positive vertices lie on one diagonal of the unit square, the negative on the other; no line separates two diagonals.

Minsky and Papert also proved deeper results — e.g. that the **parity** and **connectedness** predicates require perceptrons whose order (number of inputs per feature detector) grows with the input size. Their pessimism about multilayer extensions, combined with a lack of a training algorithm for hidden layers, is commonly cited as a trigger for the first "AI winter" in connectionism, lasting roughly until 1986.

Crucially, XOR *is* solvable by two layers: $\text{XOR}(x_1,x_2) = \text{OR}(x_1,x_2) \wedge \neg\text{AND}(x_1,x_2)$, i.e. one hidden layer of two threshold units plus an output unit.

---

## 3. ADALINE and MADALINE (Widrow & Hoff, 1960)

Bernard Widrow and Marcian E. Hoff, *"Adaptive Switching Circuits"*, **IRE WESCON Convention Record**, Part 4, pp. 96–104, 1960, introduced **ADALINE** (ADAptive LINear Element / Adaptive Linear Neuron).

The key difference from the perceptron: the **error is computed on the linear pre-activation**, not on the thresholded output. Let $z = \mathbf{w}^\top\mathbf{x} + b$ and target $d$. The objective is squared error:

$$L(\mathbf{w}) = \tfrac{1}{2}(d - z)^2$$

Differentiating, $\partial L/\partial w_i = -(d-z)x_i$, giving the **delta rule** (also known as the Widrow–Hoff rule or the **LMS**, least-mean-squares, algorithm):

$$\Delta w_i = \eta\,(d - z)\,x_i, \qquad \mathbf{w} \leftarrow \mathbf{w} + \eta\,(d - z)\,\mathbf{x}$$

Because $z$ is differentiable in $\mathbf{w}$, this *is* genuine stochastic gradient descent, and it converges (in the mean) for $0 < \eta < 2/\lambda_{\max}$ where $\lambda_{\max}$ is the largest eigenvalue of the input autocorrelation matrix $R = \mathbb{E}[\mathbf{x}\mathbf{x}^\top]$. Unlike the perceptron rule, LMS keeps improving even when the data are already separated, driving the solution toward the minimum-MSE hyperplane — and it does not diverge on non-separable data.

**MADALINE** (Many ADALINEs) stacks ADALINE units into a layered network with a fixed voting/majority output unit. MADALINE Rule I (MRI, 1962) adapts the hidden ADALINE whose pre-activation is closest to zero — the "minimum disturbance" principle — flipping it if this reduces error. MADALINE Rule II (MRII, Winter & Widrow, 1988) generalizes the trial-and-adopt search to multiple layers. MADALINE III later approximated backpropagation with sigmoidal units. LMS/ADALINE became one of the first industrially deployed neural methods, notably in adaptive echo cancellation and channel equalization in telephony, and remains the workhorse of adaptive signal processing.

---

## 4. The Multilayer Perceptron (MLP)

An MLP is a **feedforward** network of $L$ layers where each layer applies an affine map followed by a pointwise nonlinearity.

### Forward propagation, layer by layer

Let $\mathbf{a}^{[0]} = \mathbf{x} \in \mathbb{R}^{n_0}$. For $\ell = 1, \dots, L$ with $\mathbf{W}^{[\ell]} \in \mathbb{R}^{n_\ell \times n_{\ell-1}}$ and $\mathbf{b}^{[\ell]} \in \mathbb{R}^{n_\ell}$:

$$\mathbf{z}^{[\ell]} = \mathbf{W}^{[\ell]}\mathbf{a}^{[\ell-1]} + \mathbf{b}^{[\ell]}$$
$$\mathbf{a}^{[\ell]} = \sigma^{[\ell]}\!\left(\mathbf{z}^{[\ell]}\right)$$

with the network output $\hat{\mathbf{y}} = \mathbf{a}^{[L]}$. Componentwise, $z_j^{[\ell]} = \sum_{i} W_{ji}^{[\ell]} a_i^{[\ell-1]} + b_j^{[\ell]}$.

In **batched** form with a design matrix $\mathbf{X} \in \mathbb{R}^{m \times n_0}$ ($m$ examples as rows), $\mathbf{Z}^{[\ell]} = \mathbf{A}^{[\ell-1]}\mathbf{W}^{[\ell]\top} + \mathbf{1}_m \mathbf{b}^{[\ell]\top}$, which maps directly onto GEMM primitives — the reason MLPs are hardware-efficient.

The nonlinearity is essential: composing affine maps yields an affine map, so a network of any depth with $\sigma = \text{id}$ collapses to a single linear layer.

### The Universal Approximation Theorem

**Cybenko (1989)** — George Cybenko, *"Approximation by Superpositions of a Sigmoidal Function"*, **Mathematics of Control, Signals and Systems**, 2(4):303–314, 1989 — proved that finite sums of the form

$$G(\mathbf{x}) = \sum_{j=1}^{N} \alpha_j\, \sigma\!\left(\mathbf{w}_j^\top\mathbf{x} + \theta_j\right)$$

are **dense** in $C(I_n)$, the continuous functions on the unit hypercube $I_n = [0,1]^n$, under the uniform norm, whenever $\sigma$ is any continuous **sigmoidal** function ($\sigma(t)\to 1$ as $t\to+\infty$, $\sigma(t)\to 0$ as $t\to-\infty$). Formally: for any $f \in C(I_n)$ and $\varepsilon > 0$ there exists such a $G$ with $|G(\mathbf{x}) - f(\mathbf{x})| < \varepsilon$ for all $\mathbf{x} \in I_n$. Cybenko's proof is non-constructive, relying on the Hahn–Banach theorem and the Riesz representation theorem to show that the closure of the span of these functions cannot be a proper subspace.

**Hornik, Stinchcombe & White (1989)** — *"Multilayer Feedforward Networks are Universal Approximators"*, **Neural Networks**, 2(5):359–366 — obtained the result independently and more generally, showing that single-hidden-layer networks with any *squashing* activation are universal approximators for Borel measurable functions, in $L^p(\mu)$ for arbitrary finite measures $\mu$, and with derivatives (Hornik, 1991, *"Approximation Capabilities of Multilayer Feedforward Networks"*, **Neural Networks** 4(2):251–257).

**Leshno, Lin, Pinkus & Schocken (1993)** — *"Multilayer Feedforward Networks with a Nonpolynomial Activation Function Can Approximate Any Function"*, **Neural Networks**, 6(6):861–867 — gave the sharpest classical statement: a network with a locally bounded, piecewise-continuous activation is a universal approximator **if and only if the activation is not a polynomial**. This is why ReLU, despite not being sigmoidal, is universal.

Two caveats matter in practice. First, these are **existence** results: they say nothing about how many hidden units are needed (the width $N$ may be exponential in $n$), nor whether gradient descent will *find* the approximating weights. Second, they concern shallow networks; **depth-separation** results (e.g. Telgarsky 2016, Eldan & Shamir 2016) show functions representable by a deep network with polynomially many units that require exponentially many units at shallower depth — the modern justification for depth.

---

## 5. Backpropagation (Rumelhart, Hinton & Williams, 1986)

David E. Rumelhart, Geoffrey E. Hinton and Ronald J. Williams, *"Learning Representations by Back-Propagating Errors"*, **Nature**, 323:533–536, 1986 (DOI: 10.1038/323533a0), popularized the algorithm that made hidden-layer training practical. The paper's stated contribution is a procedure that "repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between the actual output vector of the net and the desired output vector," with the consequence that "hidden units come to represent important features of the task domain" — the ability to *create useful new features* is exactly what distinguishes it from the perceptron convergence procedure.

Historically the method is older: reverse-mode automatic differentiation was described by Seppo Linnainmaa (1970), applied to networks by Paul Werbos in his 1974 Harvard PhD thesis (*Beyond Regression*), and independently derived by Parker (1985) and LeCun (1985).

### Full derivation

Let $L$ be the loss on a single example, with layers indexed $\ell = 1,\dots,L$. Define the **error term** (or "delta") of layer $\ell$ as the gradient of the loss with respect to the pre-activation:

$$\boldsymbol{\delta}^{[\ell]} \;\equiv\; \frac{\partial L}{\partial \mathbf{z}^{[\ell]}} \in \mathbb{R}^{n_\ell}$$

**Output layer.** By the chain rule through $\mathbf{a}^{[L]} = \sigma^{[L]}(\mathbf{z}^{[L]})$:

$$\boldsymbol{\delta}^{[L]} = \nabla_{\mathbf{a}^{[L]}} L \;\odot\; \sigma^{[L]\prime}\!\left(\mathbf{z}^{[L]}\right)$$

where $\odot$ is the Hadamard (elementwise) product.

**Recursive backward pass.** Since $\mathbf{z}^{[\ell+1]} = \mathbf{W}^{[\ell+1]}\sigma^{[\ell]}(\mathbf{z}^{[\ell]}) + \mathbf{b}^{[\ell+1]}$, each $z_k^{[\ell+1]}$ depends on $z_j^{[\ell]}$ through $W^{[\ell+1]}_{kj}\sigma^{[\ell]\prime}(z_j^{[\ell]})$. Summing over all downstream paths:

$$\delta_j^{[\ell]} = \sum_{k} \frac{\partial L}{\partial z_k^{[\ell+1]}}\frac{\partial z_k^{[\ell+1]}}{\partial z_j^{[\ell]}} = \left(\sum_k \delta_k^{[\ell+1]} W_{kj}^{[\ell+1]}\right)\sigma^{[\ell]\prime}\!\left(z_j^{[\ell]}\right)$$

In matrix form:

$$\boxed{\;\boldsymbol{\delta}^{[\ell]} = \left(\mathbf{W}^{[\ell+1]\top}\boldsymbol{\delta}^{[\ell+1]}\right)\odot\sigma^{[\ell]\prime}\!\left(\mathbf{z}^{[\ell]}\right)\;}$$

This is where the name comes from: the error is *propagated backwards* through the transpose of the forward weight matrices.

**Parameter gradients.** Since $z_j^{[\ell]} = \sum_i W_{ji}^{[\ell]}a_i^{[\ell-1]} + b_j^{[\ell]}$, we have $\partial z_j^{[\ell]}/\partial W_{ji}^{[\ell]} = a_i^{[\ell-1]}$ and $\partial z_j^{[\ell]}/\partial b_j^{[\ell]} = 1$, hence

$$\frac{\partial L}{\partial W_{ji}^{[\ell]}} = \delta_j^{[\ell]}\,a_i^{[\ell-1]} \quad\Longleftrightarrow\quad \frac{\partial L}{\partial \mathbf{W}^{[\ell]}} = \boldsymbol{\delta}^{[\ell]}\,\mathbf{a}^{[\ell-1]\top}$$

$$\frac{\partial L}{\partial \mathbf{b}^{[\ell]}} = \boldsymbol{\delta}^{[\ell]}$$

For a mini-batch of size $m$, gradients are averaged: $\partial L/\partial\mathbf{W}^{[\ell]} = \frac{1}{m}\boldsymbol{\Delta}^{[\ell]\top}\mathbf{A}^{[\ell-1]}$ and $\partial L/\partial\mathbf{b}^{[\ell]} = \frac{1}{m}\sum_{k=1}^m \boldsymbol{\delta}^{[\ell](k)}$.

**Complexity.** One backward pass costs the same order as one forward pass, $O(\sum_\ell n_\ell n_{\ell-1})$ — the fundamental efficiency result of reverse-mode automatic differentiation: the full gradient of a scalar with respect to $P$ parameters costs $O(1)$ forward passes, not $O(P)$.

**A useful special case.** With softmax output and categorical cross-entropy, the two Jacobians cancel and the output delta collapses to $\boldsymbol{\delta}^{[L]} = \hat{\mathbf{y}} - \mathbf{y}$. The same holds for sigmoid + binary cross-entropy and for linear output + MSE. This is not a coincidence: it holds for any matched pair of a canonical link function and its exponential-family negative log-likelihood.

**Vanishing/exploding gradients.** The recursion multiplies $\sigma'$ terms at every layer. With sigmoid, $\sigma'(z) \leq 1/4$, so gradients shrink by at least $4^{-L}$ across $L$ layers — the vanishing gradient problem identified by Hochreiter (1991) and Bengio, Simard & Frasconi (1994). This motivates ReLU-family activations, careful initialization, normalization layers, and residual connections.

---

## 6. Activation Functions

| Function | Definition | Derivative | Range |
|---|---|---|---|
| Sigmoid | $\sigma(x) = \dfrac{1}{1+e^{-x}}$ | $\sigma(x)\left(1-\sigma(x)\right)$ | $(0,1)$ |
| Tanh | $\tanh(x) = \dfrac{e^{x}-e^{-x}}{e^{x}+e^{-x}}$ | $1-\tanh^2(x)$ | $(-1,1)$ |
| ReLU | $\max(0,x)$ | $\mathbb{1}[x>0]$ | $[0,\infty)$ |
| Leaky ReLU | $\max(\alpha x, x),\ \alpha{=}0.01$ | $\alpha$ if $x<0$, else $1$ | $(-\infty,\infty)$ |
| PReLU | same, $\alpha$ learned | idem, plus $\partial f/\partial\alpha = \min(0,x)$ | $(-\infty,\infty)$ |
| ELU | $x$ if $x>0$; $\alpha(e^x-1)$ else | $1$ if $x>0$; $\alpha e^x$ else | $(-\alpha,\infty)$ |
| SELU | $\lambda\cdot\text{ELU}_\alpha(x)$ | $\lambda$ if $x>0$; $\lambda\alpha e^x$ else | scaled |
| Softplus | $\ln(1+e^x)$ | $\sigma(x)$ | $(0,\infty)$ |

**Sigmoid.** Historically dominant, now largely confined to gates and binary outputs. Two defects: outputs are not zero-centred (inducing correlated gradient signs across a layer, causing zig-zag descent), and it **saturates** — $\sigma'(x)\to 0$ for $|x|\gtrsim 5$, killing gradient flow. Note $\tanh(x) = 2\sigma(2x)-1$.

**Tanh.** Zero-centred, with max derivative $1$ at the origin. Still saturating, but empirically better-conditioned than sigmoid for hidden layers; still standard in LSTM/GRU cell candidates.

**ReLU.** Used by Fukushima (1969) for visual feature extraction; repopularized by **Nair & Hinton (2010)**, *"Rectified Linear Units Improve Restricted Boltzmann Machines"* (ICML), and **Glorot, Bordes & Bengio (2011)**, *"Deep Sparse Rectifier Neural Networks"* (AISTATS); cemented by AlexNet (Krizhevsky et al., 2012). Advantages: no saturation for $x>0$, gradient exactly $1$ there, trivial to compute, induces sparse activations. Drawback: the **dying ReLU** problem — a unit pushed into the negative regime for all inputs receives zero gradient forever. Non-differentiable at $0$; frameworks conventionally set $f'(0)=0$.

**Leaky ReLU** (Maas, Hannun & Ng, 2013, *"Rectifier Nonlinearities Improve Neural Network Acoustic Models"*, ICML workshop) fixes dying units with a small negative slope. **PReLU** (He, Zhang, Ren & Sun, 2015, *"Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification"*, ICCV) makes $\alpha$ a learned per-channel parameter at negligible cost; the same paper introduced **He/Kaiming initialization**, $\mathrm{Var}(W) = 2/n_{\text{in}}$, which corrects Xavier initialization for the fact that ReLU zeroes half the activations.

**ELU** (Clevert, Unterthiner & Hochreiter, 2015, *"Fast and Accurate Deep Network Learning by Exponential Linear Units"*, ICLR 2016) saturates smoothly to $-\alpha$ for large negative inputs, pushing mean activations toward zero (a batch-norm-like effect) while remaining robust to noise.

**SELU** (Klambauer, Unterthiner, Mayr & Hochreiter, 2017, *"Self-Normalizing Neural Networks"*, NIPS) fixes

$$\lambda \approx 1.0507009873554804934193349852946, \qquad \alpha \approx 1.6732632423543772848170429916717$$

$$\text{SELU}(x) = \lambda\begin{cases} x & x > 0\\ \alpha(e^x - 1) & x \leq 0\end{cases}$$

These constants are derived so that, under LeCun-normal initialization ($\mathrm{Var}(W)=1/n_{\text{in}}$) and with the "alpha-dropout" variant, activation mean and variance have an **attracting fixed point at $(0,1)$** — the network self-normalizes without batch norm. The guarantee is fragile: it requires the specific initialization, fully-connected architecture, and alpha-dropout.

**GELU** (Hendrycks & Gimpel, 2016, *"Gaussian Error Linear Units (GELUs)"*, arXiv:1606.08415) weights the input by the probability that a standard Gaussian falls below it:

$$\text{GELU}(x) = x\,\Phi(x) = \frac{x}{2}\left[1 + \operatorname{erf}\!\left(\frac{x}{\sqrt{2}}\right)\right], \qquad \frac{d}{dx}\text{GELU}(x) = \Phi(x) + x\,\phi(x)$$

where $\phi$ is the standard normal density. The widely-used tanh approximation (used in BERT and GPT-2) is

$$\text{GELU}(x) \approx 0.5\,x\left(1 + \tanh\!\left[\sqrt{\tfrac{2}{\pi}}\left(x + 0.044715\,x^3\right)\right]\right)$$

GELU can be read as a *deterministic* version of stochastic-regularizer gating: instead of dropping a unit with probability $1-\Phi(x)$, it scales by the expected mask. It is the default in most Transformers.

**Swish / SiLU** (Ramachandran, Zoph & Le, 2017, *"Searching for Activation Functions"*, discovered by automated search; the $\beta=1$ case, SiLU, appears earlier in Hendrycks & Gimpel 2016 and Elfwing et al. 2017):

$$\text{Swish}_\beta(x) = x\,\sigma(\beta x), \qquad \text{SiLU}(x) = \frac{x}{1+e^{-x}}$$
$$\text{SiLU}'(x) = \sigma(x)\left(1 + x\left(1 - \sigma(x)\right)\right) = \frac{1 + e^{-x} + x e^{-x}}{(1+e^{-x})^2}$$

Smooth, non-monotonic (a small negative dip near $x\approx-1.28$), unbounded above, bounded below; $\beta$ may be learned, with $\beta\to 0$ recovering a linear unit and $\beta\to\infty$ recovering ReLU. **SwiGLU** — a gated variant $\text{Swish}(xW)\odot(xV)$ (Shazeer, 2020) — is now standard in LLM feed-forward blocks.

**Mish** (Misra, 2019, *"Mish: A Self Regularized Non-Monotonic Activation Function"*, BMVC 2020):

$$\text{Mish}(x) = x\tanh\left(\text{softplus}(x)\right) = x\tanh\left(\ln(1+e^x)\right)$$

Similar in shape to Swish, with a smoother profile; adopted in several YOLO variants.

**Softmax** (Bridle, 1990) converts a logit vector to a probability simplex:

$$\text{softmax}(\mathbf{z})_i = \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}, \qquad \frac{\partial\,\text{softmax}_i}{\partial z_j} = \text{softmax}_i\left(\delta_{ij} - \text{softmax}_j\right)$$

with $\delta_{ij}$ the Kronecker delta. It is shift-invariant, so implementations subtract $\max_j z_j$ before exponentiating for numerical stability. With a temperature $T$, $\text{softmax}(\mathbf{z}/T)$ interpolates between argmax ($T\to0$) and uniform ($T\to\infty$).

**Softplus** $= \ln(1+e^x)$ is the smooth ReLU; its derivative is exactly the sigmoid. Implemented stably as $\max(x,0) + \ln(1+e^{-|x|})$.

---

## 7. Loss Functions

**Mean Squared Error (L2).** For regression, with $n$ examples:

$$L_{\text{MSE}} = \frac{1}{n}\sum_{i=1}^{n}\left(y_i - \hat{y}_i\right)^2, \qquad \frac{\partial L}{\partial \hat{y}_i} = -\frac{2}{n}(y_i - \hat{y}_i)$$

Corresponds to Gaussian-noise maximum likelihood; strongly penalizes outliers (quadratic growth).

**Mean Absolute Error (L1).**

$$L_{\text{MAE}} = \frac{1}{n}\sum_{i=1}^{n}\left|y_i - \hat{y}_i\right|, \qquad \frac{\partial L}{\partial\hat{y}_i} = -\frac{1}{n}\operatorname{sign}(y_i - \hat{y}_i)$$

Robust to outliers (Laplace-noise MLE; the minimizer is the conditional median rather than the mean), but non-differentiable at zero and with constant gradient magnitude, which impedes fine convergence.

**Huber loss** (Peter J. Huber, *"Robust Estimation of a Location Parameter"*, **Annals of Mathematical Statistics**, 35(1):73–101, 1964), the standard compromise. With $r = y - \hat{y}$ and threshold $\delta$:

$$L_\delta(r) = \begin{cases} \frac{1}{2}r^2 & |r| \leq \delta\\[4pt] \delta\left(|r| - \frac{1}{2}\delta\right) & |r| > \delta\end{cases}, \qquad \frac{dL_\delta}{dr} = \begin{cases} r & |r|\leq\delta\\ \delta\operatorname{sign}(r) & |r| > \delta\end{cases}$$

Quadratic near zero (fast convergence on small residuals), linear in the tails (bounded gradient, outlier-resistant), and $C^1$ everywhere. The *smooth L1* loss used in object detection is $L_\delta/\delta$ with $\delta=1$.

**Binary cross-entropy (log loss).** For $y \in \{0,1\}$ and $\hat{y} = \sigma(z) \in (0,1)$:

$$L_{\text{BCE}} = -\frac{1}{n}\sum_{i=1}^{n}\left[y_i \ln \hat{y}_i + (1-y_i)\ln(1-\hat{y}_i)\right]$$

$$\frac{\partial L}{\partial \hat{y}} = \frac{\hat{y}-y}{\hat{y}(1-\hat{y})}, \qquad \frac{\partial L}{\partial z} = \hat{y} - y$$

The clean gradient w.r.t. the logit is why BCE is paired with sigmoid: the $\sigma'$ factor cancels the denominator, avoiding the learning slowdown that MSE-plus-sigmoid suffers when the output saturates on a wrong prediction.

**Categorical cross-entropy.** With one-hot $\mathbf{y}$ and $\hat{\mathbf{y}} = \text{softmax}(\mathbf{z})$ over $K$ classes:

$$L_{\text{CE}} = -\sum_{k=1}^{K} y_k \ln\hat{y}_k, \qquad \nabla_{\mathbf{z}} L_{\text{CE}} = \hat{\mathbf{y}} - \mathbf{y}$$

Equivalently the KL divergence $D_{\text{KL}}(\mathbf{y}\,\|\,\hat{\mathbf{y}})$ up to the constant entropy of $\mathbf{y}$, and equivalently the negative log-likelihood of a categorical model.

**Hinge loss** (the SVM loss; Cortes & Vapnik, 1995). For $y \in \{-1,+1\}$ and raw score $\hat{y}$:

$$L_{\text{hinge}} = \max\left(0,\; 1 - y\hat{y}\right), \qquad \frac{\partial L}{\partial\hat{y}} = \begin{cases}-y & y\hat{y} < 1\\ 0 & \text{otherwise}\end{cases}$$

Zero loss once the example is correctly classified *with margin at least 1* — unlike cross-entropy, which never reaches exactly zero and keeps pushing confident predictions. The squared hinge $\max(0,1-y\hat{y})^2$ is differentiable everywhere. The multiclass version (Crammer–Singer / Weston–Watkins) is $\sum_{k \neq y}\max(0, \hat{y}_k - \hat{y}_y + 1)$.

---

## 8. Optimizers

Throughout, $\theta_t$ are parameters at step $t$, $g_t = \nabla_\theta L(\theta_t)$ the (mini-batch) gradient, and $\eta$ the learning rate. Operations are elementwise.

**SGD** (Robbins & Monro, 1951, *"A Stochastic Approximation Method"*):

$$\theta_{t+1} = \theta_t - \eta\, g_t$$

Robbins–Monro convergence requires $\sum_t \eta_t = \infty$ and $\sum_t \eta_t^2 < \infty$.

**Momentum / heavy ball** (Boris Polyak, *"Some Methods of Speeding Up the Convergence of Iteration Methods"*, **USSR Comp. Math. and Math. Physics**, 4(5):1–17, 1964):

$$v_{t} = \beta v_{t-1} + g_t, \qquad \theta_{t+1} = \theta_t - \eta\, v_t$$

(equivalently $v_t = \beta v_{t-1} + (1-\beta)g_t$ in the EMA convention, $\beta$ typically $0.9$). The velocity accumulates consistent gradient directions and cancels oscillatory ones; effective step size in a consistent direction is amplified by $1/(1-\beta)$.

**Nesterov Accelerated Gradient** (Yurii Nesterov, 1983, *"A method for solving the convex programming problem with convergence rate $O(1/k^2)$"*). The gradient is evaluated at the *look-ahead* point:

$$v_t = \beta v_{t-1} + \nabla_\theta L\left(\theta_t - \eta\beta v_{t-1}\right), \qquad \theta_{t+1} = \theta_t - \eta v_t$$

The essential difference from Polyak: momentum is applied *before* the gradient is measured, letting the update "see" where it is heading and correct in advance. For smooth convex objectives NAG attains the optimal $O(1/k^2)$ rate versus $O(1/k)$ for plain gradient descent. Deep learning frameworks implement the Sutskever et al. (2013) reparameterization.

**AdaGrad** (Duchi, Hazan & Singer, *"Adaptive Subgradient Methods for Online Learning and Stochastic Optimization"*, **JMLR** 12:2121–2159, 2011):

$$G_t = G_{t-1} + g_t^2, \qquad \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{G_t} + \epsilon}\odot g_t$$

Per-coordinate learning rates inversely proportional to the accumulated gradient magnitude: rare (sparse) features get large steps, frequent ones small steps. Excellent for sparse, convex problems (NLP with bag-of-words); its flaw in deep learning is that $G_t$ grows monotonically, so the effective learning rate decays to zero and learning stalls.

**RMSProp** (Tieleman & Hinton, Coursera *Neural Networks for Machine Learning*, Lecture 6.5, 2012 — never formally published) replaces the sum with an exponential moving average:

$$E[g^2]_t = \rho\,E[g^2]_{t-1} + (1-\rho)\,g_t^2, \qquad \theta_{t+1} = \theta_t - \frac{\eta}{\sqrt{E[g^2]_t}+\epsilon}\odot g_t$$

with $\rho \approx 0.9$. Old gradients are forgotten, so the effective learning rate no longer collapses. AdaDelta (Zeiler, 2012) is a closely related variant that additionally eliminates $\eta$ via a second accumulator of parameter updates.

**Adam** (Diederik P. Kingma & Jimmy Ba, *"Adam: A Method for Stochastic Optimization"*, ICLR 2015, arXiv:1412.6980) — "adaptive moment estimation," combining momentum (first moment) and RMSProp (second moment) with bias correction:

$$m_t = \beta_1 m_{t-1} + (1-\beta_1)\,g_t$$
$$v_t = \beta_2 v_{t-1} + (1-\beta_2)\,g_t^2$$
$$\hat{m}_t = \frac{m_t}{1-\beta_1^{\,t}}, \qquad \hat{v}_t = \frac{v_t}{1-\beta_2^{\,t}}$$
$$\theta_{t+1} = \theta_t - \eta\,\frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}$$

Defaults: $\eta = 0.001$, $\beta_1 = 0.9$, $\beta_2 = 0.999$, $\epsilon = 10^{-8}$. The bias correction matters because $m_0 = v_0 = 0$ biases the early EMAs toward zero; since $\mathbb{E}[m_t] \approx (1-\beta_1^t)\mathbb{E}[g_t]$, dividing by $(1-\beta_1^t)$ removes the bias. Without it, the first steps would be drastically too small — especially for $v$ with $\beta_2 = 0.999$. The ratio $\hat{m}/\sqrt{\hat{v}}$ is a signal-to-noise estimate, and the effective step is bounded by roughly $\eta$ regardless of gradient scale, making Adam invariant to gradient rescaling. Reddi et al. (2018) showed a flaw in the original convergence proof and proposed AMSGrad (using $\max$ of past $v_t$).

**AdamW** (Ilya Loshchilov & Frank Hutter, *"Decoupled Weight Decay Regularization"*, ICLR 2019, arXiv:1711.05101). The key observation: L2 regularization and weight decay are equivalent for plain SGD but **not** for adaptive methods. Adding $\lambda\theta$ to the gradient makes the decay be divided by $\sqrt{\hat v_t}$, so parameters with large historical gradients are decayed *less* — the opposite of the intent. AdamW decouples the two:

$$\theta_{t+1} = \theta_t - \eta_t\left(\frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon} + \lambda\,\theta_t\right)$$

where $\eta_t$ is a schedule multiplier applied to *both* terms simultaneously (so that the tuned $\lambda$ remains valid under learning-rate schedules). This substantially improves Adam's generalization and decouples the optimal $\lambda$ from the optimal $\eta$; AdamW is the default optimizer for essentially all modern Transformers.

---

## 9. Regularization

**L2 regularization / weight decay / ridge.**

$$L_{\text{total}} = L_{\text{data}} + \frac{\lambda}{2}\sum_{\ell}\left\|\mathbf{W}^{[\ell]}\right\|_F^2, \qquad \frac{\partial L_{\text{total}}}{\partial \mathbf{W}} = \frac{\partial L_{\text{data}}}{\partial\mathbf{W}} + \lambda\mathbf{W}$$

The SGD update becomes $\mathbf{W} \leftarrow (1-\eta\lambda)\mathbf{W} - \eta\,\partial L_{\text{data}}/\partial\mathbf{W}$ — an explicit multiplicative shrinkage, hence "weight decay." From a Bayesian standpoint this is a zero-mean Gaussian prior on the weights (MAP estimation). Biases are conventionally excluded.

**L1 regularization / lasso.**

$$L_{\text{total}} = L_{\text{data}} + \lambda\sum_{\ell}\left\|\mathbf{W}^{[\ell]}\right\|_1, \qquad \frac{\partial}{\partial\mathbf{W}} = \frac{\partial L_{\text{data}}}{\partial\mathbf{W}} + \lambda\operatorname{sign}(\mathbf{W})$$

The constant-magnitude gradient drives small weights exactly to zero, producing **sparse** solutions (feature selection). Corresponds to a Laplace prior. **Elastic net** combines both: $\lambda_1\|\mathbf{W}\|_1 + \frac{\lambda_2}{2}\|\mathbf{W}\|_2^2$.

**Dropout** (Srivastava, Hinton, Krizhevsky, Sutskever & Salakhutdinov, *"Dropout: A Simple Way to Prevent Neural Networks from Overfitting"*, **JMLR** 15:1929–1958, 2014; building on Hinton et al., 2012). During training, each unit is deleted independently with probability $1-p$:

$$r_j^{[\ell]} \sim \text{Bernoulli}(p), \qquad \tilde{\mathbf{a}}^{[\ell]} = \mathbf{r}^{[\ell]}\odot\mathbf{a}^{[\ell]}, \qquad \mathbf{z}^{[\ell+1]} = \mathbf{W}^{[\ell+1]}\tilde{\mathbf{a}}^{[\ell]} + \mathbf{b}^{[\ell+1]}$$

At test time all units are kept and weights are scaled: $\mathbf{W}_{\text{test}} = p\,\mathbf{W}$, so that the expected input to each unit matches training. The practical implementation is **inverted dropout**, which divides by $p$ during training instead:

$$\tilde{\mathbf{a}}^{[\ell]} = \frac{1}{p}\,\mathbf{r}^{[\ell]}\odot\mathbf{a}^{[\ell]} \quad\text{(train)}, \qquad \tilde{\mathbf{a}}^{[\ell]} = \mathbf{a}^{[\ell]}\quad\text{(test)}$$

leaving inference untouched. Typical $p = 0.5$ for hidden layers, $0.8$ for inputs. The mechanism prevents **co-adaptation** — no unit can rely on the presence of any particular other unit — and can be interpreted as training an exponential ensemble of $2^N$ thinned subnetworks with shared weights, approximately averaged at test time by the geometric mean.

**Batch Normalization** (Sergey Ioffe & Christian Szegedy, *"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift"*, ICML 2015, arXiv:1502.03167). For each feature, over a mini-batch $\mathcal{B} = \{x_1,\dots,x_m\}$:

$$\mu_{\mathcal{B}} = \frac{1}{m}\sum_{i=1}^{m} x_i \qquad\text{(mini-batch mean)}$$
$$\sigma^2_{\mathcal{B}} = \frac{1}{m}\sum_{i=1}^{m}\left(x_i - \mu_{\mathcal{B}}\right)^2 \qquad\text{(mini-batch variance)}$$
$$\hat{x}_i = \frac{x_i - \mu_{\mathcal{B}}}{\sqrt{\sigma^2_{\mathcal{B}}+\epsilon}} \qquad\text{(normalize)}$$
$$y_i = \gamma\hat{x}_i + \beta \equiv \text{BN}_{\gamma,\beta}(x_i) \qquad\text{(scale and shift)}$$

The learned $\gamma,\beta$ restore representational capacity — setting $\gamma = \sqrt{\sigma^2_\mathcal{B}+\epsilon}$, $\beta=\mu_\mathcal{B}$ recovers the identity, so BN never *removes* expressiveness. Backpropagation must flow through $\mu_\mathcal{B}$ and $\sigma^2_\mathcal{B}$ as well:

$$\frac{\partial L}{\partial \hat x_i} = \frac{\partial L}{\partial y_i}\gamma, \quad \frac{\partial L}{\partial\sigma^2_\mathcal{B}} = \sum_i \frac{\partial L}{\partial \hat x_i}(x_i-\mu_\mathcal{B})\cdot\frac{-1}{2}(\sigma^2_\mathcal{B}+\epsilon)^{-3/2}$$
$$\frac{\partial L}{\partial\mu_\mathcal{B}} = \sum_i\frac{\partial L}{\partial\hat x_i}\cdot\frac{-1}{\sqrt{\sigma^2_\mathcal{B}+\epsilon}} + \frac{\partial L}{\partial\sigma^2_\mathcal{B}}\cdot\frac{-2\sum_i(x_i-\mu_\mathcal{B})}{m}$$
$$\frac{\partial L}{\partial x_i} = \frac{\partial L}{\partial\hat x_i}\frac{1}{\sqrt{\sigma^2_\mathcal{B}+\epsilon}} + \frac{\partial L}{\partial\sigma^2_\mathcal{B}}\frac{2(x_i-\mu_\mathcal{B})}{m} + \frac{\partial L}{\partial\mu_\mathcal{B}}\frac{1}{m}$$
$$\frac{\partial L}{\partial\gamma} = \sum_i\frac{\partial L}{\partial y_i}\hat x_i, \qquad \frac{\partial L}{\partial\beta} = \sum_i\frac{\partial L}{\partial y_i}$$

At **inference**, batch statistics are unavailable, so running (EMA) estimates collected during training are used: $\mathbb{E}[x] \leftarrow \text{EMA}(\mu_\mathcal{B})$, $\mathrm{Var}[x] \leftarrow \frac{m}{m-1}\text{EMA}(\sigma^2_\mathcal{B})$, and the whole transform becomes a fixed affine map foldable into the preceding convolution. BN permits much larger learning rates, reduces sensitivity to initialization, and has a mild regularizing effect from mini-batch noise. Its explanation via "internal covariate shift" was later challenged — Santurkar et al. (2018) argue the real benefit is a smoother loss landscape. Its weaknesses: dependence on batch size (poor for $m \lesssim 8$) and awkwardness in recurrent and online settings.

**Layer Normalization** (Jimmy Lei Ba, Jamie Ryan Kiros & Geoffrey E. Hinton, *"Layer Normalization"*, arXiv:1607.06450, 2016) transposes the computation: statistics are taken over the **features of a single example**, not over the batch. For $\mathbf{x}\in\mathbb{R}^d$:

$$\mu = \frac{1}{d}\sum_{i=1}^{d}x_i, \qquad \sigma^2 = \frac{1}{d}\sum_{i=1}^{d}(x_i-\mu)^2$$
$$\text{LN}(\mathbf{x}) = \boldsymbol{\gamma}\odot\frac{\mathbf{x}-\mu}{\sqrt{\sigma^2+\epsilon}} + \boldsymbol{\beta}$$

Because it is batch-independent, LN behaves identically at train and test time, works with batch size 1, and handles variable-length sequences — which is why it, not BN, is used in RNNs and Transformers. Related variants: **GroupNorm** (Wu & He, 2018) normalizes over feature groups; **RMSNorm** (Zhang & Sennrich, 2019) drops the mean-centring, $\text{RMSNorm}(\mathbf{x}) = \boldsymbol{\gamma}\odot\mathbf{x}/\sqrt{\frac{1}{d}\sum_i x_i^2 + \epsilon}$, and is now standard in LLaMA-class models.

Other standard regularizers: **early stopping** (implicit L2 for quadratic objectives), **data augmentation**, **label smoothing** ($y_k \leftarrow (1-\varepsilon)y_k + \varepsilon/K$), and **max-norm constraints** $\|\mathbf{w}_j\|_2 \leq c$, which the dropout paper recommends alongside dropout.

---

## 10. Radial Basis Function (RBF) Networks

RBF networks were introduced by **D. S. Broomhead & D. Lowe**, *"Multivariable Functional Interpolation and Adaptive Networks"*, **Complex Systems**, 2:321–355, 1988 (RSRE Memorandum 4148), with independent contemporaneous work by Lee & Kil (1988) and Niranjan & Fallside (1988), and refined by **J. Moody & C. Darken**, *"Fast Learning in Networks of Locally-Tuned Processing Units"*, **Neural Computation**, 1(2):281–294, 1989. **Michael Powell**'s earlier work on radial basis function interpolation (1985–1987) supplied the mathematical basis, and **Park & Sandberg** (1991) proved universal approximation for RBF networks.

### Architecture

Strictly three layers, with only one hidden layer:

1. **Input layer** — $n$ nodes, pure fan-out.
2. **Hidden layer** — $J$ *locally tuned* units, each with a centre $\boldsymbol{\mu}_j \in \mathbb{R}^n$ and a width $\sigma_j$, computing a radial function of the distance between input and centre.
3. **Output layer** — $L$ **linear** units.

The output is a weighted sum of basis functions:

$$f_l(\mathbf{x}) = \sum_{j=1}^{J} w_{lj}\,\varphi\!\left(\left\|\mathbf{x} - \boldsymbol{\mu}_j\right\|\right) + b_l$$

The most common kernel is the **Gaussian**:

$$\varphi_j(\mathbf{x}) = \exp\!\left(-\frac{\left\|\mathbf{x}-\boldsymbol{\mu}_j\right\|^2}{2\sigma_j^2}\right)$$

often written $\exp(-\beta_j\|\mathbf{x}-\boldsymbol{\mu}_j\|^2)$ with $\beta_j = 1/(2\sigma_j^2)$. Anisotropic units generalize to a Mahalanobis form $\exp\!\left(-\tfrac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_j)^\top\boldsymbol{\Sigma}_j^{-1}(\mathbf{x}-\boldsymbol{\mu}_j)\right)$. Other classical kernels: multiquadric $\sqrt{r^2+c^2}$, inverse multiquadric $1/\sqrt{r^2+c^2}$, and thin-plate spline $r^2\ln r$.

### The essential contrast with the MLP

An MLP hidden unit computes an **inner product** $\mathbf{w}^\top\mathbf{x}$ and responds along a hyperplane — a *global*, distributed representation. An RBF unit computes a **distance** $\|\mathbf{x}-\boldsymbol{\mu}\|$ and responds only in a localized neighbourhood — a *local*, hypersphere-shaped receptive field. Consequences: RBF networks train much faster (the output layer is linear), interpolate cleanly, and degrade gracefully outside the data (activations vanish, so the output tends to the bias); but they suffer the curse of dimensionality, since covering a high-dimensional space with local bumps requires exponentially many centres.

### Training

The standard procedure is **two-stage / hybrid**:

1. **Unsupervised** placement of centres $\boldsymbol{\mu}_j$: random subsampling of the training data or a coarse lattice (Broomhead & Lowe, 1988), or **$k$-means clustering** (Moody & Darken, 1989). Widths are then set heuristically, e.g. $\sigma_j = d_{\max}/\sqrt{2J}$ with $d_{\max}$ the maximum inter-centre distance, or by the $p$-nearest-neighbour rule $\sigma_j = \left(\frac{1}{p}\sum_{k=1}^{p}\|\boldsymbol{\mu}_j-\boldsymbol{\mu}_k\|^2\right)^{1/2}$ (Moody & Darken).

2. **Supervised** solution of the output weights. Since the model is linear in $\mathbf{W}$, the least-squares solution is closed-form. With the design (interpolation) matrix $\Phi_{ij} = \varphi_j(\mathbf{x}_i)$:

$$\mathbf{W} = \left(\boldsymbol{\Phi}^\top\boldsymbol{\Phi} + \lambda\mathbf{I}\right)^{-1}\boldsymbol{\Phi}^\top\mathbf{Y} = \boldsymbol{\Phi}^{+}\mathbf{Y}$$

using the Moore–Penrose pseudoinverse (Broomhead & Lowe's approach), with $\lambda$ a ridge term. In the *exact interpolation* case $J = m$ (one centre per data point), Michelli's theorem guarantees $\boldsymbol{\Phi}$ is nonsingular for Gaussian kernels and distinct points, giving $\mathbf{W}=\boldsymbol{\Phi}^{-1}\mathbf{Y}$ — but this overfits, hence the use of $J \ll m$.

All parameters $\{w_{lj}, \boldsymbol{\mu}_j, \sigma_j\}$ can alternatively be trained jointly by gradient descent, e.g. $\partial L/\partial\boldsymbol{\mu}_j = \sum_l \delta_l w_{lj}\varphi_j(\mathbf{x})\frac{\mathbf{x}-\boldsymbol{\mu}_j}{\sigma_j^2}$, at the cost of the convexity that makes the hybrid scheme attractive.

RBF networks are close relatives of kernel methods (an SVM with a Gaussian kernel is an RBF network whose centres are the support vectors and whose weights come from the dual QP), of Gaussian mixture models, and of normalized-RBF / Nadaraya–Watson regression. They remain in use for function interpolation, meshless PDE solvers, time-series prediction, and control.

---

## Chronological Summary of Founding Papers

| Year | Authors | Contribution |
|---|---|---|
| 1943 | McCulloch & Pitts | *A Logical Calculus of the Ideas Immanent in Nervous Activity* — threshold neuron |
| 1949 | Hebb | *The Organization of Behavior* — Hebbian learning |
| 1958 | Rosenblatt | *The Perceptron: A Probabilistic Model…* — first learning rule |
| 1960 | Widrow & Hoff | *Adaptive Switching Circuits* — ADALINE, LMS/delta rule |
| 1962 | Novikoff | *On Convergence Proofs for Perceptrons* — mistake bound |
| 1964 | Polyak / Huber | Heavy-ball momentum / robust loss |
| 1969 | Minsky & Papert | *Perceptrons* — XOR and the limits of linear separability |
| 1974 | Werbos | PhD thesis — backpropagation (reverse-mode AD) |
| 1983 | Nesterov | Accelerated gradient, $O(1/k^2)$ |
| 1986 | Rumelhart, Hinton & Williams | *Learning Representations by Back-Propagating Errors*, **Nature** |
| 1988–89 | Broomhead & Lowe; Moody & Darken | RBF networks |
| 1989 | Cybenko; Hornik, Stinchcombe & White | Universal approximation |
| 1993 | Leshno, Lin, Pinkus & Schocken | Universal approximation iff non-polynomial |
| 2010–11 | Nair & Hinton; Glorot, Bordes & Bengio | ReLU for deep networks |
| 2011 | Duchi, Hazan & Singer | AdaGrad |
| 2014 | Kingma & Ba; Srivastava et al. | Adam; Dropout |
| 2015 | Ioffe & Szegedy; He et al.; Clevert et al. | BatchNorm; PReLU + He init; ELU |
| 2016 | Ba, Kiros & Hinton; Hendrycks & Gimpel | LayerNorm; GELU |
| 2017 | Klambauer et al.; Ramachandran et al. | SELU; Swish |
| 2019 | Loshchilov & Hutter; Misra | AdamW; Mish |

Sources: [A Logical Calculus (Wikipedia)](https://en.wikipedia.org/wiki/A_Logical_Calculus_of_the_Ideas_Immanent_in_Nervous_Activity), [Perceptrons (book)](https://en.wikipedia.org/wiki/Perceptrons_(book)), [Widrow & Hoff / ADALINE](https://en.wikipedia.org/wiki/Bernard_Widrow), [Hornik, Stinchcombe & White 1989](https://www.cs.cmu.edu/~epxing/Class/10715/reading/Kornick_et_al.pdf), [Hornik 1991](https://web.njit.edu/~usman/courses/cs677/hornik-nn-1991.pdf), [Note on Cybenko's UAT](https://arxiv.org/html/2508.18893v1), [Rumelhart, Hinton & Williams 1986 (Nature)](https://www.nature.com/articles/323533a0), [Activation function reference](https://en.wikipedia.org/wiki/Activation_function), [Self-Normalizing Neural Networks (SELU)](https://arxiv.org/pdf/1706.02515), [Delving Deep into Rectifiers (PReLU)](https://arxiv.org/pdf/1502.01852), [Rectifier (ReLU) history](https://en.wikipedia.org/wiki/ReLU), [Swish function](https://en.wikipedia.org/wiki/Swish_function), [Batch Normalization (Ioffe & Szegedy)](https://arxiv.org/abs/1502.03167), [Dropout (Srivastava et al. 2014)](https://nitishsrivastava.github.io/publication/2014-01-01), [Layer Normalization (Ba, Kiros & Hinton)](https://www.semanticscholar.org/paper/Layer-Normalization-Ba-Kiros/97fb4e3d45bb098e27e0071448b6152217bd35a5), [Decoupled Weight Decay Regularization (AdamW)](https://arxiv.org/pdf/1711.05101), [AdaGrad (Cornell Optimization Wiki)](https://optimization.cbe.cornell.edu/index.php?title=AdaGrad), [Novikoff perceptron convergence proof](https://apps.dtic.mil/sti/tr/pdf/AD0298258.pdf), [Leshno et al. 1993](https://www.sciencedirect.com/science/article/abs/pii/S0893608005801315), [RBF networks (MIT book, ch. 6)](https://neuron.eng.wayne.edu/tarek/MITbook/chap6/6_1.html), [Broomhead & Lowe 1988](https://www.sciepub.com/reference/93721)
# Convolutional Neural Networks (CNNs): History, Mathematics, and Architectures

## 1. The Neocognitron (Fukushima, 1980): The Precursor

The direct ancestor of modern CNNs is the **Neocognitron**, proposed by Kunihiko Fukushima (Fukushima, K., 1980, *"Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position"*, Biological Cybernetics, 36, 193–202). Invented in 1979 at NHK Science & Technical Research Laboratories, it was directly inspired by the neurophysiological work of **Hubel and Wiesel (1959, 1962)** on the cat's visual cortex, which identified *simple cells* (responding to oriented edges at specific positions) and *complex cells* (responding to the same features with positional tolerance).

The Neocognitron alternates two layer types in a hierarchy:

- **S-cells (simple)**: extract local features via receptive fields with shared, learnable weights — the conceptual ancestor of the convolutional layer;
- **C-cells (complex)**: pool responses of S-cells over a local neighborhood to gain invariance to small shifts — the ancestor of the pooling layer.

The network was trained by unsupervised, competitive self-organization ("learning without a teacher") and achieved shift-invariant pattern recognition. It lacked two ingredients of modern CNNs: end-to-end supervised training by **backpropagation** (introduced to CNNs by LeCun et al., 1989) and large-scale data/compute. Nevertheless, its S/C alternation is exactly the convolution/pooling alternation of LeNet and its successors.

## 2. The Convolution Operation

### 2.1 Discrete 2D equation

For an input image (or feature map) $I$ and a kernel (filter) $K$ of size $k_h \times k_w$, the 2D discrete convolution is:

$$
S(i, j) = (I * K)(i, j) = \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} I(i - m,\; j - n)\, K(m, n)
$$

In practice, deep learning frameworks implement **cross-correlation** (no kernel flip), which is equivalent up to a re-parameterization of learned weights:

$$
S(i, j) = (I \star K)(i, j) = \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} I(i + m,\; j + n)\, K(m, n)
$$

For a multi-channel input $x \in \mathbb{R}^{C_{in} \times H \times W}$ producing output channel $c_{out}$, with stride $s$ and bias $b$:

$$
y_{c_{out}}(i, j) = b_{c_{out}} + \sum_{c=1}^{C_{in}} \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} W_{c_{out}, c}(m, n)\; x_c(s\,i + m,\; s\,j + n)
$$

Key properties: **local connectivity** (each output depends only on a small receptive field), **weight sharing** (the same kernel slides over the whole image, giving translation *equivariance* and drastically reducing parameters), and **hierarchical composition** (stacked layers grow the receptive field, building edge → texture → part → object features).

### 2.2 Stride, padding, dilation

- **Stride** $s$: the step of the sliding window; $s > 1$ downsamples the output.
- **Padding** $p$: zeros (typically) added around the border. "Valid" = no padding; "same" padding ($p = \lfloor k/2 \rfloor$ for odd $k$, $s=1$) preserves spatial size.
- **Dilation** $d$: inserts $d - 1$ gaps between kernel taps (à trous convolution; Yu & Koltun, 2016, *"Multi-Scale Context Aggregation by Dilated Convolutions"*, ICLR). The effective kernel size becomes

$$
k_{\text{eff}} = d\,(k - 1) + 1
$$

which enlarges the receptive field exponentially when stacked, without adding parameters — central to segmentation networks such as DeepLab.

### 2.3 Output size formula

For input size $W_{in}$, kernel $k$, padding $p$, stride $s$, dilation $d$:

$$
W_{out} = \left\lfloor \frac{W_{in} + 2p - d\,(k - 1) - 1}{s} \right\rfloor + 1
$$

which reduces to the classic formula when $d = 1$:

$$
W_{out} = \left\lfloor \frac{W_{in} - k + 2p}{s} \right\rfloor + 1
$$

The parameter count of a layer is $C_{out} \times (C_{in} \times k_h \times k_w + 1)$, independent of the spatial resolution — the essential advantage over fully connected layers.

## 3. Pooling

Pooling summarizes local neighborhoods, providing small translation invariance and downsampling. For a pooling window $\mathcal{R}_{ij}$ of size $k \times k$ with stride $s$:

**Max pooling:**

$$
y_{c}(i, j) = \max_{(m, n) \in \mathcal{R}_{ij}} x_{c}(m, n)
$$

**Average pooling:**

$$
y_{c}(i, j) = \frac{1}{|\mathcal{R}_{ij}|} \sum_{(m, n) \in \mathcal{R}_{ij}} x_{c}(m, n)
$$

**Global average pooling (GAP)** (Lin, Chen & Yan, 2014, *"Network in Network"*, ICLR) collapses each channel's entire $H \times W$ map to one scalar:

$$
y_c = \frac{1}{H W} \sum_{i=1}^{H} \sum_{j=1}^{W} x_c(i, j)
$$

GAP replaces the huge fully connected layers of AlexNet/VGG (which held most of their parameters), acts as a structural regularizer, and makes the network accept variable input sizes; it is standard from GoogLeNet and ResNet onward.

## 4. LeNet-5 (LeCun et al., 1998)

**LeNet-5** (LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998, *"Gradient-Based Learning Applied to Document Recognition"*, Proceedings of the IEEE, 86(11), 2278–2324) was the first widely deployed CNN, reading millions of bank checks. It takes $32 \times 32$ grayscale inputs and stacks 7 trainable layers:

| Layer | Type | Output | Details |
|-------|------|--------|---------|
| C1 | Convolution $5\times5$ | $6 \times 28 \times 28$ | 156 parameters |
| S2 | Subsampling (avg pool $2\times2$) | $6 \times 14 \times 14$ | trainable coefficient + bias, sigmoid |
| C3 | Convolution $5\times5$ | $16 \times 10 \times 10$ | **sparse connectivity table** between S2 and C3 maps (breaks symmetry, saves computation) |
| S4 | Subsampling $2\times2$ | $16 \times 5 \times 5$ | |
| C5 | Convolution $5\times5$ | $120 \times 1 \times 1$ | effectively fully connected |
| F6 | Fully connected | 84 units | tanh activation |
| Output | Euclidean RBF units | 10 classes | |

Total: ~60k parameters. LeNet-5 established the canonical pattern *[conv → pool] × N → FC → output* and demonstrated end-to-end gradient-based training on raw pixels (MNIST error ~0.95%, ~0.8% with augmentation).

## 5. AlexNet (2012): The Deep Learning Detonator

**AlexNet** (Krizhevsky, A., Sutskever, I., Hinton, G. E., 2012, *"ImageNet Classification with Deep Convolutional Neural Networks"*, NeurIPS) won ILSVRC-2012 with **15.3% top-5 error** versus 26.2% for the runner-up — the gap that ignited the deep learning revolution.

Architecture: 8 learned layers — 5 convolutional (kernels $11\times11$ stride 4, then $5\times5$, then three $3\times3$) + 3 fully connected (4096, 4096, 1000), ~60M parameters, trained on 1.2M ImageNet images.

Key innovations:

- **ReLU** activation, $f(x) = \max(0, x)$: non-saturating, it trains ~6× faster than tanh and mitigates gradient saturation in deep stacks;
- **Dropout** (rate 0.5 in FC layers): randomly zeroing units at training time to prevent co-adaptation and overfitting (Hinton et al., 2012; Srivastava et al., 2014);
- **Dual-GPU training** (two GTX 580, 3 GB each): the model was split across GPUs, pioneering large-scale GPU training;
- **Data augmentation** (random crops, horizontal flips, PCA color jitter), **overlapping max pooling** ($3\times3$, stride 2), and **local response normalization** (LRN, later abandoned in favor of batch norm).

## 6. The Golden Age: VGG, GoogLeNet, ResNet

### 6.1 VGG (Simonyan & Zisserman, 2014)

**VGG** (Simonyan, K., Zisserman, A., 2015, *"Very Deep Convolutional Networks for Large-Scale Image Recognition"*, ICLR; arXiv 2014) systematized depth using only $3\times3$ convolutions. Two stacked $3\times3$ layers have the receptive field of one $5\times5$; three match a $7\times7$ — with fewer parameters ($3 \cdot 9C^2 = 27C^2$ vs $49C^2$) and more nonlinearities. VGG-16/VGG-19 (16/19 weight layers, ~138M parameters, channels doubling 64→128→256→512 after each max pool) took 2nd place in ILSVRC-2014 classification and 1st in localization; its uniform design made it the default feature-extraction backbone for years.

### 6.2 GoogLeNet / Inception (Szegedy et al., 2014)

**GoogLeNet** (Szegedy, C., et al., 2015, *"Going Deeper with Convolutions"*, CVPR; ILSVRC-2014 classification winner, 6.7% top-5) is a 22-layer network built from **Inception modules**: parallel branches of $1\times1$, $3\times3$, $5\times5$ convolutions and $3\times3$ max pooling, concatenated along the channel axis:

$$
y = \big[\, f_{1\times1}(x)\; \|\; f_{3\times3}(f^{r}_{1\times1}(x))\; \|\; f_{5\times5}(f^{r}_{1\times1}(x))\; \|\; f_{1\times1}(\text{pool}(x)) \,\big]
$$

The $1\times1$ "bottleneck" convolutions ($f^r_{1\times1}$) reduce channel dimension before the expensive $3\times3/5\times5$ operations, so the network captures multi-scale features cheaply: only **~7M parameters** (vs 60M for AlexNet, 138M for VGG). Auxiliary classifiers injected gradient mid-network during training. Successors: Inception-v2/v3 (Szegedy et al., 2016; factorized convolutions, batch norm), Inception-v4 / Inception-ResNet (2017).

### 6.3 ResNet (He et al., 2015) and the residual connection

**ResNet** (He, K., Zhang, X., Ren, S., Sun, J., 2016, *"Deep Residual Learning for Image Recognition"*, CVPR; arXiv:1512.03385, Dec 2015) solved the **degradation problem**: naively stacking more layers made even *training* error worse. The fix is to have each block learn a **residual function** with respect to its input via an identity shortcut:

$$
y = \mathcal{F}(x, \{W_i\}) + x
$$

where typically $\mathcal{F}(x) = W_2\, \sigma(\text{BN}(W_1 x))$ (two or three conv-BN-ReLU stages), followed by $\sigma(y)$. When dimensions change, a projection is used: $y = \mathcal{F}(x) + W_s x$. Deep ResNets use a **bottleneck block** ($1\times1$ reduce → $3\times3$ → $1\times1$ expand).

**Why it fixes vanishing gradients / degradation.** Consider stacked residual blocks $x_{l+1} = x_l + \mathcal{F}(x_l)$. Unrolling to any deeper layer $L$:

$$
x_L = x_l + \sum_{i=l}^{L-1} \mathcal{F}(x_i)
$$

and by the chain rule the gradient of the loss $\mathcal{L}$ is:

$$
\frac{\partial \mathcal{L}}{\partial x_l} = \frac{\partial \mathcal{L}}{\partial x_L}\left(1 + \frac{\partial}{\partial x_l} \sum_{i=l}^{L-1} \mathcal{F}(x_i)\right)
$$

The additive "$1$" term means the gradient flows **directly** from any layer to any shallower layer without being multiplied through dozens of weight matrices; it cannot vanish even if the residual branch's Jacobian is small (He et al., 2016, *"Identity Mappings in Deep Residual Networks"*, ECCV). Moreover, learning $\mathcal{F} \approx 0$ (an identity mapping) is trivial — the network can only improve on shallower counterparts. ResNet-152 (8× deeper than VGG-19, yet cheaper in FLOPs) achieved **3.57% top-5 error** as an ensemble, winning ILSVRC-2015 classification, detection, and localization, plus COCO detection and segmentation. The residual connection is arguably the most influential architectural idea in deep learning, adopted by Transformers as well.

## 7. Efficient and Modern Architectures

### 7.1 DenseNet (Huang et al., 2017)

**DenseNet** (Huang, G., Liu, Z., van der Maaten, L., Weinberger, K. Q., 2017, *"Densely Connected Convolutional Networks"*, CVPR Best Paper) generalizes shortcuts: within a dense block, layer $\ell$ receives the **concatenation** of all preceding feature maps:

$$
x_\ell = H_\ell\big([\,x_0, x_1, \ldots, x_{\ell-1}\,]\big)
$$

where $H_\ell$ is BN → ReLU → conv and $[\cdot]$ denotes channel-wise concatenation (vs ResNet's addition). Each layer adds only $k$ channels (the *growth rate*, e.g. $k = 32$), so features are **reused** rather than recomputed, yielding strong parameter efficiency, implicit deep supervision, and excellent gradient flow. Transition layers ($1\times1$ conv + $2\times2$ average pooling) compress channels between blocks.

### 7.2 MobileNet (Howard et al., 2017): depthwise separable convolution

**MobileNet** (Howard, A. G., et al., 2017, *"MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications"*, arXiv:1704.04861) factorizes a standard convolution into:

1. **Depthwise convolution** — one $D_K \times D_K$ filter per input channel (no cross-channel mixing):
$$
\hat{y}_m(i, j) = \sum_{u,v} \hat{K}_m(u, v)\; x_m(i + u,\; j + v)
$$
2. **Pointwise convolution** — a $1\times1$ convolution mixing channels:
$$
y_n(i, j) = \sum_{m=1}^{M} W_{n,m}\; \hat{y}_m(i, j)
$$

Cost comparison on a $D_F \times D_F$ feature map with $M$ input and $N$ output channels:

$$
\text{Standard: } D_K^2 \cdot M \cdot N \cdot D_F^2
\qquad
\text{Separable: } D_K^2 \cdot M \cdot D_F^2 + M \cdot N \cdot D_F^2
$$

Reduction ratio:

$$
\frac{D_K^2 \, M \, D_F^2 + M N D_F^2}{D_K^2 \, M \, N \, D_F^2} = \frac{1}{N} + \frac{1}{D_K^2}
$$

For $3\times3$ kernels this is an ~8–9× reduction in computation with a small accuracy loss. MobileNetV2 (Sandler et al., 2018) added *inverted residuals with linear bottlenecks*; MobileNetV3 (Howard et al., 2019) added squeeze-and-excitation and neural architecture search.

### 7.3 EfficientNet (Tan & Le, 2019): compound scaling

**EfficientNet** (Tan, M., Le, Q. V., 2019, *"EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks"*, ICML) observed that scaling depth, width, or resolution *in isolation* saturates. **Compound scaling** scales all three jointly with one coefficient $\phi$:

$$
\text{depth } d = \alpha^{\phi}, \qquad \text{width } w = \beta^{\phi}, \qquad \text{resolution } r = \gamma^{\phi}
$$

$$
\text{subject to } \alpha \cdot \beta^2 \cdot \gamma^2 \approx 2, \quad \alpha, \beta, \gamma \geq 1
$$

Since FLOPs scale as $d \cdot w^2 \cdot r^2$, the constraint makes total FLOPs grow as $\approx 2^{\phi}$. From a NAS-found baseline (EfficientNet-B0, built on MBConv blocks with squeeze-and-excitation; grid search gave $\alpha = 1.2$, $\beta = 1.1$, $\gamma = 1.15$), scaling produced the B1–B7 family; B7 reached 84.3% ImageNet top-1 with 8.4× fewer parameters than the best prior CNN. EfficientNetV2 (Tan & Le, 2021) improved training speed.

### 7.4 ConvNeXt (Liu et al., 2022)

**ConvNeXt** (Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., Xie, S., 2022, *"A ConvNet for the 2020s"*, CVPR) answered the Vision Transformer wave by "modernizing" a ResNet step by step with Transformer-era design choices, while remaining a pure ConvNet: stage compute ratio 3:3:9:3 (like Swin), a **patchify stem** ($4\times4$ conv, stride 4), depthwise convolutions enlarged to $7\times7$, **inverted bottlenecks**, **GELU** instead of ReLU (and fewer activations), **LayerNorm** instead of BatchNorm (and fewer norms), separate downsampling layers, and modern training recipes (AdamW, 300 epochs, heavy augmentation). ConvNeXt matches or beats Swin Transformer (up to **87.8% ImageNet top-1**, and superior COCO/ADE20K transfer), proving that much of ViT's advantage was training methodology and design details, not attention per se.

## 8. Batch Normalization in CNNs

**Batch Normalization** (Ioffe, S., Szegedy, C., 2015, *"Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift"*, ICML) normalizes each activation over the mini-batch, then rescales with learnable parameters:

$$
\mu_{\mathcal{B}} = \frac{1}{m}\sum_{i=1}^{m} x_i, \qquad
\sigma_{\mathcal{B}}^2 = \frac{1}{m}\sum_{i=1}^{m} (x_i - \mu_{\mathcal{B}})^2
$$

$$
\hat{x}_i = \frac{x_i - \mu_{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}}, \qquad
y_i = \gamma\, \hat{x}_i + \beta
$$

**CNN specificity**: to respect convolutional weight sharing, normalization statistics are computed **per channel**, over the batch *and* all spatial positions jointly — the effective "batch" for channel $c$ has $m \cdot H \cdot W$ elements, and there is one pair $(\gamma_c, \beta_c)$ per channel, not per pixel. At inference, running (moving-average) estimates of $\mu$ and $\sigma^2$ replace batch statistics, allowing BN to be folded into the preceding convolution.

Benefits: much higher learning rates, faster convergence, reduced sensitivity to initialization, regularization (reducing the need for dropout), and smoothing of the optimization landscape (Santurkar et al., 2018, showed the original "internal covariate shift" explanation is incomplete). BN is integral to Inception-v2+, ResNet, DenseNet, MobileNet, EfficientNet. Alternatives for small batches or other modalities: Layer Norm (Ba et al., 2016), Instance Norm, **Group Norm** (Wu & He, 2018).

## 9. Object Detection Architectures

### 9.1 The R-CNN family (two-stage detectors)

- **R-CNN** (Girshick, R., Donahue, J., Darrell, T., Malik, J., 2014, *"Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation"*, CVPR): ~2000 region proposals from **selective search**, each warped and passed through a CNN, classified by per-class SVMs, with bounding-box regression. Accurate (mAP 58.5% on VOC07) but extremely slow (~47 s/image) since the CNN runs once per region.
- **Fast R-CNN** (Girshick, R., 2015, ICCV): runs the CNN **once** on the whole image; an **RoI Pooling** layer extracts a fixed-size feature vector per proposal; a single network jointly predicts class (softmax) and box offsets, trained with a multi-task loss $\mathcal{L} = \mathcal{L}_{cls} + \lambda [u \geq 1]\, \mathcal{L}_{loc}$ (smooth-$L_1$ for boxes). mAP 70.0% on VOC07, >200× faster inference than R-CNN.
- **Faster R-CNN** (Ren, S., He, K., Girshick, R., Sun, J., 2015, *"Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks"*, NeurIPS): replaces selective search with a **Region Proposal Network (RPN)** — a small fully convolutional head sliding over shared feature maps, predicting objectness and box offsets relative to $k$ **anchors** (multi-scale, multi-aspect-ratio) at each position. Proposals become nearly free; the whole detector is end-to-end trainable at ~5 fps, and it remains the template for two-stage detection.

### 9.2 YOLO (one-stage) and its loss

**YOLO** (Redmon, J., Divvala, S., Girshick, R., Farhadi, A., 2016, *"You Only Look Once: Unified, Real-Time Object Detection"*, CVPR) reframes detection as a **single regression**: the image is divided into an $S \times S$ grid ($S = 7$); each cell predicts $B$ boxes ($B = 2$) with confidence, plus $C$ class probabilities — one forward pass, 45 fps (155 fps for Fast YOLO). The sum-squared-error loss:

$$
\begin{aligned}
\mathcal{L} = \;& \lambda_{\text{coord}} \sum_{i=0}^{S^2} \sum_{j=0}^{B} \mathbb{1}_{ij}^{\text{obj}} \left[ (x_i - \hat{x}_i)^2 + (y_i - \hat{y}_i)^2 \right] \\
+\;& \lambda_{\text{coord}} \sum_{i=0}^{S^2} \sum_{j=0}^{B} \mathbb{1}_{ij}^{\text{obj}} \left[ \left(\sqrt{w_i} - \sqrt{\hat{w}_i}\right)^2 + \left(\sqrt{h_i} - \sqrt{\hat{h}_i}\right)^2 \right] \\
+\;& \sum_{i=0}^{S^2} \sum_{j=0}^{B} \mathbb{1}_{ij}^{\text{obj}} \left( C_i - \hat{C}_i \right)^2
\;+\; \lambda_{\text{noobj}} \sum_{i=0}^{S^2} \sum_{j=0}^{B} \mathbb{1}_{ij}^{\text{noobj}} \left( C_i - \hat{C}_i \right)^2 \\
+\;& \sum_{i=0}^{S^2} \mathbb{1}_{i}^{\text{obj}} \sum_{c \in \text{classes}} \left( p_i(c) - \hat{p}_i(c) \right)^2
\end{aligned}
$$

with $\lambda_{\text{coord}} = 5$ (emphasize localization) and $\lambda_{\text{noobj}} = 0.5$ (most cells are background — prevents confidence collapse); $\mathbb{1}_{ij}^{\text{obj}}$ selects the predictor "responsible" for the object (highest IoU); square roots on $w, h$ make errors on small boxes matter more. Successors: YOLOv2/9000 (2017, anchors), YOLOv3 (2018, multi-scale FPN-style heads), then YOLOv4–v11+ by other teams.

### 9.3 SSD

**SSD** (Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., Berg, A. C., 2016, *"SSD: Single Shot MultiBox Detector"*, ECCV) is a one-stage detector predicting class scores and offsets for **default boxes** of multiple aspect ratios on **feature maps at several scales** (early layers detect small objects, deep layers large ones). Loss: $\mathcal{L} = \frac{1}{N}(\mathcal{L}_{conf} + \alpha\, \mathcal{L}_{loc})$ with hard negative mining (3:1 negative:positive). SSD300 matched Faster R-CNN accuracy at real-time speed (59 fps). The one-stage class-imbalance problem was later addressed by **Focal Loss** in RetinaNet (Lin et al., 2017): $\mathcal{L}_{FL} = -\alpha_t (1 - p_t)^{\gamma} \log(p_t)$.

## 10. Segmentation Architectures

### 10.1 FCN

**FCN** (Long, J., Shelhamer, E., Darrell, T., 2015, *"Fully Convolutional Networks for Semantic Segmentation"*, CVPR) is the founding work of dense prediction: replace the fully connected layers of a classification CNN with $1\times1$ convolutions, so the network outputs a spatial class map for arbitrary input sizes; upsample with learned **transposed convolutions** ("deconvolutions"); and fuse coarse deep predictions with shallow, fine-grained features via **skip fusions** (FCN-32s → FCN-16s → FCN-8s), trained end-to-end with per-pixel cross-entropy.

### 10.2 U-Net

**U-Net** (Ronneberger, O., Fischer, P., Brox, T., 2015, *"U-Net: Convolutional Networks for Biomedical Image Segmentation"*, MICCAI) is a symmetric **encoder–decoder**:

- **Contracting path (encoder)**: repeated [two $3\times3$ convs + ReLU] → $2\times2$ max pool, doubling channels at each of 4 levels (64→128→256→512→1024) — captures *context*;
- **Expanding path (decoder)**: $2\times2$ up-convolution halving channels, **concatenation with the corresponding encoder feature map (skip connection)**, then two $3\times3$ convs — recovers *localization*;
- final $1\times1$ conv maps to class scores.

The skip connections reinject high-resolution spatial detail lost to pooling, enabling pixel-accurate boundaries, and provide short gradient paths. Trained with heavy elastic augmentation and a weighted cross-entropy emphasizing boundaries between touching cells, U-Net excels with very few annotated images and is the dominant architecture in medical imaging — and, notably, the standard denoising backbone of diffusion models. Variants: 3D U-Net (Çiçek et al., 2016), V-Net (Milletari et al., 2016, with Dice loss), U-Net++ (2018), nnU-Net (Isensee et al., 2021).

### 10.3 Mask R-CNN

**Mask R-CNN** (He, K., Gkioxari, G., Dollár, P., Girshick, R., 2017, *"Mask R-CNN"*, ICCV, Marr Prize) extends Faster R-CNN for **instance segmentation** with a third, FCN-based branch predicting a binary mask per RoI, alongside classification and box regression:

$$
\mathcal{L} = \mathcal{L}_{cls} + \mathcal{L}_{box} + \mathcal{L}_{mask}
$$

where $\mathcal{L}_{mask}$ is the average per-pixel binary cross-entropy applied **only to the mask of the ground-truth class** — decoupling mask and class prediction (no inter-class competition). Its key technical contribution is **RoIAlign**, which replaces RoI Pooling's harsh coordinate quantization with **bilinear interpolation** at exactly computed sampling points, preserving pixel-level spatial alignment — essential for masks and for keypoint estimation.

## 11. 1D and 3D CNNs

### 11.1 1D CNNs (signals, audio, text)

The 1D convolution over a sequence $x$ with kernel of size $k$:

$$
y(i) = \sum_{m=0}^{k-1} \sum_{c=1}^{C_{in}} K_c(m)\; x_c(i + m)
$$

Applications:

- **Text**: Kim, Y. (2014, *"Convolutional Neural Networks for Sentence Classification"*, EMNLP) convolves filters of widths 3/4/5 over word-embedding sequences (each filter an n-gram detector), followed by max-over-time pooling — a strong, simple sentence classifier. See also character-level CNNs (Zhang et al., 2015).
- **Audio / time series**: **WaveNet** (van den Oord et al., 2016) generates raw audio with stacked **dilated causal** 1D convolutions (dilations 1, 2, 4, …, 512) for exponentially large receptive fields; Temporal Convolutional Networks (TCN; Bai et al., 2018) apply the same recipe to generic sequence modeling and often beat RNNs. 1D CNNs are standard for ECG/EEG analysis, fault detection, and sensor data (Kiranyaz et al., 2021, survey).

### 11.2 3D CNNs (video, medical imaging)

3D convolution adds a depth/time axis; for a spatiotemporal kernel $k_t \times k_h \times k_w$:

$$
y(t, i, j) = \sum_{l=0}^{k_t-1} \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} K(l, m, n)\; x(t + l,\; i + m,\; j + n)
$$

so features capture motion as well as appearance. Landmarks:

- Ji et al. (2013, TPAMI), *"3D Convolutional Neural Networks for Human Action Recognition"* — first 3D CNN for video;
- **C3D** (Tran, D., et al., 2015, *"Learning Spatiotemporal Features with 3D Convolutional Networks"*, ICCV): homogeneous $3\times3\times3$ kernels shown to be the best choice; generic video features;
- **I3D** (Carreira, J., Zisserman, A., 2017, CVPR): "inflates" 2D ImageNet-pretrained kernels into 3D ($k\times k \to t \times k \times k$), two-stream RGB+flow, state of the art on Kinetics;
- Factorized variants — **P3D** (Qiu et al., 2017), **R(2+1)D** (Tran et al., 2018): decompose $3\times3\times3$ into a $1\times3\times3$ spatial plus $3\times1\times1$ temporal convolution, cheaper and often more accurate; **SlowFast** (Feichtenhofer et al., 2019) uses dual pathways at different frame rates.
- **Medical imaging**: 3D U-Net (Çiçek et al., 2016) and V-Net (Milletari et al., 2016) segment volumetric CT/MRI data directly, exploiting full 3D context at the cost of cubic memory growth — hence patch-based training and hybrid 2.5D approaches.

## Summary Timeline

| Year | Milestone | Reference |
|------|-----------|-----------|
| 1980 | Neocognitron (S/C cells) | Fukushima, Biol. Cybernetics |
| 1989–98 | Backprop CNNs → LeNet-5 | LeCun et al., Proc. IEEE 1998 |
| 2012 | AlexNet: ReLU, dropout, GPUs — 15.3% top-5 | Krizhevsky, Sutskever, Hinton, NeurIPS |
| 2014 | VGG (3×3 depth), GoogLeNet (Inception), R-CNN | Simonyan & Zisserman; Szegedy et al.; Girshick et al. |
| 2015 | BatchNorm; ResNet ($y = \mathcal{F}(x) + x$, 3.57%); FCN; U-Net; Faster R-CNN | Ioffe & Szegedy; He et al.; Long et al.; Ronneberger et al.; Ren et al. |
| 2016 | YOLO, SSD; dilated convs | Redmon et al.; Liu et al.; Yu & Koltun |
| 2017 | DenseNet, MobileNet, Mask R-CNN | Huang et al.; Howard et al.; He et al. |
| 2019 | EfficientNet (compound scaling) | Tan & Le, ICML |
| 2022 | ConvNeXt (87.8% top-1, pure ConvNet) | Liu et al., CVPR |

Sources: [Fukushima 1980 (Springer)](https://link.springer.com/article/10.1007/BF00344251), [output-size formula (Baeldung)](https://www.baeldung.com/cs/convolutional-layer-size), [LeNet-5 architecture](https://www.analyticsvidhya.com/blog/2021/03/the-architecture-of-lenet-5/), [AlexNet paper (PDF)](https://cvml.ista.ac.at/courses/DLWT_W17/material/AlexNet.pdf), [ResNet arXiv:1512.03385](https://arxiv.org/abs/1512.03385), [DenseNet journal version](https://www.cs.cornell.edu/~kilian/resources/DenseNet_Journal.pdf), [EfficientNet (PMLR)](https://proceedings.mlr.press/v97/tan19a.html), [ConvNeXt arXiv:2201.03545](https://arxiv.org/pdf/2201.03545), [GoogLeNet overview](https://medium.com/@saba99/googlenet-bbe1dc996f0e), [YOLOv1 loss walkthrough](https://pyimagesearch.com/2022/04/11/understanding-a-real-time-object-detection-network-you-only-look-once-yolov1/), [Faster R-CNN (NeurIPS 2015)](https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf), [Fast R-CNN (ICCV 2015)](https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Girshick_Fast_R-CNN_ICCV_2015_paper.pdf), [SSD (Springer)](https://link.springer.com/chapter/10.1007/978-3-319-46448-0_2), [FCN (CVPR 2015)](https://openaccess.thecvf.com/content_cvpr_2015/html/Long_Fully_Convolutional_Networks_2015_CVPR_paper.html), [U-Net guide](https://medium.com/@alejandro.itoaramendia/decoding-the-u-net-a-complete-guide-810b1c6d56d8), [Mask R-CNN / RoIAlign](https://saeedmehrang.github.io/blogs/computer-vision/segmentation-models/mask-rcnn/), [BatchNorm (PMLR)](https://proceedings.mlr.press/v37/ioffe15.html), [C3D arXiv:1412.0767](https://arxiv.org/abs/1412.0767), [Kim 2014 arXiv:1408.5882](https://arxiv.org/abs/1408.5882).
# Recurrent Networks and Sequence Models

## 1. Vanilla Recurrent Neural Networks (Elman, Jordan)

Recurrent neural networks (RNNs) process sequences $x_1, x_2, \dots, x_T$ by maintaining a **hidden state** $h_t$ that acts as a compressed summary of everything seen so far. Two foundational architectures established this idea:

- **Jordan networks** (Jordan, 1986, *"Serial Order: A Parallel Distributed Processing Approach"*) feed the network's *output* back into a set of "state units" that serve as extra inputs at the next step.
- **Elman networks** (Elman, 1990, *"Finding Structure in Time"*, Cognitive Science 14:179–211) instead copy the *hidden layer* into "context units" and feed those back into the hidden layer — this is the modern "vanilla RNN."

**Elman RNN equations.** At each time step $t$:

$$h_t = \tanh\left(W_{hh}\, h_{t-1} + W_{xh}\, x_t + b_h\right)$$

$$y_t = W_{hy}\, h_t + b_y \qquad \text{(often followed by a softmax: } \hat{y}_t = \mathrm{softmax}(W_{hy} h_t + b_y)\text{)}$$

where $x_t \in \mathbb{R}^d$ is the input, $h_t \in \mathbb{R}^n$ the hidden state, $W_{xh} \in \mathbb{R}^{n \times d}$, $W_{hh} \in \mathbb{R}^{n \times n}$, $W_{hy} \in \mathbb{R}^{m \times n}$. The crucial property is **weight sharing across time**: the same $(W_{hh}, W_{xh})$ are applied at every step, making the RNN a dynamical system $h_t = f(h_{t-1}, x_t; \theta)$ and, in principle, Turing-complete (Siegelmann & Sontag, 1995).

**Jordan RNN** differs only in the recurrence source:

$$h_t = \tanh\left(W_{hh}\, y_{t-1} + W_{xh}\, x_t + b_h\right), \qquad y_t = \sigma_y(W_{hy} h_t + b_y)$$

## 2. Backpropagation Through Time (BPTT) and the Vanishing/Exploding Gradient Problem

**BPTT** (Werbos, 1990, *"Backpropagation through time: what it does and how to do it"*, Proc. IEEE) trains an RNN by **unrolling** it into a deep feedforward network with $T$ layers sharing the same weights, then applying standard backpropagation. For a loss $L = \sum_t L_t$, the gradient with respect to the recurrent matrix sums contributions over all time-step pairs:

$$\frac{\partial L}{\partial W_{hh}} = \sum_{t=1}^{T} \sum_{k=1}^{t} \frac{\partial L_t}{\partial h_t} \left( \prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}} \right) \frac{\partial h_k}{\partial W_{hh}}$$

The critical term is the product of Jacobians:

$$\frac{\partial h_t}{\partial h_k} = \prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}} = \prod_{i=k+1}^{t} W_{hh}^\top \, \mathrm{diag}\!\left(\tanh'(a_i)\right)$$

**Eigenvalue analysis** (Bengio, Simard & Frasconi, 1994, *"Learning long-term dependencies with gradient descent is difficult"*, IEEE Trans. Neural Networks; Pascanu, Mikolov & Bengio, 2013, *"On the difficulty of training recurrent neural networks"*, ICML): let $\rho(W_{hh})$ be the spectral radius (largest absolute eigenvalue). Since $\|\partial h_t / \partial h_k\| \le (\sigma_{\max}(W_{hh}) \cdot \gamma)^{t-k}$ where $\gamma$ bounds $|\tanh'| \le 1$:

- If the largest singular value satisfies $\sigma_{\max} < 1/\gamma$, gradients **vanish exponentially**: $\|\partial h_t/\partial h_k\| \sim \lambda^{t-k} \to 0$. It is *sufficient* for the largest eigenvalue to be $< 1$ for long-term components to vanish. The network then cannot learn dependencies longer than a few dozen steps.
- If $\rho(W_{hh}) > 1$ (a *necessary* condition), gradients can **explode exponentially**, causing loss spikes and NaNs.

**Gradient clipping** (Pascanu et al., 2013) is the standard remedy for explosion — rescale the gradient when its norm exceeds a threshold $\tau$:

$$g \leftarrow \begin{cases} \dfrac{\tau}{\|g\|}\, g & \text{if } \|g\| > \tau \\ g & \text{otherwise} \end{cases}$$

Vanishing gradients have no such simple fix; they motivated gated architectures (LSTM/GRU), careful initialization (orthogonal/identity recurrent matrices), and **truncated BPTT** (backpropagating only $k$ steps, trading bias for tractability).

## 3. Long Short-Term Memory (LSTM)

Introduced by **Hochreiter & Schmidhuber (1997, *"Long Short-Term Memory"*, Neural Computation 9(8):1735–1780)**, the LSTM solves vanishing gradients with a **cell state** $c_t$ traversed by an additive (rather than multiplicative) recurrence — the "constant error carousel." The original 1997 paper had input and output gates only; the **forget gate** was added by **Gers, Schmidhuber & Cummins (2000, *"Learning to Forget: Continual Prediction with LSTM"*)**.

**Complete equations of the standard (modern) LSTM**, with $\sigma$ the logistic sigmoid and $\odot$ elementwise product:

$$f_t = \sigma\left(W_f x_t + U_f h_{t-1} + b_f\right) \qquad \text{(forget gate: how much of } c_{t-1} \text{ to keep)}$$

$$i_t = \sigma\left(W_i x_t + U_i h_{t-1} + b_i\right) \qquad \text{(input gate: how much new content to write)}$$

$$\tilde{c}_t = \tanh\left(W_c x_t + U_c h_{t-1} + b_c\right) \qquad \text{(candidate cell content)}$$

$$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \qquad \text{(cell state update — additive path)}$$

$$o_t = \sigma\left(W_o x_t + U_o h_{t-1} + b_o\right) \qquad \text{(output gate: how much cell to expose)}$$

$$h_t = o_t \odot \tanh(c_t) \qquad \text{(hidden state)}$$

**Why it works:** the gradient through the cell path is $\partial c_t / \partial c_{t-1} = \mathrm{diag}(f_t)$ (plus gate-dependent terms). When $f_t \approx 1$, error flows back essentially unattenuated over hundreds of steps — no repeated multiplication by $W_{hh}$. A practical trick is initializing $b_f$ to a positive value (e.g., 1 or 2) so the network starts by remembering (Jozefowicz, Zaremba & Sutskever, 2015).

**Peephole variant** (Gers & Schmidhuber, 2000, *"Recurrent Nets that Time and Count"*): the gates also see the cell state directly, enabling precise timing behavior:

$$f_t = \sigma(W_f x_t + U_f h_{t-1} + V_f \odot c_{t-1} + b_f), \quad i_t = \sigma(W_i x_t + U_i h_{t-1} + V_i \odot c_{t-1} + b_i)$$

$$o_t = \sigma(W_o x_t + U_o h_{t-1} + V_o \odot c_{t} + b_o)$$

The large ablation study of Greff et al. (2017, *"LSTM: A Search Space Odyssey"*, IEEE TNNLS) found the forget gate and output activation to be the most critical components, with most variants (including peepholes) not significantly beating the vanilla formulation.

## 4. Gated Recurrent Unit (GRU)

Proposed by **Cho et al. (2014, *"Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation"*, EMNLP; arXiv:1406.1078)**, the GRU merges the cell and hidden state and uses only **two gates**:

$$z_t = \sigma\left(W_z x_t + U_z h_{t-1} + b_z\right) \qquad \text{(update gate)}$$

$$r_t = \sigma\left(W_r x_t + U_r h_{t-1} + b_r\right) \qquad \text{(reset gate)}$$

$$\tilde{h}_t = \tanh\left(W_h x_t + U_h (r_t \odot h_{t-1}) + b_h\right) \qquad \text{(candidate state)}$$

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$

(The convex-combination convention $h_t = z_t \odot h_{t-1} + (1-z_t)\odot \tilde h_t$ also appears in the literature; the two are equivalent up to relabeling $z \leftrightarrow 1-z$.) The **reset gate** $r_t$ controls how much past state contributes to the candidate (allowing the unit to "forget" and act like a fresh network), while the **update gate** $z_t$ interpolates between copying $h_{t-1}$ and writing $\tilde{h}_t$ — the same leaky-integration principle as the LSTM's forget/input pair, with ~25% fewer parameters ($3$ weight blocks vs $4$). Empirically, GRU and LSTM perform comparably (Chung et al., 2014, *"Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling"*), with LSTM slightly more robust on tasks needing precise counting.

## 5. Bidirectional and Deep (Stacked) RNNs

**Bidirectional RNNs** (Schuster & Paliwal, 1997, *"Bidirectional Recurrent Neural Networks"*, IEEE Trans. Signal Processing 45(11):2673–2681) run two independent RNNs over the sequence — one forward, one backward — and combine their states, so each output sees both past and future context:

$$\overrightarrow{h}_t = f\left(\overrightarrow{W} x_t + \overrightarrow{U}\, \overrightarrow{h}_{t-1}\right), \qquad \overleftarrow{h}_t = f\left(\overleftarrow{W} x_t + \overleftarrow{U}\, \overleftarrow{h}_{t+1}\right)$$

$$y_t = g\left(V\, [\overrightarrow{h}_t ; \overleftarrow{h}_t] + b\right)$$

BiLSTMs (Graves & Schmidhuber, 2005) became the workhorse of speech recognition, tagging, and pre-Transformer contextual encoders (e.g., ELMo, 2018). They require the full sequence in advance, so they suit offline labeling, not streaming generation.

**Deep (stacked) RNNs** (Graves, Mohamed & Hinton, 2013, *"Speech Recognition with Deep Recurrent Neural Networks"*) stack $L$ recurrent layers, layer $\ell$ taking layer $\ell-1$'s states as input:

$$h_t^{(\ell)} = f\left(W^{(\ell)} h_t^{(\ell-1)} + U^{(\ell)} h_{t-1}^{(\ell)} + b^{(\ell)}\right), \qquad h_t^{(0)} = x_t$$

This adds depth "vertically" (representation hierarchy) on top of depth "in time." Typical setups use 2–8 layers with dropout applied only to non-recurrent connections (Zaremba et al., 2014) or variational dropout with masks shared across time (Gal & Ghahramani, 2016).

## 6. Seq2Seq / Encoder–Decoder

The **encoder–decoder** paradigm was introduced concurrently by Cho et al. (2014) and **Sutskever, Vinyals & Le (2014, *"Sequence to Sequence Learning with Neural Networks"*, NeurIPS; arXiv:1409.3215)**. An encoder RNN consumes the source $x_1,\dots,x_{T_x}$ into a fixed vector $v = h_{T_x}$ (Sutskever used a 4-layer LSTM); a decoder RNN then models the target autoregressively:

$$p(y_1, \dots, y_{T'} \mid x_1, \dots, x_T) = \prod_{t=1}^{T'} p\left(y_t \mid v, y_1, \dots, y_{t-1}\right)$$

with $s_t = \mathrm{LSTM}(s_{t-1}, y_{t-1})$, $s_0$ initialized from $v$, and $p(y_t \mid \cdot) = \mathrm{softmax}(W_o s_t)$. Training maximizes log-likelihood with teacher forcing; inference uses beam search. Two findings from Sutskever et al. proved influential: (i) **reversing the source sentence** markedly improved BLEU (34.8 on WMT'14 En→Fr) by creating short-range dependencies that ease optimization; (ii) the fixed-size vector $v$ is an **information bottleneck** — performance degrades on long sentences — which directly motivated attention.

## 7. Attention: Bahdanau (2014) and Luong (2015)

**Bahdanau, Cho & Bengio (2014/2015, *"Neural Machine Translation by Jointly Learning to Align and Translate"*, ICLR 2015; arXiv:1409.0473)** removed the bottleneck by letting the decoder attend to *all* encoder states $h_1, \dots, h_{T_x}$ (from a bidirectional GRU encoder). At decoder step $t$, with previous decoder state $s_{t-1}$:

**Alignment scores** (additive/MLP attention):

$$e_{tj} = a(s_{t-1}, h_j) = v_a^\top \tanh\left(W_a s_{t-1} + U_a h_j\right)$$

**Softmax normalization** into attention weights:

$$\alpha_{tj} = \frac{\exp(e_{tj})}{\sum_{k=1}^{T_x} \exp(e_{tk})}$$

**Context vector** (expected annotation):

$$c_t = \sum_{j=1}^{T_x} \alpha_{tj}\, h_j$$

**Decoder update and prediction:**

$$s_t = f(s_{t-1}, y_{t-1}, c_t), \qquad p(y_t \mid \cdot) = \mathrm{softmax}\left(g(s_t, y_{t-1}, c_t)\right)$$

**Luong, Pham & Manning (2015, *"Effective Approaches to Attention-based Neural Machine Translation"*, EMNLP; arXiv:1508.04025)** simplified and systematized this. Differences: attention uses the *current* decoder state $s_t$ (not $s_{t-1}$); the context is combined *after* the RNN step via $\tilde{h}_t = \tanh(W_c [c_t; s_t])$, then $p(y_t) = \mathrm{softmax}(W_s \tilde{h}_t)$. Luong proposed three **score functions**:

$$\mathrm{score}(s_t, h_j) = \begin{cases} s_t^\top h_j & \text{(dot)} \\ s_t^\top W_a h_j & \text{(general)} \\ v_a^\top \tanh\left(W_a [s_t; h_j]\right) & \text{(concat)} \end{cases}$$

plus **global** attention (over all source positions) versus **local** attention (a Gaussian-weighted window around a predicted position $p_t$). The dot-product form is the direct ancestor of Transformer attention $\mathrm{softmax}(QK^\top/\sqrt{d_k})V$ (Vaswani et al., 2017, *"Attention Is All You Need"*), which discarded recurrence entirely.

## 8. Echo State Networks, Reservoir Computing, Liquid State Machines

**Reservoir computing** sidesteps BPTT entirely: keep a large, random, *fixed* recurrent network (the reservoir) and train **only a linear readout**.

**Echo State Networks** (Jaeger, 2001, *"The 'Echo State' Approach to Analysing and Training Recurrent Neural Networks"*, GMD Report 148):

$$h_t = (1-\alpha)\, h_{t-1} + \alpha \tanh\left(W_{\text{in}} x_t + W\, h_{t-1}\right), \qquad y_t = W_{\text{out}} [x_t; h_t]$$

with leak rate $\alpha$, sparse random $W$ (~1% connectivity) rescaled so its **spectral radius** $\rho(W)$ is typically just below 1. This (heuristically) ensures the **echo state property**: the reservoir asymptotically washes out initial conditions and becomes a fading-memory function of the input history; larger $\rho$ gives longer memory, smaller $\rho$ shorter. Only $W_{\text{out}}$ is learned, in closed form by ridge regression:

$$W_{\text{out}} = Y H^\top \left(H H^\top + \lambda I\right)^{-1}$$

Training is thus convex, fast, and immune to vanishing gradients. ESNs excel at chaotic time-series prediction (e.g., Mackey–Glass; Jaeger & Haas, 2004, Science).

**Liquid State Machines** (Maass, Natschläger & Markram, 2002, *"Real-Time Computing Without Stable States"*, Neural Computation 14:2531–2560) are the spiking-neuron, biologically motivated counterpart: a recurrent "liquid" of leaky integrate-and-fire neurons provides a high-dimensional temporal expansion of input spike trains; a memoryless readout is trained on the liquid state. Maass et al. proved universal real-time computing power given the **separation property** (liquid) and **approximation property** (readout). The field survives today in physical reservoir computing (photonic, memristive, mechanical reservoirs).

## 9. Hopfield Networks (1982) and Modern Hopfield Networks (2020)

**Classical Hopfield network** (Hopfield, 1982, *"Neural networks and physical systems with emergent collective computational abilities"*, PNAS 79:2554–2558): a fully connected network of $N$ binary units $s_i \in \{-1, +1\}$ with symmetric weights ($w_{ij} = w_{ji}$, $w_{ii} = 0$) acting as **content-addressable associative memory**.

**Energy function:**

$$E = -\frac{1}{2} \sum_{i,j} w_{ij}\, s_i s_j + \sum_i \theta_i s_i$$

**Asynchronous update rule** (pick a unit, update):

$$s_i \leftarrow \mathrm{sign}\left(\sum_j w_{ij} s_j - \theta_i\right)$$

Each update never increases $E$, so the dynamics converge to a local minimum — an attractor. Patterns $\{\xi^\mu\}_{\mu=1}^{P}$ are stored via the **Hebbian rule** $w_{ij} = \frac{1}{N} \sum_\mu \xi_i^\mu \xi_j^\mu$.

**Capacity:** reliable retrieval holds only up to $P_{\max} \approx 0.138\, N$ patterns (Amit, Gutfreund & Sompolinsky, 1985, via spin-glass statistical mechanics); beyond this, spurious states and catastrophic interference dominate. (For essentially error-free storage the bound tightens to $N / (2 \ln N)$.)

**Modern Hopfield networks** (Ramsauer et al., 2020, *"Hopfield Networks is All You Need"*, ICLR 2021; building on Krotov & Hopfield, 2016, dense associative memories with polynomial energies): continuous states $q \in \mathbb{R}^d$, stored patterns as columns of $X = [x_1, \dots, x_P]$, and the **log-sum-exp energy**

$$E(q) = -\frac{1}{\beta} \log \sum_{i=1}^{P} \exp\left(\beta\, x_i^\top q\right) + \frac{1}{2} \|q\|^2 + \text{const}$$

whose update rule (a concave–convex procedure step) is

$$q^{\text{new}} = X\, \mathrm{softmax}\left(\beta X^\top q\right)$$

This yields **exponential storage capacity** (in $d$), retrieval in typically one step, and — the celebrated result — is *exactly the Transformer attention update* with $q$ as query and $X$ providing keys/values, unifying associative memory and attention. Hopfield received the 2024 Nobel Prize in Physics (shared with Hinton) for this line of work.

## 10. Boltzmann Machines, RBMs, Contrastive Divergence, Deep Belief Networks

**Boltzmann machines** (Ackley, Hinton & Sejnowski, 1985, *"A Learning Algorithm for Boltzmann Machines"*, Cognitive Science) are *stochastic* Hopfield networks with hidden units: binary units sampled from a **Boltzmann distribution** over the energy

$$p(s) = \frac{e^{-E(s)/T}}{Z}, \qquad Z = \sum_{s'} e^{-E(s')/T}$$

Exact learning requires intractable expectations over $Z$, so general Boltzmann machines were impractical.

**Restricted Boltzmann Machines** (Smolensky, 1986, as "Harmonium"; popularized by Hinton) impose a **bipartite** structure — visible units $v$, hidden units $h$, no intra-layer connections — with energy

$$E(v, h) = -\sum_i b_i v_i - \sum_j c_j h_j - \sum_{i,j} v_i\, w_{ij}\, h_j = -b^\top v - c^\top h - v^\top W h$$

and joint distribution $p(v,h) = e^{-E(v,h)}/Z$. Bipartiteness makes the conditionals **factorize**:

$$p(h_j = 1 \mid v) = \sigma\left(c_j + \sum_i w_{ij} v_i\right), \qquad p(v_i = 1 \mid h) = \sigma\left(b_i + \sum_j w_{ij} h_j\right)$$

enabling efficient block Gibbs sampling. The exact log-likelihood gradient is

$$\frac{\partial \log p(v)}{\partial w_{ij}} = \langle v_i h_j \rangle_{\text{data}} - \langle v_i h_j \rangle_{\text{model}}$$

The model term requires equilibrium sampling. **Contrastive Divergence** (Hinton, 2002, *"Training Products of Experts by Minimizing Contrastive Divergence"*, Neural Computation 14:1771–1800) approximates it with just $k$ Gibbs steps (usually $k=1$) started **from the data**:

$$\Delta w_{ij} \propto \langle v_i h_j \rangle_{0} - \langle v_i h_j \rangle_{k} \qquad \text{(CD-}k\text{)}$$

Biased but effective; Persistent CD (Tieleman, 2008) improves the negative-phase samples.

**Deep Belief Networks** (Hinton, Osindero & Teh, 2006, *"A Fast Learning Algorithm for Deep Belief Nets"*, Neural Computation 18:1527–1554): stack RBMs, training each layer greedily on the hidden activations of the layer below, then optionally fine-tune with backprop or wake–sleep. This **greedy layer-wise unsupervised pre-training** was the spark that launched the deep learning renaissance — it was the first practical recipe for training deep networks (pre-ReLU, pre-good-init), even though modern practice (ReLU, batch norm, residuals, large data) later made pre-training unnecessary for supervised tasks. Hinton's 2024 Nobel Prize citation prominently features Boltzmann machines.

## 11. Temporal Convolutional Networks (TCN)

**Bai, Kolter & Koltun (2018, *"An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling"*, arXiv:1803.01271)** distilled convolutional sequence modeling (lineage: WaveNet, van den Oord et al., 2016) into a generic architecture and showed it **outperforms LSTMs/GRUs on a broad benchmark suite** while exhibiting longer effective memory. Ingredients:

- **Causal convolutions:** output at time $t$ depends only on inputs $\le t$ (achieved by left padding).
- **Dilated convolutions:** with dilation $d$ and kernel size $k$,

$$F(t) = \sum_{i=0}^{k-1} f(i) \cdot x_{t - d \cdot i}$$

Doubling $d$ per layer ($d = 1, 2, 4, 8, \dots$) makes the **receptive field grow exponentially with depth**: $R = 1 + (k-1)\sum_{\ell} d_\ell \approx 1 + (k-1)(2^L - 1)$.
- **Residual blocks** (two dilated conv layers + weight norm + ReLU + dropout, with a $1{\times}1$ skip projection) stabilize deep stacks.

**Trade-offs vs RNNs:** TCNs train **in parallel across time** (no sequential state dependency), have stable gradients (backprop path length is depth, not sequence length — no temporal vanishing gradient), and offer flexible receptive-field control; but they need the whole input window at inference (larger memory for streaming), and history beyond the receptive field is truly gone, whereas an RNN's state can in principle carry unbounded history in $O(1)$ memory. The TCN paper, together with Transformers, drove the field's conclusion that recurrence is not necessary for most sequence tasks.

## 12. Neural Turing Machines and Differentiable Neural Computers

**Neural Turing Machines** (Graves, Wayne & Danihelka, 2014, *"Neural Turing Machines"*, arXiv:1410.5401, DeepMind) couple a controller network (LSTM or feedforward) to an **external memory matrix** $M_t \in \mathbb{R}^{N \times W}$ through fully **differentiable** read/write heads, so the whole system trains end-to-end by gradient descent.

**Reading** is attention-weighted: with weighting $w_t$ over $N$ locations ($\sum_i w_t(i) = 1$),

$$r_t = \sum_i w_t(i)\, M_t(i)$$

**Writing** decomposes into erase ($e_t \in [0,1]^W$) and add ($a_t$) vectors:

$$\tilde{M}_t(i) = M_{t-1}(i)\left[\mathbf{1} - w_t(i)\, e_t\right], \qquad M_t(i) = \tilde{M}_t(i) + w_t(i)\, a_t$$

**Addressing** combines: (i) **content-based** — cosine similarity to an emitted key $k_t$, sharpened by $\beta_t$:

$$w_t^c(i) = \frac{\exp\left(\beta_t\, K[k_t, M_t(i)]\right)}{\sum_j \exp\left(\beta_t\, K[k_t, M_t(j)]\right)}, \qquad K[u,v] = \frac{u \cdot v}{\|u\|\,\|v\|}$$

with (ii) **location-based** addressing: interpolation with the previous weighting ($g_t$), convolutional **rotational shift** ($s_t$), and sharpening ($\gamma_t$). NTMs learn algorithmic tasks — copy, repeat-copy, associative recall, sorting — and generalize to longer sequences than seen in training.

**Differentiable Neural Computers** (Graves et al., 2016, *"Hybrid computing using a neural network with dynamic external memory"*, Nature 538:471–476) refine the NTM: they drop the location-shift mechanism and add

- **dynamic memory allocation** via per-slot usage vectors $u_t$ (a differentiable "free list" enabling allocation and de-allocation),
- a **temporal link matrix** $L_t \in [0,1]^{N \times N}$ recording write order, letting read heads step forward/backward through the sequence in which data was written,
- multiple read heads combining content, forward, and backward modes.

DNCs solved bAbI question answering, graph traversal (e.g., London Underground shortest paths), and blocks-puzzle planning. Though hard to train and now superseded by Transformers in practice, NTM/DNC established the **memory-augmented neural network** paradigm and prefigured today's retrieval-augmented and tool-using architectures.

---

### Key references

- Jordan, M. I. (1986). *Serial Order: A Parallel Distributed Processing Approach.* ICS Report 8604, UCSD.
- Elman, J. L. (1990). *Finding Structure in Time.* Cognitive Science, 14(2), 179–211.
- Werbos, P. (1990). *Backpropagation Through Time: What It Does and How to Do It.* Proc. IEEE, 78(10).
- Bengio, Y., Simard, P., Frasconi, P. (1994). *Learning Long-Term Dependencies with Gradient Descent is Difficult.* IEEE Trans. Neural Networks, 5(2).
- Hochreiter, S., Schmidhuber, J. (1997). *Long Short-Term Memory.* Neural Computation, 9(8), 1735–1780.
- Schuster, M., Paliwal, K. K. (1997). *Bidirectional Recurrent Neural Networks.* IEEE Trans. Signal Processing, 45(11), 2673–2681.
- Gers, F., Schmidhuber, J., Cummins, F. (2000). *Learning to Forget: Continual Prediction with LSTM.* Neural Computation, 12(10).
- Jaeger, H. (2001). *The "Echo State" Approach to Analysing and Training Recurrent Neural Networks.* GMD Report 148.
- Maass, W., Natschläger, T., Markram, H. (2002). *Real-Time Computing Without Stable States.* Neural Computation, 14(11), 2531–2560.
- Hinton, G. E. (2002). *Training Products of Experts by Minimizing Contrastive Divergence.* Neural Computation, 14(8), 1771–1800.
- Hinton, G. E., Osindero, S., Teh, Y. W. (2006). *A Fast Learning Algorithm for Deep Belief Nets.* Neural Computation, 18(7), 1527–1554.
- Pascanu, R., Mikolov, T., Bengio, Y. (2013). *On the Difficulty of Training Recurrent Neural Networks.* ICML.
- Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y. (2014). *Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation.* EMNLP.
- Sutskever, I., Vinyals, O., Le, Q. V. (2014). *Sequence to Sequence Learning with Neural Networks.* NeurIPS.
- Bahdanau, D., Cho, K., Bengio, Y. (2015). *Neural Machine Translation by Jointly Learning to Align and Translate.* ICLR (arXiv:1409.0473, 2014).
- Luong, M.-T., Pham, H., Manning, C. D. (2015). *Effective Approaches to Attention-based Neural Machine Translation.* EMNLP.
- Graves, A., Wayne, G., Danihelka, I. (2014). *Neural Turing Machines.* arXiv:1410.5401.
- Graves, A., et al. (2016). *Hybrid Computing Using a Neural Network with Dynamic External Memory.* Nature, 538, 471–476.
- Greff, K., et al. (2017). *LSTM: A Search Space Odyssey.* IEEE TNNLS, 28(10).
- Bai, S., Kolter, J. Z., Koltun, V. (2018). *An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling.* arXiv:1803.01271.
- Hopfield, J. J. (1982). *Neural Networks and Physical Systems with Emergent Collective Computational Abilities.* PNAS, 79(8), 2554–2558.
- Amit, D. J., Gutfreund, H., Sompolinsky, H. (1985). *Storing Infinite Numbers of Patterns in a Spin-Glass Model of Neural Networks.* Phys. Rev. Lett., 55(14).
- Ackley, D. H., Hinton, G. E., Sejnowski, T. J. (1985). *A Learning Algorithm for Boltzmann Machines.* Cognitive Science, 9(1).
- Ramsauer, H., et al. (2021). *Hopfield Networks is All You Need.* ICLR (arXiv:2008.02217, 2020).

Sources consulted for validation: [Pascanu et al. 2013 (arXiv)](https://arxiv.org/pdf/1211.5063), [Sutskever et al. 2014 (arXiv)](https://arxiv.org/abs/1409.3215), [Bai et al. 2018 overview](https://www.alphaxiv.org/overview/1803.01271v2), [Baeldung — Luong vs Bahdanau attention](https://www.baeldung.com/cs/attention-luong-vs-bahdanau), [LMU seminar — Attention for NLP](https://slds-lmu.github.io/seminar_nlp_ss20/attention-and-self-attention-for-nlp.html), [Scholarpedia — Echo State Network](http://www.scholarpedia.org/article/Echo_state_network), [Elman 1990 (PDF)](https://gwern.net/doc/ai/nn/rnn/1990-elman.pdf), [Schuster & Paliwal 1997 (PDF)](https://deeplearning.cs.cmu.edu/F20/document/readings/Bidirectional%20Recurrent%20Neural%20Networks.pdf), [Modern Hopfield Networks (arXiv 2502.10122)](https://arxiv.org/html/2502.10122v4), [GeeksforGeeks — Contrastive Divergence in RBMs](https://www.geeksforgeeks.org/deep-learning/contrastive-divergence-in-restricted-boltzmann-machines/), [Tieleman — PCD (PDF)](https://www.cs.toronto.edu/~tijmen/pcd/pcd.pdf), [GM-RKB — GRU](http://www.gabormelli.com/RKB/Gated_Recurrent_Unit_(GRU)), [Brain-inspired DNC (arXiv 2301.02809)](https://arxiv.org/pdf/2301.02809).
# Transformers and Modern Attention Architectures: A Technical Survey

## 1. The Original Transformer — "Attention Is All You Need" (Vaswani et al., 2017)

The Transformer (Vaswani et al., 2017, *Attention Is All You Need*, NeurIPS) dispensed entirely with recurrence and convolutions, relying solely on attention mechanisms to model dependencies between sequence positions. This enabled full parallelization over sequence length during training and became the foundation of virtually all modern large-scale models.

### 1.1 Scaled Dot-Product Attention

Given queries $Q \in \mathbb{R}^{n \times d_k}$, keys $K \in \mathbb{R}^{m \times d_k}$, and values $V \in \mathbb{R}^{m \times d_v}$:

$$\mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V$$

Each query is compared against all keys via dot products; the softmax converts these similarity scores into a probability distribution over positions, which is then used to compute a weighted average of the values. The scaling factor $1/\sqrt{d_k}$ is essential: for large $d_k$, the dot products $q \cdot k = \sum_{i=1}^{d_k} q_i k_i$ grow in magnitude with variance proportional to $d_k$ (assuming unit-variance components), pushing the softmax into regions of extremely small gradients. Dividing by $\sqrt{d_k}$ keeps the logits at unit variance and stabilizes training.

### 1.2 Multi-Head Attention

Rather than a single attention function over $d_{\text{model}}$-dimensional vectors, the Transformer projects $Q$, $K$, $V$ into $h$ lower-dimensional subspaces and applies attention in parallel:

$$\mathrm{MultiHead}(Q, K, V) = \mathrm{Concat}(\mathrm{head}_1, \dots, \mathrm{head}_h)\,W^O$$

$$\mathrm{head}_i = \mathrm{Attention}(QW_i^Q,\; KW_i^K,\; VW_i^V)$$

with learned projections $W_i^Q \in \mathbb{R}^{d_{\text{model}} \times d_k}$, $W_i^K \in \mathbb{R}^{d_{\text{model}} \times d_k}$, $W_i^V \in \mathbb{R}^{d_{\text{model}} \times d_v}$, and $W^O \in \mathbb{R}^{hd_v \times d_{\text{model}}}$. In the base model, $h = 8$ and $d_k = d_v = d_{\text{model}}/h = 64$. Multiple heads allow the model to jointly attend to information from different representation subspaces at different positions — e.g., one head tracking syntactic dependencies, another tracking coreference.

### 1.3 Sinusoidal Positional Encoding

Since attention is permutation-invariant, position information must be injected. The original paper adds fixed sinusoidal encodings to the input embeddings:

$$PE_{(pos, 2i)} = \sin\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right), \qquad PE_{(pos, 2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$

where $pos$ is the position and $i$ indexes the dimension pair. Wavelengths form a geometric progression from $2\pi$ to $10000 \cdot 2\pi$. The key property: for any fixed offset $k$, $PE_{pos+k}$ is a linear function of $PE_{pos}$ (a rotation), allowing the model to learn relative positioning easily.

### 1.4 Encoder–Decoder Architecture, Feed-Forward, Residuals, LayerNorm

- **Encoder**: a stack of $N = 6$ identical layers, each containing (i) multi-head self-attention and (ii) a position-wise feed-forward network, each wrapped in a residual connection followed by layer normalization: $\mathrm{LayerNorm}(x + \mathrm{Sublayer}(x))$ ("post-LN"; modern models typically use pre-LN for stability).
- **Decoder**: also $N = 6$ layers, with three sub-layers: *masked* self-attention (a causal mask sets $-\infty$ on positions $j > i$ before the softmax, preserving the autoregressive property), *cross-attention* where queries come from the decoder and keys/values from the encoder output, and the feed-forward network.
- **Position-wise feed-forward network**, applied identically at each position:

$$\mathrm{FFN}(x) = \max(0,\; xW_1 + b_1)\,W_2 + b_2$$

with inner dimension $d_{ff} = 2048$ (a $4\times$ expansion over $d_{\text{model}} = 512$).

- **Layer normalization** (Ba et al., 2016): $\mathrm{LN}(x) = \gamma \odot \frac{x - \mu}{\sigma} + \beta$, where $\mu, \sigma$ are the mean and standard deviation over the feature dimension.

Self-attention costs $O(n^2 \cdot d)$ per layer in time and $O(n^2)$ in memory — the quadratic bottleneck motivating Section 4.

## 2. Positional Encoding Variants

**Learned absolute embeddings.** A trainable matrix $E_{pos} \in \mathbb{R}^{L_{\max} \times d}$ is added to token embeddings (GPT-2, BERT, ViT). Simple but does not extrapolate beyond $L_{\max}$.

**RoPE — Rotary Position Embedding** (Su et al., 2021, *RoFormer: Enhanced Transformer with Rotary Position Embedding*, arXiv:2104.09864). Instead of adding position vectors, RoPE *rotates* each 2D pair of query/key components by an angle proportional to the position $m$. For dimension pair $i$ with frequency $\theta_i = 10000^{-2i/d}$:

$$f(x, m) = R_{\Theta, m}\, x, \qquad R_{\Theta,m} = \bigoplus_{i=1}^{d/2} \begin{pmatrix} \cos m\theta_i & -\sin m\theta_i \\ \sin m\theta_i & \cos m\theta_i \end{pmatrix}$$

The crucial property is that the attention score depends only on relative position:

$$\langle f(q, m),\, f(k, n) \rangle = \langle R_{\Theta, m} q,\; R_{\Theta, n} k \rangle = q^\top R_{\Theta, n-m}\, k$$

RoPE unifies absolute encoding (applied per position) with relative behavior (in the inner product), and is used in GPT-NeoX, LLaMA, Mistral, Qwen, and most modern LLMs. Long-context extensions (position interpolation, NTK-aware scaling, YaRN) rescale its frequencies.

**ALiBi — Attention with Linear Biases** (Press et al., 2021/2022, *Train Short, Test Long*, ICLR). No embeddings at all; instead a static distance-proportional penalty is added to attention logits:

$$\mathrm{softmax}\!\left(q_i K^\top / \sqrt{d_k} \;+\; m \cdot [-(i-1), \dots, -1, 0]\right)$$

where $m$ is a fixed, head-specific slope (a geometric sequence such as $2^{-8/h}, 2^{-16/h}, \dots$). ALiBi gives strong length extrapolation: models trained on short sequences degrade gracefully at much longer inference lengths (used in BLOOM and MPT).

## 3. The Major Model Families

**BERT** (Devlin et al., 2018, *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*). Encoder-only, bidirectional attention. Pre-trained with **Masked Language Modeling**: 15% of tokens are selected; of these, 80% replaced by `[MASK]`, 10% by a random token, 10% left unchanged, and the model predicts the originals by minimizing cross-entropy over masked positions:

$$\mathcal{L}_{\text{MLM}} = -\mathbb{E}\left[\sum_{i \in \mathcal{M}} \log p_\theta(x_i \mid x_{\setminus \mathcal{M}})\right]$$

plus Next Sentence Prediction (later dropped in RoBERTa, Liu et al., 2019). Ideal for understanding/classification tasks, not generation.

**GPT** (Radford et al., 2018; GPT-2, 2019; GPT-3, Brown et al., 2020, *Language Models are Few-Shot Learners*; GPT-4, OpenAI, 2023). Decoder-only, causal attention. Trained by maximizing the autoregressive log-likelihood:

$$\mathcal{L}(\theta) = \sum_{t=1}^{T} \log p_\theta(x_t \mid x_1, \dots, x_{t-1}), \qquad p_\theta(x) = \prod_{t=1}^{T} p_\theta(x_t \mid x_{<t})$$

GPT-2 (1.5B parameters) demonstrated zero-shot transfer; GPT-3 (175B) established in-context/few-shot learning as an emergent capability of scale; GPT-4 added multimodality and RLHF-refined alignment.

**T5** (Raffel et al., 2020, *Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer*). Full encoder–decoder; every NLP task cast as text-to-text. Pre-trained with span corruption (masking contiguous spans replaced by sentinel tokens); uses relative position biases and RMSNorm-like simplifications.

**LLaMA** (Touvron et al., 2023, LLaMA and LLaMA 2; Meta, 2024, LLaMA 3). The canonical open decoder-only recipe, combining:

- **RMSNorm** (Zhang & Sennrich, 2019), pre-normalization without mean-centering:

$$\mathrm{RMSNorm}(x) = \frac{x}{\mathrm{RMS}(x)} \odot \gamma, \qquad \mathrm{RMS}(x) = \sqrt{\frac{1}{d}\sum_{i=1}^{d} x_i^2 + \epsilon}$$

Cheaper than LayerNorm (no mean subtraction, no bias) with equal or better stability.

- **SwiGLU** feed-forward (Shazeer, 2020, *GLU Variants Improve Transformer*):

$$\mathrm{FFN}_{\text{SwiGLU}}(x) = \left(\mathrm{Swish}_1(xW_1) \otimes xW_3\right)W_2, \qquad \mathrm{Swish}_\beta(x) = x\,\sigma(\beta x)$$

a gated linear unit with SiLU gating and three weight matrices (inner dimension scaled to $\tfrac{2}{3} \cdot 4d$ to keep parameter count constant).

- **RoPE** for positions, and **GQA** (grouped-query attention, Section 4) from LLaMA 2 70B onward.

## 4. Efficient Attention

**Sparse Transformers** (Child et al., 2019, *Generating Long Sequences with Sparse Transformers*). Factorize the full attention matrix into strided and local patterns so each position attends to $O(\sqrt{n})$ others, reducing complexity to $O(n\sqrt{n})$. Precursor to Longformer and BigBird (sliding window + global + random attention).

**Linformer** (Wang et al., 2020). Exploits the empirically low rank of the attention matrix: project keys and values along the sequence axis with learned matrices $E, F \in \mathbb{R}^{k \times n}$, giving $\mathrm{softmax}\big(Q(EK)^\top/\sqrt{d_k}\big)(FV)$ — linear $O(nk)$ complexity.

**Performer** (Choromanski et al., 2020, *Rethinking Attention with Performers*). Approximates the softmax kernel with random features (FAVOR+): $\exp(q^\top k) \approx \phi(q)^\top \phi(k)$ where $\phi$ uses positive orthogonal random features. Attention then factorizes as $\phi(Q)\big(\phi(K)^\top V\big)$, computed in $O(n)$ by changing the multiplication order.

**FlashAttention** (Dao et al., 2022, *FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness*, NeurIPS; FlashAttention-2, 2023; FlashAttention-3, 2024). Not an approximation — an **exact**, IO-aware algorithm. Key insight: the bottleneck is data movement between GPU high-bandwidth memory (HBM) and on-chip SRAM, not FLOPs. FlashAttention (i) **tiles** $Q, K, V$ into blocks that fit in SRAM, (ii) computes the softmax incrementally with the **online softmax** trick (maintaining running max $m$ and normalizer $\ell$ per row, rescaling partial outputs as new blocks arrive), and (iii) never materializes the $n \times n$ attention matrix, **recomputing** it during the backward pass. Memory drops from $O(n^2)$ to $O(n)$, with 2–4× wall-clock speedups; it is now standard in every LLM stack.

**Sliding Window Attention** (Mistral 7B, Jiang et al., 2023). Each token attends only to the previous $W$ tokens ($W = 4096$); with $L$ layers, information still propagates over $L \times W$ positions through the stacked receptive field. Combined with a rolling KV cache of fixed size $W$.

**Multi-Query and Grouped-Query Attention.** **MQA** (Shazeer, 2019, *Fast Transformer Decoding*): all $h$ query heads share a *single* K/V head, shrinking the KV cache by a factor $h$ and dramatically accelerating decoding, at a slight quality cost. **GQA** (Ainslie et al., 2023, *GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints*): interpolates between MHA and MQA by grouping the $h$ query heads into $g$ groups, each sharing one K/V head ($g = h$ recovers MHA, $g = 1$ recovers MQA). LLaMA 2/3 70B use $g = 8$, achieving near-MHA quality at near-MQA speed.

## 5. Mixture of Experts (MoE)

Sparse MoE (Shazeer et al., 2017, *Outrageously Large Neural Networks*) replaces the dense FFN with $E$ expert FFNs plus a learned router; each token activates only $k \ll E$ experts, decoupling parameter count from per-token compute.

**Gating equation.** With router weights $W_g$:

$$G(x) = \mathrm{softmax}\big(\mathrm{TopK}(x \cdot W_g,\; k)\big), \qquad y = \sum_{i \in \mathrm{TopK}} G(x)_i \cdot E_i(x)$$

where $\mathrm{TopK}$ sets non-selected logits to $-\infty$. An auxiliary **load-balancing loss** prevents router collapse:

$$\mathcal{L}_{\text{aux}} = \alpha \cdot E \cdot \sum_{i=1}^{E} f_i \cdot P_i$$

with $f_i$ the fraction of tokens dispatched to expert $i$ and $P_i$ the mean router probability for expert $i$.

**Switch Transformer** (Fedus, Zoph & Shazeer, 2021/2022, JMLR). Simplified to **top-1 routing** ($k = 1$) — each token goes to exactly one expert, with the router probability as multiplicative weight — plus capacity factors and selective precision, scaling stably to 1.6 trillion parameters with 7× pre-training speedup over T5 at equal FLOPs.

**Mixtral 8×7B** (Mistral AI, Jiang et al., 2024). 8 experts per layer, **top-2 routing**. Total 47B parameters but only ~13B active per token; matched or exceeded LLaMA 2 70B and GPT-3.5 on most benchmarks. The same design underlies GPT-4 (reported), DeepSeek-V2/V3 (fine-grained + shared experts), and Gemini 1.5.

## 6. Vision Transformers

**ViT** (Dosovitskiy et al., 2020/2021, *An Image is Worth 16×16 Words*, ICLR). An image $x \in \mathbb{R}^{H \times W \times C}$ is split into $N = HW/P^2$ non-overlapping patches of size $P \times P$ (typically 16×16), each flattened and linearly projected to dimension $D$ by $E \in \mathbb{R}^{(P^2 C) \times D}$. A learnable `[class]` token is prepended and learned position embeddings added:

$$z_0 = [x_{\text{class}};\; x_p^1 E;\; x_p^2 E;\; \dots;\; x_p^N E] + E_{pos}, \qquad E_{pos} \in \mathbb{R}^{(N+1) \times D}$$

Then standard pre-LN Transformer encoder blocks:

$$z'_\ell = \mathrm{MSA}(\mathrm{LN}(z_{\ell-1})) + z_{\ell-1}, \qquad z_\ell = \mathrm{MLP}(\mathrm{LN}(z'_\ell)) + z'_\ell$$

with classification from $\mathrm{LN}(z_L^0)$. ViT lacks convolutional inductive biases (locality, translation equivariance), so it underperforms CNNs on small data but surpasses them when pre-trained on large datasets (JFT-300M).

**DeiT** (Touvron et al., 2021, *Training Data-Efficient Image Transformers & Distillation Through Attention*). Matches ViT quality using ImageNet-1k only, via strong augmentation/regularization and a **distillation token** that learns from a CNN teacher's hard labels through attention.

**Swin Transformer** (Liu et al., 2021, ICCV best paper). Hierarchical ViT for dense prediction: attention computed within non-overlapping local windows ($M \times M = 7 \times 7$ patches), giving **linear** complexity in image size versus ViT's quadratic; **shifted windows** in alternating layers ($\lfloor M/2 \rfloor$ displacement) enable cross-window information flow; patch merging builds a multi-scale feature pyramid usable by detection/segmentation heads. Complexity per window layer: $\Omega(\mathrm{W\text{-}MSA}) = 4hwC^2 + 2M^2hwC$, linear in $hw$.

## 7. Multimodal Models

**CLIP** (Radford et al., 2021, *Learning Transferable Visual Models From Natural Language Supervision*). Dual encoders (image + text) trained on 400M web pairs with a **symmetric InfoNCE contrastive loss**. For a batch of $N$ pairs with L2-normalized embeddings $I_i, T_i$ and learned temperature $\tau$:

$$\mathcal{L} = \frac{1}{2}\left[ -\frac{1}{N}\sum_{i=1}^{N} \log \frac{\exp(I_i \cdot T_i / \tau)}{\sum_{j=1}^{N} \exp(I_i \cdot T_j / \tau)} \;-\; \frac{1}{N}\sum_{i=1}^{N} \log \frac{\exp(I_i \cdot T_i / \tau)}{\sum_{j=1}^{N} \exp(I_j \cdot T_i / \tau)} \right]$$

i.e., cross-entropy over the $N \times N$ cosine-similarity matrix, applied both image→text and text→image. Enables zero-shot classification by embedding class names as prompts ("a photo of a {class}"). SigLIP (Zhai et al., 2023) replaces the softmax with a pairwise sigmoid loss.

**Flamingo** (Alayrac et al., 2022, DeepMind). Bridges a *frozen* vision encoder and a *frozen* LLM (Chinchilla) using a **Perceiver Resampler** (compressing variable visual features into a fixed set of latents) and interleaved **gated cross-attention** layers ($\tanh$-gated, initialized at zero so the LLM starts unperturbed). Handles arbitrarily interleaved image-text sequences; strong few-shot visual learning.

**LLaVA** (Liu et al., 2023, *Visual Instruction Tuning*). Minimalist recipe: CLIP ViT-L/14 features mapped into the LLM (Vicuna) token space by a simple linear projection (an MLP in LLaVA-1.5), then **visual instruction tuning** on GPT-4-generated multimodal conversations. Established the dominant open-source VLM template (adopted conceptually by Qwen-VL, InternVL, etc.).

## 8. State Space Models as an Alternative

SSMs replace attention with a linear dynamical system, offering $O(n)$ scaling and constant-memory recurrent inference.

**Continuous formulation.** A 1D input $u(t)$ maps to output $y(t)$ through a hidden state $h(t) \in \mathbb{R}^N$:

$$h'(t) = A\,h(t) + B\,u(t), \qquad y(t) = C\,h(t) \;(+\, D\,u(t))$$

**Discretization** with step size $\Delta$ via zero-order hold (ZOH):

$$\bar{A} = \exp(\Delta A), \qquad \bar{B} = (\Delta A)^{-1}\big(\exp(\Delta A) - I\big)\,\Delta B$$

$$h_t = \bar{A}\,h_{t-1} + \bar{B}\,u_t, \qquad y_t = C\,h_t$$

**S4** (Gu, Goel & Ré, 2021, *Efficiently Modeling Long Sequences with Structured State Spaces*, ICLR 2022). Uses HiPPO-initialized structured $A$ matrices for long-range memory; because the system is **linear time-invariant (LTI)**, the recurrence unrolls into a convolution $y = u * \bar{K}$ with kernel $\bar{K} = (C\bar{B},\, C\bar{A}\bar{B},\, C\bar{A}^2\bar{B}, \dots)$, computable in $O(n \log n)$ via FFT. Dominated the Long Range Arena benchmark but lagged Transformers on language.

**Mamba** (Gu & Dao, 2023, *Mamba: Linear-Time Sequence Modeling with Selective State Spaces*, arXiv:2312.00752). The **selective SSM (S6)**: makes $B_t$, $C_t$, and $\Delta_t$ **functions of the input** $u_t$ (e.g., $\Delta_t = \mathrm{softplus}(W_\Delta u_t)$), so the model can selectively remember or forget content — recovering a data-dependent gating that LTI SSMs cannot express:

$$h_t = \bar{A}_t\, h_{t-1} + \bar{B}_t\, u_t, \qquad y_t = C_t^\top h_t$$

Input dependence breaks the convolutional shortcut, so Mamba uses a **hardware-aware parallel scan** (associative scan with kernel fusion, keeping states in SRAM — FlashAttention-style IO-awareness). Mamba-3B matched Transformers of twice its size with linear-time training and $O(1)$-memory inference. Mamba-2 (Dao & Gu, 2024) established the SSM–attention duality (SSD); hybrids (Jamba, Zamba, Nemotron-H) interleave Mamba and attention layers.

**RWKV** (Peng et al., 2023, *RWKV: Reinventing RNNs for the Transformer Era*). A linear-attention RNN with channel-wise time decay $w$ — WKV mechanism: $wkv_t = \frac{\sum_{i<t} e^{-(t-1-i)w + k_i} v_i + e^{u+k_t} v_t}{\sum_{i<t} e^{-(t-1-i)w + k_i} + e^{u+k_t}}$ — trainable in parallel like a Transformer, deployable as a pure RNN with constant memory; scaled to 14B+ parameters.

**Hyena** (Poli et al., 2023). Replaces attention with interleaved **implicitly parametrized long convolutions** (filters generated by an MLP over positional encodings) and element-wise multiplicative gating, achieving sub-quadratic $O(n \log n)$ complexity and matching Transformer perplexity at reduced compute; basis of genomic models (HyenaDNA) and Striped Hyena / Evo.

## 9. Scaling Laws

**Kaplan et al., 2020** (*Scaling Laws for Neural Language Models*, OpenAI). Cross-entropy loss follows power laws in parameters $N$, dataset tokens $D$, and compute $C$ over many orders of magnitude:

$$L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}, \quad L(D) = \left(\frac{D_c}{D}\right)^{\alpha_D}, \quad L(C) = \left(\frac{C_c}{C}\right)^{\alpha_C}$$

with $\alpha_N \approx 0.076$, $\alpha_D \approx 0.095$, $\alpha_C \approx 0.050$, and a combined form $L(N, D) = \left[\left(\frac{N_c}{N}\right)^{\alpha_N/\alpha_D} + \frac{D_c}{D}\right]^{\alpha_D}$. Kaplan's prescription — grow $N$ much faster than $D$ ($N \propto C^{0.73}$) — led to under-trained giants like GPT-3 and Gopher.

**Chinchilla — Hoffmann et al., 2022** (*Training Compute-Optimal Large Language Models*, DeepMind). Refit with corrected methodology (three approaches, including IsoFLOP profiles), yielding the parametric loss:

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$$

with fitted values $E \approx 1.69$ (irreducible entropy of text), $A \approx 406.4$, $B \approx 410.7$, $\alpha \approx 0.34$, $\beta \approx 0.28$. Minimizing $L$ subject to $C \approx 6ND$ gives $N_{\text{opt}} \propto C^{a}$, $D_{\text{opt}} \propto C^{b}$ with $a \approx b \approx 0.5$: **parameters and tokens should scale equally**, at roughly **~20 tokens per parameter**. Chinchilla (70B, 1.4T tokens) beat Gopher (280B, 300B tokens) at identical compute. Modern practice (LLaMA 3 trained on 15T tokens) deliberately "over-trains" past Chinchilla-optimal to minimize *inference* cost. Note: Epoch AI's replication (Besiroglu et al., 2024) found minor fitting inconsistencies in Approach 3 but confirmed the ~20 tokens/parameter conclusion.

## 10. Kolmogorov–Arnold Networks (KAN, 2024)

**Theoretical basis.** The Kolmogorov–Arnold representation (superposition) theorem (Kolmogorov, 1957; Arnold): any continuous function $f: [0,1]^n \to \mathbb{R}$ can be written as

$$f(x_1, \dots, x_n) = \sum_{q=1}^{2n+1} \Phi_q\!\left(\sum_{p=1}^{n} \phi_{q,p}(x_p)\right)$$

where $\Phi_q: \mathbb{R} \to \mathbb{R}$ and $\phi_{q,p}: [0,1] \to \mathbb{R}$ are continuous **univariate** functions — multivariate functions decompose into sums and compositions of 1D functions.

**KAN** (Liu et al., 2024, *KAN: Kolmogorov–Arnold Networks*, arXiv:2404.19756; ICLR 2025). Where an MLP layer computes $\sigma(Wx + b)$ — **fixed** activations on nodes, **learnable linear weights** on edges — a KAN layer places a **learnable univariate function on every edge** and simply sums at nodes:

$$x_{l+1, j} = \sum_{i=1}^{n_l} \phi_{l, j, i}(x_{l, i}), \qquad \mathrm{KAN}(x) = (\Phi_{L-1} \circ \cdots \circ \Phi_1 \circ \Phi_0)(x)$$

Each edge function is parametrized as a B-spline plus a residual basis:

$$\phi(x) = w_b\, \mathrm{silu}(x) + w_s \sum_{i} c_i\, B_i(x)$$

with learnable spline coefficients $c_i$ on a grid that can be progressively refined. KANs generalize the depth-2, width-$(2n+1)$ theorem to arbitrary depths and widths (the authors stress it is *inspired by*, not an exact implementation of, the theorem).

**Differences from MLPs.** (i) Learnable activations on edges vs. fixed activations on nodes; (ii) no linear weight matrices — every weight is replaced by a 1D function; (iii) empirically favorable neural scaling on scientific/symbolic-regression tasks, with better accuracy at small scale; (iv) high interpretability — learned splines can be visualized, pruned, and symbolically identified ($\sin$, $x^2$, $\exp$, …), making KANs attractive for physics and "AI for Science"; (v) drawbacks: slower training (spline evaluations parallelize less efficiently than GEMMs) and unproven advantages at LLM scale. Variants include FastKAN (RBFs), Chebyshev-KAN, and KAN 2.0 (Liu et al., 2024).

---

## Key References

1. Vaswani, A. et al. (2017). *Attention Is All You Need*. NeurIPS.
2. Devlin, J. et al. (2018). *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*. NAACL 2019.
3. Radford, A. et al. (2018, 2019); Brown, T. et al. (2020). GPT, GPT-2, *Language Models are Few-Shot Learners* (GPT-3). OpenAI (2023), *GPT-4 Technical Report*.
4. Raffel, C. et al. (2020). *Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer* (T5). JMLR.
5. Zhang, B. & Sennrich, R. (2019). *Root Mean Square Layer Normalization*. NeurIPS.
6. Shazeer, N. (2019). *Fast Transformer Decoding: One Write-Head is All You Need* (MQA); (2020) *GLU Variants Improve Transformer* (SwiGLU).
7. Su, J. et al. (2021). *RoFormer: Enhanced Transformer with Rotary Position Embedding*. arXiv:2104.09864.
8. Press, O., Smith, N. & Lewis, M. (2022). *Train Short, Test Long: Attention with Linear Biases* (ALiBi). ICLR.
9. Child, R. et al. (2019). *Generating Long Sequences with Sparse Transformers*; Wang, S. et al. (2020) *Linformer*; Choromanski, K. et al. (2020) *Performer*.
10. Dao, T. et al. (2022). *FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness*. NeurIPS; Dao (2023) FlashAttention-2.
11. Ainslie, J. et al. (2023). *GQA: Training Generalized Multi-Query Transformer Models*. EMNLP.
12. Shazeer, N. et al. (2017). *Outrageously Large Neural Networks* (sparse MoE); Fedus, W., Zoph, B. & Shazeer, N. (2022). *Switch Transformers*. JMLR; Jiang, A. et al. (2023, 2024). *Mistral 7B*; *Mixtral of Experts*.
13. Touvron, H. et al. (2023). *LLaMA*; *LLaMA 2*; (2021) *DeiT*.
14. Dosovitskiy, A. et al. (2021). *An Image is Worth 16×16 Words* (ViT). ICLR; Liu, Z. et al. (2021). *Swin Transformer*. ICCV.
15. Radford, A. et al. (2021). *Learning Transferable Visual Models From Natural Language Supervision* (CLIP). ICML; Alayrac, J.-B. et al. (2022). *Flamingo*. NeurIPS; Liu, H. et al. (2023). *Visual Instruction Tuning* (LLaVA). NeurIPS.
16. Gu, A., Goel, K. & Ré, C. (2022). *Efficiently Modeling Long Sequences with Structured State Spaces* (S4). ICLR; Gu, A. & Dao, T. (2023). *Mamba*. arXiv:2312.00752; Peng, B. et al. (2023). *RWKV*. EMNLP Findings; Poli, M. et al. (2023). *Hyena Hierarchy*. ICML.
17. Kaplan, J. et al. (2020). *Scaling Laws for Neural Language Models*. arXiv:2001.08361; Hoffmann, J. et al. (2022). *Training Compute-Optimal Large Language Models* (Chinchilla). NeurIPS.
18. Liu, Z. et al. (2024). *KAN: Kolmogorov–Arnold Networks*. arXiv:2404.19756 / ICLR 2025.

Sources consulted during verification: [NeurIPS — Attention Is All You Need](https://papers.neurips.cc/paper/7181-attention-is-all-you-need.pdf), [arXiv 2104.09864 — RoFormer](https://arxiv.org/pdf/2104.09864), [EleutherAI — Rotary Embeddings](https://blog.eleuther.ai/rotary-embeddings/), [arXiv 2312.00752 — Mamba](https://arxiv.org/abs/2312.00752), [A Visual Guide to Mamba](https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-mamba-and-state), [Epoch AI — Chinchilla replication](https://epoch.ai/blog/chinchilla-scaling-a-replication-attempt), [lifearchitect.ai — Chinchilla](https://lifearchitect.ai/chinchilla/), [Wikipedia — Kolmogorov-Arnold Networks](https://en.wikipedia.org/wiki/Kolmogorov-Arnold_Networks), [ICLR 2025 — KAN](https://proceedings.iclr.cc/paper_files/paper/2025/file/afaed89642ea100935e39d39a4da602c-Paper-Conference.pdf), [IBM — Mixture of Experts](https://www.ibm.com/think/topics/mixture-of-experts), [Switch Transformer routing](https://mbrenndoerfer.com/writing/switch-transformer-top-1-routing-trillion-parameter-scaling), [Lil'Log — Contrastive Representation Learning](https://lilianweng.github.io/posts/2021-05-31-contrastive/), [EmergentMind — CLIP](https://www.emergentmind.com/topics/clip), [NeurIPS — FlashAttention](https://proceedings.neurips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html), [AI Summer — ViT](https://theaisummer.com/vision-transformer/), [TinyLlama (LLaMA components)](https://arxiv.org/html/2401.02385v2).
# Generative Neural Networks: A Comprehensive Technical Overview

Generative models learn to represent a data distribution $p_{\text{data}}(x)$ so that new samples can be drawn from it. The major families differ in how they represent the density: explicitly (autoregressive models, normalizing flows), approximately via a variational bound (VAEs), implicitly via a sampling procedure (GANs), through an unnormalized energy (EBMs), or through an iterative denoising process (diffusion and score-based models). This section covers each family with its core architecture, training objective, and canonical equations.

---

## 1. Autoencoders (AE)

An autoencoder (Rumelhart, Hinton & Williams, 1986; popularized for deep learning by Hinton & Salakhutdinov, 2006, "Reducing the Dimensionality of Data with Neural Networks") consists of an **encoder** $f_\phi: \mathcal{X} \to \mathcal{Z}$ mapping an input $x$ to a low-dimensional latent code $z = f_\phi(x)$, and a **decoder** $g_\theta: \mathcal{Z} \to \mathcal{X}$ producing a reconstruction $\hat{x} = g_\theta(z)$. Training minimizes the **reconstruction loss**:

$$\mathcal{L}_{\text{AE}}(\theta, \phi) = \frac{1}{N}\sum_{i=1}^{N} \| x_i - g_\theta(f_\phi(x_i)) \|_2^2$$

(or binary cross-entropy for Bernoulli-modeled pixels). The bottleneck $\dim(z) \ll \dim(x)$ forces the network to learn a compressed representation. A plain AE is *not* a true generative model — its latent space has no imposed prior structure — but it is the conceptual ancestor of the VAE. Key regularized variants:

- **Denoising Autoencoder (DAE)** (Vincent et al., 2008, "Extracting and Composing Robust Features with Denoising Autoencoders"): the input is corrupted, $\tilde{x} \sim C(\tilde{x}|x)$ (e.g., Gaussian noise or masking), and the network must recover the clean input:
$$\mathcal{L}_{\text{DAE}} = \mathbb{E}_{x, \tilde{x}} \left[ \| x - g_\theta(f_\phi(\tilde{x})) \|_2^2 \right]$$
Vincent (2011) showed the DAE implicitly learns the score $\nabla_x \log p(x)$ — a direct precursor of score-based diffusion models.

- **Sparse Autoencoder**: adds an L1 penalty on activations, $\mathcal{L} = \mathcal{L}_{\text{rec}} + \lambda \|z\|_1$, or a KL penalty $\sum_j \mathrm{KL}(\rho \,\|\, \hat{\rho}_j)$ forcing the average activation $\hat{\rho}_j$ of each latent unit toward a small target sparsity $\rho$.

- **Contractive Autoencoder (CAE)** (Rifai et al., 2011): penalizes the Frobenius norm of the encoder's Jacobian to make the representation locally invariant to input perturbations:
$$\mathcal{L}_{\text{CAE}} = \mathcal{L}_{\text{rec}} + \lambda \left\| \frac{\partial f_\phi(x)}{\partial x} \right\|_F^2$$

---

## 2. Variational Autoencoders (VAE)

**Reference:** Kingma & Welling, 2013/2014, "Auto-Encoding Variational Bayes" (ICLR 2014); also Rezende, Mohamed & Wierstra, 2014, "Stochastic Backpropagation and Approximate Inference in Deep Generative Models".

The VAE posits a latent-variable model $p_\theta(x) = \int p_\theta(x|z)\, p(z)\, dz$ with prior $p(z) = \mathcal{N}(0, I)$. The marginal likelihood is intractable, so we introduce an approximate posterior $q_\phi(z|x)$ (the probabilistic encoder).

**ELBO derivation.** Starting from the log-likelihood and inserting $q_\phi$:

$$\log p_\theta(x) = \mathbb{E}_{q_\phi(z|x)}\!\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] + D_{\mathrm{KL}}\!\big(q_\phi(z|x)\,\|\,p_\theta(z|x)\big)$$

Since $D_{\mathrm{KL}} \geq 0$, the first term is the **Evidence Lower BOund (ELBO)**:

$$\log p_\theta(x) \;\geq\; \mathcal{L}_{\text{ELBO}}(\theta, \phi; x) = \underbrace{\mathbb{E}_{q_\phi(z|x)}\left[\log p_\theta(x|z)\right]}_{\text{reconstruction}} \;-\; \underbrace{D_{\mathrm{KL}}\!\big(q_\phi(z|x)\,\|\,p(z)\big)}_{\text{regularization}}$$

The gap between $\log p_\theta(x)$ and the ELBO is exactly $D_{\mathrm{KL}}(q_\phi(z|x)\|p_\theta(z|x))$: maximizing the ELBO simultaneously maximizes the likelihood and tightens the posterior approximation.

**Reparameterization trick.** To backpropagate through the sampling step $z \sim q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \mathrm{diag}(\sigma_\phi^2(x)))$, sampling is rewritten as a deterministic function of the parameters plus exogenous noise:

$$z = \mu_\phi(x) + \sigma_\phi(x) \odot \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, I)$$

This yields a low-variance, unbiased pathwise gradient estimator of the ELBO with respect to $\phi$.

**Closed-form KL for Gaussians** (Appendix B of Kingma & Welling). For $q = \mathcal{N}(\mu, \mathrm{diag}(\sigma^2))$ and $p = \mathcal{N}(0, I)$ in $J$ dimensions:

$$D_{\mathrm{KL}}\big(q_\phi(z|x)\,\|\,\mathcal{N}(0,I)\big) = -\frac{1}{2}\sum_{j=1}^{J}\left(1 + \log \sigma_j^2 - \mu_j^2 - \sigma_j^2\right)$$

**β-VAE** (Higgins et al., 2017, "β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework", ICLR): weights the KL term with $\beta > 1$,

$$\mathcal{L}_{\beta\text{-VAE}} = \mathbb{E}_{q_\phi}[\log p_\theta(x|z)] - \beta \, D_{\mathrm{KL}}(q_\phi(z|x)\,\|\,p(z))$$

which encourages **disentangled** latent factors at the cost of reconstruction fidelity.

**VQ-VAE** (van den Oord, Vinyals & Kavukcuoglu, 2017, "Neural Discrete Representation Learning", NeurIPS): replaces the continuous latent with a **discrete codebook** $\{e_k\}_{k=1}^{K}$. The encoder output $z_e(x)$ is quantized to its nearest code: $z_q(x) = e_k$ with $k = \arg\min_j \|z_e(x) - e_j\|_2$. Since quantization is non-differentiable, gradients are passed to the encoder via the **straight-through estimator** (copying the decoder's gradient past the quantizer). The loss has three terms:

$$\mathcal{L}_{\text{VQ-VAE}} = \underbrace{\|x - D(z_q(x))\|_2^2}_{\text{reconstruction}} + \underbrace{\|\,\mathrm{sg}[z_e(x)] - e\,\|_2^2}_{\text{codebook loss}} + \beta \underbrace{\|\,z_e(x) - \mathrm{sg}[e]\,\|_2^2}_{\text{commitment loss}}$$

where $\mathrm{sg}[\cdot]$ is the stop-gradient operator. A powerful autoregressive prior (PixelCNN, later Transformers) is then fit over the discrete codes — the blueprint for DALL·E 1 and modern latent tokenizers. VQ-VAE-2 (Razavi et al., 2019) added a hierarchical codebook.

---

## 3. Generative Adversarial Networks (GAN)

**Reference:** Goodfellow et al., 2014, "Generative Adversarial Nets" (NeurIPS).

A generator $G(z)$, $z \sim p_z$ (e.g., $\mathcal{N}(0,I)$), and a discriminator $D(x) \in [0,1]$ play a two-player **minimax game**:

$$\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}}\big[\log D(x)\big] + \mathbb{E}_{z \sim p_z}\big[\log\big(1 - D(G(z))\big)\big]$$

For a fixed $G$, the optimal discriminator is $D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}$, and substituting back shows the generator minimizes $2\,\mathrm{JSD}(p_{\text{data}} \| p_g) - \log 4$. The unique **Nash equilibrium** is $p_g = p_{\text{data}}$, at which $D^* \equiv 1/2$ and $V = -\log 4$.

**Non-saturating loss.** Early in training $D$ easily rejects fakes and $\log(1 - D(G(z)))$ saturates (vanishing gradients). Goodfellow proposed instead maximizing $\log D(G(z))$, i.e., the generator minimizes:

$$\mathcal{L}_G^{\text{NS}} = -\mathbb{E}_{z}\big[\log D(G(z))\big], \qquad \mathcal{L}_D = -\mathbb{E}_{x}[\log D(x)] - \mathbb{E}_{z}[\log(1 - D(G(z)))]$$

**Mode collapse** is the classic failure mode: $G$ maps many $z$'s to a few high-scoring outputs, covering only part of $p_{\text{data}}$'s modes. Causes include the JSD's poor behavior on disjoint supports and the alternating-gradient dynamics not converging to the Nash equilibrium.

**Key variants:**

- **DCGAN** (Radford, Metz & Chintala, 2015, "Unsupervised Representation Learning with Deep Convolutional GANs"): architectural recipe — strided/transposed convolutions instead of pooling, batch normalization, ReLU/LeakyReLU, no fully-connected hidden layers — that made GAN training stable on images.

- **cGAN** (Mirza & Osindero, 2014, "Conditional Generative Adversarial Nets"): both networks receive a condition $y$: $\min_G \max_D \; \mathbb{E}_{x,y}[\log D(x|y)] + \mathbb{E}_{z,y}[\log(1 - D(G(z|y)|y))]$.

- **WGAN** (Arjovsky, Chintala & Bottou, 2017, "Wasserstein GAN"): replaces JSD with the **Wasserstein-1 (Earth Mover) distance**, which by Kantorovich–Rubinstein duality is

$$W(p_{\text{data}}, p_g) = \sup_{\|f\|_L \leq 1} \; \mathbb{E}_{x \sim p_{\text{data}}}[f(x)] - \mathbb{E}_{x \sim p_g}[f(x)]$$

The "critic" $f_w$ (no sigmoid) approximates the supremum; the 1-Lipschitz constraint was originally enforced by crude **weight clipping** ($w \leftarrow \mathrm{clip}(w, -c, c)$, e.g., $c = 0.01$). $W$ provides meaningful gradients even for distributions with disjoint supports, greatly reducing mode collapse and correlating with sample quality.

- **WGAN-GP** (Gulrajani et al., 2017, "Improved Training of Wasserstein GANs"): replaces clipping with a **gradient penalty** enforcing $\|\nabla D\| \approx 1$ on interpolates $\hat{x} = \epsilon x + (1-\epsilon)\tilde{x}$, $\epsilon \sim U[0,1]$, $x \sim p_{\text{data}}$, $\tilde{x} = G(z)$:

$$\mathcal{L}_{\text{critic}} = \mathbb{E}_{\tilde{x}}[D(\tilde{x})] - \mathbb{E}_{x}[D(x)] + \lambda \, \mathbb{E}_{\hat{x}}\Big[\big(\|\nabla_{\hat{x}} D(\hat{x})\|_2 - 1\big)^2\Big], \quad \lambda = 10$$

- **StyleGAN 1/2/3** (Karras et al., 2019 "A Style-Based Generator Architecture for GANs"; 2020 "Analyzing and Improving the Image Quality of StyleGAN"; 2021 "Alias-Free GANs"). StyleGAN1: an 8-layer MLP **mapping network** transforms $z \in \mathcal{Z}$ into an intermediate, more disentangled latent $w \in \mathcal{W}$; $w$ modulates each resolution level of the synthesis network via **Adaptive Instance Normalization**:

$$\mathrm{AdaIN}(x_i, y) = y_{s,i} \, \frac{x_i - \mu(x_i)}{\sigma(x_i)} + y_{b,i}$$

where $(y_s, y_b)$ are affine projections of $w$, plus per-pixel noise injection for stochastic detail. StyleGAN2 removed AdaIN's "droplet" artifacts by replacing it with **weight modulation/demodulation** ($w'_{ijk} = s_i \cdot w_{ijk}$, then $w''_{ijk} = w'_{ijk} / \sqrt{\sum_{i,k} {w'_{ijk}}^2 + \epsilon}$), and added path-length regularization. StyleGAN3 fixed **aliasing** ("texture sticking") by treating features as continuous signals with proper low-pass filtering, achieving translation/rotation equivariance.

- **Pix2Pix** (Isola et al., 2017, "Image-to-Image Translation with Conditional Adversarial Networks"): paired image translation with a cGAN plus an L1 term, $\mathcal{L} = \mathcal{L}_{\text{cGAN}} + \lambda \, \mathbb{E}\|y - G(x)\|_1$, using a U-Net generator and PatchGAN discriminator.

- **CycleGAN** (Zhu et al., 2017, "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks"): *unpaired* translation with two generators $G: X \to Y$, $F: Y \to X$ and a **cycle-consistency loss**:

$$\mathcal{L}_{\text{cyc}}(G, F) = \mathbb{E}_{x}\big[\|F(G(x)) - x\|_1\big] + \mathbb{E}_{y}\big[\|G(F(y)) - y\|_1\big]$$

added to the two adversarial losses with weight $\lambda$ (typically 10).

---

## 4. Normalizing Flows

A normalizing flow (Rezende & Mohamed, 2015, "Variational Inference with Normalizing Flows"; earlier Tabak & Vanden-Eijnden, 2010) builds an **invertible**, differentiable map $f: \mathcal{X} \to \mathcal{Z}$ from data to a simple base density $p_Z$ (standard Gaussian). The exact likelihood follows from the **change-of-variables formula**:

$$\log p_X(x) = \log p_Z(f(x)) + \log \left| \det \frac{\partial f(x)}{\partial x} \right|$$

For a composition $f = f_K \circ \cdots \circ f_1$, the log-determinants add: $\log p_X(x) = \log p_Z(z_K) + \sum_{k=1}^{K} \log |\det J_{f_k}|$. Training maximizes exact log-likelihood; sampling inverts the flow: $x = f^{-1}(z)$, $z \sim p_Z$. The design challenge is making $\det J$ computable in $O(D)$ instead of $O(D^3)$.

- **RealNVP** (Dinh, Sohl-Dickstein & Bengio, 2016, "Density Estimation Using Real NVP"): **affine coupling layers**. Split $x$ into $(x_{1:d}, x_{d+1:D})$:

$$y_{1:d} = x_{1:d}, \qquad y_{d+1:D} = x_{d+1:D} \odot \exp\big(s(x_{1:d})\big) + t(x_{1:d})$$

where $s, t$ are arbitrary neural networks (never inverted). The Jacobian is lower triangular, so $\log|\det J| = \sum_j s(x_{1:d})_j$, and inversion is trivial: $x_{d+1:D} = (y_{d+1:D} - t) \odot \exp(-s)$. Alternating masks and multi-scale squeezing give expressivity.

- **Glow** (Kingma & Dhariwal, 2018, "Glow: Generative Flow with Invertible 1×1 Convolutions", NeurIPS): each step = **actnorm** (per-channel affine) → **invertible 1×1 convolution** (a learned, LU-decomposed permutation generalization, with $\log|\det| = H \cdot W \cdot \log|\det W_{1\times1}|$) → affine coupling. Produced the first high-quality flow-based face samples and smooth latent interpolations.

- **Autoregressive flows.** **MAF** (Papamakarios, Pavlakou & Murray, 2017, "Masked Autoregressive Flow for Density Estimation") uses $x_i = z_i \sigma_i(x_{1:i-1}) + \mu_i(x_{1:i-1})$: density evaluation is one parallel pass (fast training), but sampling is sequential. **IAF** (Kingma et al., 2016, "Improved Variational Inference with Inverse Autoregressive Flow") inverts the conditioning — $x_i = z_i \sigma_i(z_{1:i-1}) + \mu_i(z_{1:i-1})$ — making *sampling* parallel and density evaluation sequential; ideal as a flexible VAE posterior. Both are triangular-Jacobian flows: $\log|\det J| = \sum_i \log \sigma_i$.

---

## 5. Diffusion Models

### 5.1 DDPM

**Reference:** Ho, Jain & Abbeel, 2020, "Denoising Diffusion Probabilistic Models" (NeurIPS); building on Sohl-Dickstein et al., 2015, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics".

**Forward (diffusion) process** — a fixed Markov chain gradually adding Gaussian noise over $T$ steps (typically $T = 1000$) with variance schedule $\beta_1, \dots, \beta_T$:

$$q(x_t \mid x_{t-1}) = \mathcal{N}\big(x_t;\; \sqrt{1 - \beta_t}\, x_{t-1},\; \beta_t I\big), \qquad q(x_{1:T}|x_0) = \prod_{t=1}^{T} q(x_t|x_{t-1})$$

With $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$, one can jump directly to any $t$ (the key computational trick):

$$q(x_t \mid x_0) = \mathcal{N}\big(x_t;\; \sqrt{\bar{\alpha}_t}\, x_0,\; (1 - \bar{\alpha}_t) I\big) \quad\Longleftrightarrow\quad x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \varepsilon,\;\; \varepsilon \sim \mathcal{N}(0, I)$$

**Reverse (generative) process** — a learned Markov chain starting from $p(x_T) = \mathcal{N}(0, I)$:

$$p_\theta(x_{t-1} \mid x_t) = \mathcal{N}\big(x_{t-1};\; \mu_\theta(x_t, t),\; \sigma_t^2 I\big)$$

The true posterior $q(x_{t-1}|x_t, x_0)$ is a tractable Gaussian with mean $\tilde{\mu}_t(x_t, x_0) = \frac{\sqrt{\bar\alpha_{t-1}}\beta_t}{1-\bar\alpha_t}x_0 + \frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}x_t$ and variance $\tilde{\beta}_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t$. Parameterizing the model to predict the noise $\varepsilon$ instead of the mean,

$$\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}}\, \varepsilon_\theta(x_t, t)\right)$$

the variational bound reduces (dropping time-dependent weights) to the remarkably **simple loss**:

$$\mathcal{L}_{\text{simple}} = \mathbb{E}_{t \sim U[1,T],\, x_0,\, \varepsilon \sim \mathcal{N}(0,I)} \Big[ \big\| \varepsilon - \varepsilon_\theta\big(\sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \varepsilon,\; t\big) \big\|^2 \Big]$$

— i.e., train a U-Net to predict the added noise, at a random timestep, in one step. Ho et al. used a **linear schedule** ($\beta_1 = 10^{-4}$ to $\beta_T = 0.02$); Nichol & Dhariwal (2021, "Improved DDPM") proposed the **cosine schedule** $\bar\alpha_t = \cos^2\!\big(\frac{t/T + s}{1+s}\cdot\frac{\pi}{2}\big)$ and learned variances.

### 5.2 DDIM

Song, Meng & Ermon, 2020, "Denoising Diffusion Implicit Models" (ICLR 2021): defines a family of **non-Markovian** processes sharing DDPM's marginals (so the same trained $\varepsilon_\theta$ works). The update

$$x_{t-1} = \sqrt{\bar{\alpha}_{t-1}} \underbrace{\left(\frac{x_t - \sqrt{1 - \bar{\alpha}_t}\, \varepsilon_\theta(x_t, t)}{\sqrt{\bar{\alpha}_t}}\right)}_{\text{predicted } x_0} + \sqrt{1 - \bar{\alpha}_{t-1} - \sigma_t^2}\; \varepsilon_\theta(x_t, t) + \sigma_t \varepsilon_t$$

with $\sigma_t = 0$ gives a **deterministic** sampler (a probability-flow ODE discretization), enabling 10–50 step sampling instead of 1000 and semantically meaningful latent interpolation/inversion.

### 5.3 Score-based models and the SDE formulation

Song & Ermon, 2019, "Generative Modeling by Estimating Gradients of the Data Distribution" (NeurIPS): learn the **score function** $s_\theta(x) \approx \nabla_x \log p(x)$ via denoising score matching at multiple noise levels $\{\sigma_i\}$:

$$\mathcal{L} = \frac{1}{L}\sum_{i=1}^{L} \lambda(\sigma_i)\, \mathbb{E}_{x, \tilde{x} \sim \mathcal{N}(x, \sigma_i^2 I)} \left[ \left\| s_\theta(\tilde{x}, \sigma_i) + \frac{\tilde{x} - x}{\sigma_i^2} \right\|^2 \right]$$

and sample with **annealed Langevin dynamics**: $x_{k+1} = x_k + \frac{\eta}{2} s_\theta(x_k, \sigma) + \sqrt{\eta}\, \varepsilon_k$.

Song et al., 2021, "Score-Based Generative Modeling through Stochastic Differential Equations" (ICLR, oral) unified DDPM and score matching in continuous time. Forward SDE: $dx = f(x, t)\, dt + g(t)\, dw$. By Anderson (1982), the **reverse-time SDE** is

$$dx = \big[f(x, t) - g(t)^2\, \nabla_x \log p_t(x)\big]\, dt + g(t)\, d\bar{w}$$

DDPM corresponds to a variance-preserving SDE; NCSN to variance-exploding. There is also a deterministic **probability-flow ODE**, $dx = [f(x,t) - \tfrac{1}{2}g(t)^2 \nabla_x \log p_t(x)]\,dt$, with the same marginals — the basis for DDIM-style samplers and exact likelihoods. Note the identity $\nabla_{x_t} \log p(x_t) = -\varepsilon_\theta(x_t, t)/\sqrt{1 - \bar\alpha_t}$: noise prediction *is* score estimation.

### 5.4 Guidance

**Classifier guidance** (Dhariwal & Nichol, 2021, "Diffusion Models Beat GANs on Image Synthesis") shifts the score by $\nabla_{x_t} \log p_\phi(y|x_t)$ from an external classifier. **Classifier-free guidance** (Ho & Salimans, 2021/2022, "Classifier-Free Diffusion Guidance", NeurIPS workshop) instead trains one network with the condition randomly dropped ($y \to \varnothing$ with ~10% probability), then extrapolates at sampling time:

$$\tilde{\varepsilon}_\theta(x_t, y) = (1 + w)\, \varepsilon_\theta(x_t, y) - w\, \varepsilon_\theta(x_t, \varnothing)$$

(equivalently $\varepsilon_\theta(x_t,\varnothing) + s\,[\varepsilon_\theta(x_t,y) - \varepsilon_\theta(x_t,\varnothing)]$ with $s = 1 + w$). Larger $w$ trades diversity for fidelity/prompt-adherence; CFG is the workhorse of all modern text-to-image systems.

### 5.5 Latent diffusion / Stable Diffusion

Rombach et al., 2022, "High-Resolution Image Synthesis with Latent Diffusion Models" (CVPR): run diffusion not in pixel space but in the **latent space of a pretrained perceptual autoencoder** (KL- or VQ-regularized, ~8× spatial downsampling), slashing compute. Conditioning (text via a frozen CLIP encoder in Stable Diffusion) enters the denoising U-Net through **cross-attention**: $\mathrm{Attention}(Q, K, V) = \mathrm{softmax}(QK^\top/\sqrt{d})V$ with $Q$ from image features and $K, V$ from text embeddings. Loss: $\mathcal{L}_{\text{LDM}} = \mathbb{E}_{\mathcal{E}(x), \varepsilon, t}\big[\|\varepsilon - \varepsilon_\theta(z_t, t, \tau_\theta(y))\|^2\big]$.

### 5.6 Flow Matching

Lipman et al., 2023, "Flow Matching for Generative Modeling" (ICLR); concurrently Liu et al., 2022 ("Rectified Flow") and Albergo & Vanden-Eijnden, 2022. Instead of learning a score for an SDE, directly regress a **velocity field** $v_\theta(x, t)$ of an ODE $\frac{dx}{dt} = v_\theta(x, t)$ transporting noise $p_0 = \mathcal{N}(0,I)$ to data $p_1$. The marginal objective is intractable, but the **Conditional Flow Matching** loss has identical gradients ($\nabla_\theta \mathcal{L}_{\text{CFM}} = \nabla_\theta \mathcal{L}_{\text{FM}}$). With the simplest (optimal-transport / linear) path $x_t = (1-t)x_0 + t\,x_1$:

$$\mathcal{L}_{\text{CFM}}(\theta) = \mathbb{E}_{t \sim U[0,1],\, x_0 \sim p_0,\, x_1 \sim p_{\text{data}}} \Big[ \big\| v_\theta\big(x_t, t\big) - (x_1 - x_0) \big\|^2 \Big]$$

Straight probability paths yield fewer ODE steps and simpler training; flow matching underlies Stable Diffusion 3, Flux, and Meta's Movie Gen.

---

## 6. Autoregressive Models

These models use the **chain-rule factorization** of the joint density — exact likelihood, sequential sampling:

$$p(x) = \prod_{i=1}^{n} p\big(x_i \mid x_1, \dots, x_{i-1}\big)$$

- **PixelRNN / PixelCNN** (van den Oord, Kalchbrenner & Kavukcuoglu, 2016, "Pixel Recurrent Neural Networks", ICML best paper; and "Conditional Image Generation with PixelCNN Decoders", NeurIPS 2016): model images pixel by pixel in raster order, each pixel's 256-way (or mixture-of-logistics, PixelCNN++, Salimans et al. 2017) distribution conditioned on all previous pixels. PixelRNN uses row/diagonal LSTMs; PixelCNN uses **masked convolutions** (type A/B masks) for parallel training; gated PixelCNN fixes the blind spot with vertical + horizontal stacks.

- **WaveNet** (van den Oord et al., 2016, "WaveNet: A Generative Model for Raw Audio"): the same factorization on raw audio samples, using stacks of **dilated causal convolutions** (dilations 1, 2, 4, …, 512, repeated) so the receptive field grows exponentially with depth, gated activations $z = \tanh(W_f * x) \odot \sigma(W_g * x)$, residual/skip connections, and 8-bit μ-law quantized outputs via softmax.

This paradigm — next-token prediction — scaled up over discrete tokens is precisely what powers GPT-style LLMs, and, combined with VQ tokenizers, image generators like DALL·E 1 and Parti (Yu et al., 2022).

---

## 7. Energy-Based Models (EBM)

**References:** LeCun et al., 2006, "A Tutorial on Energy-Based Learning"; Du & Mordatch, 2019, "Implicit Generation and Modeling with Energy-Based Models". An EBM defines a Boltzmann/Gibbs density via a scalar energy network $E_\theta$:

$$p_\theta(x) = \frac{e^{-E_\theta(x)}}{Z(\theta)}, \qquad Z(\theta) = \int e^{-E_\theta(x)}\, dx$$

The partition function $Z$ is intractable, so maximum-likelihood gradients use the contrastive identity

$$\nabla_\theta \log p_\theta(x) = -\nabla_\theta E_\theta(x) + \mathbb{E}_{x' \sim p_\theta}\big[\nabla_\theta E_\theta(x')\big]$$

with negative samples $x'$ drawn by MCMC — typically **Langevin dynamics**, $x_{k+1} = x_k - \frac{\eta}{2}\nabla_x E_\theta(x_k) + \sqrt{\eta}\,\varepsilon_k$ (note $\nabla_x \log p_\theta(x) = -\nabla_x E_\theta(x)$: EBMs and score-based models are two views of the same object). Historical instances: Boltzmann machines and RBMs (Hinton) trained with contrastive divergence. EBMs are flexible (any architecture, easy composition of constraints) but slow to sample.

---

## 8. Recent Landmark Systems

- **DALL·E** (Ramesh et al., 2021, "Zero-Shot Text-to-Image Generation"): a 12B discrete-VAE + autoregressive Transformer over text-and-image tokens. **DALL·E 2 / unCLIP** (Ramesh et al., 2022, "Hierarchical Text-Conditional Image Generation with CLIP Latents"): a diffusion *prior* maps text to CLIP image embeddings, then a diffusion decoder generates the image. DALL·E 3 (2023) emphasized recaptioned training data for prompt fidelity.

- **Imagen** (Saharia et al., 2022, "Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding", NeurIPS): a frozen large language model (T5-XXL) as text encoder + a cascade of pixel-space diffusion models (64² → 256² → 1024²), with **dynamic thresholding** to allow high guidance weights. Key finding: scaling the *text encoder* matters more than scaling the image U-Net.

- **Diffusion Transformer (DiT)** (Peebles & Xie, 2022/2023, "Scalable Diffusion Models with Transformers", ICCV): replaces the U-Net with a ViT over latent patches, conditioned via **adaLN-Zero** (adaptive layer norm whose scale/shift/gating come from timestep + class embeddings). FID scales smoothly with compute (Gflops), establishing transformers as the diffusion backbone.

- **Sora** (OpenAI, 2024, technical report "Video Generation Models as World Simulators"; Sora 2 in 2025): a latent **diffusion transformer** over **spacetime patches** — video is compressed by a video autoencoder into a spatiotemporal latent, cut into patch tokens, and denoised by a scaled DiT — enabling variable durations, resolutions, and aspect ratios, with emergent 3D consistency and object permanence. The same DiT recipe underlies Stable Diffusion 3 (Esser et al., 2024, MM-DiT + rectified flow) and most 2024–2026 video models.

- **Consistency Models** (Song, Dhariwal, Chen & Sutskever, 2023, "Consistency Models", ICML): learn a function $f_\theta(x_t, t)$ that maps *any* point on a probability-flow ODE trajectory directly to its origin, enforcing **self-consistency** $f_\theta(x_t, t) = f_\theta(x_{t'}, t')$ for all $t, t'$ on the same trajectory, with boundary condition $f_\theta(x_\epsilon, \epsilon) = x_\epsilon$. Trained by distilling a diffusion teacher (consistency distillation: $\mathcal{L} = \mathbb{E}[\,d(f_\theta(x_{t_{n+1}}, t_{n+1}), f_{\theta^-}(\hat{x}_{t_n}, t_n))\,]$ with an EMA target $\theta^-$) or standalone (consistency training). Result: **one-step generation** with quality approaching multi-step diffusion; successors include Latent Consistency Models (Luo et al., 2023) and sCM (Lu & Song, 2024).

---

## Summary Comparison

| Family | Density | Sampling | Training | Weakness |
|---|---|---|---|---|
| VAE | Lower bound (ELBO) | 1 pass, fast | Stable | Blurry samples |
| GAN | Implicit | 1 pass, fast | Unstable (Nash) | Mode collapse |
| Flows | Exact | Fast | Stable (MLE) | Invertibility constrains architecture |
| Autoregressive | Exact | Slow (sequential) | Stable (MLE) | No latent; slow generation |
| EBM | Unnormalized | Slow (MCMC) | Contrastive, tricky | Intractable $Z$ |
| Diffusion / FM | Bound / exact (ODE) | Many steps (→1 with consistency) | Very stable | Sampling cost |

Diffusion (increasingly in its flow-matching formulation on transformer backbones) dominates image/video/audio generation as of 2026, while autoregressive transformers dominate text — and hybrids of the two (e.g., diffusion heads on AR backbones, AR-over-VQ-latents) are an active frontier.

**Sources:**
- [DDPM theory (LearnOpenCV)](https://learnopencv.com/denoising-diffusion-probabilistic-models/), [Improved DDPM (Nichol & Dhariwal)](https://arxiv.org/pdf/2102.09672), [Lecture Notes in Probabilistic Diffusion Models](https://arxiv.org/pdf/2312.10393)
- [Kingma & Welling VAE slides](https://berkeley-deep-learning.github.io/cs294-131-s17/slides/VAE%20talk.compressed.pdf), [ELBO derivation tutorial](https://arxiv.org/pdf/1907.08956), [Reparameterization trick (Gundersen)](https://gregorygundersen.com/blog/2018/04/29/reparameterization/)
- [WGAN-GP overview (EmergentMind)](https://www.emergentmind.com/topics/wasserstein-gan-loss-with-gradient-penalty-wgan-gp), [Gradient penalty analysis](https://arxiv.org/pdf/1910.06922)
- [VQ-VAE explained](https://leeyngdo.github.io/blog/generative-model/2023-09-02-VQ-VAE/), [VQ-VAE-2 paper](http://papers.neurips.cc/paper/9625-generating-diverse-high-fidelity-images-with-vq-vae-2.pdf), [HuggingFace VQ post](https://huggingface.co/blog/ariG23498/understand-vq)
- [Classifier-Free Diffusion Guidance (arXiv 2207.12598)](https://arxiv.org/abs/2207.12598), [CFG notes (Khungurn)](https://pkhungurn.github.io/notes/notes/ml/ddpm-classifier-free-guidance/ddpm-classifier-free-guidance.pdf)
- [StyleGAN2 paper (Karras et al.)](https://users.aalto.fi/~laines9/publications/karras2020cvpr_paper.pdf), [StyleGAN3 project page](https://nvlabs.github.io/stylegan3/), [StyleGAN3 explained](https://medium.com/@steinsfu/stylegan3-clearly-explained-793edbcc8048)
- [Score SDE (arXiv 2011.13456)](https://arxiv.org/abs/2011.13456), [score_sde repo](https://github.com/yang-song/score_sde)
- [Consistency Models (arXiv 2303.01469)](https://arxiv.org/abs/2303.01469), [ICML PMLR version](https://proceedings.mlr.press/v202/song23a.html), [openai/consistency_models](https://github.com/openai/consistency_models)
- Flow matching: [closed-form analysis (OpenReview)](https://openreview.net/pdf?id=kVz9uvqUna), [CFM for Bayesian inference](https://arxiv.org/pdf/2510.09534)
# Specialized and Emerging Neural Network Architectures

## 1. Graph Neural Networks (GNNs)

### 1.1 The message-passing framework

Most GNNs are instances of the **Message Passing Neural Network (MPNN)** formalism unified by Gilmer et al. (2017), *Neural Message Passing for Quantum Chemistry*, ICML 2017 (arXiv:1704.01212). Given a graph $G=(V,E)$ with node features $x_v$ and edge features $e_{vw}$, the forward pass runs $T$ propagation steps:

$$m_v^{(t+1)} = \sum_{w \in \mathcal{N}(v)} M_t\!\left(h_v^{(t)}, h_w^{(t)}, e_{vw}\right), \qquad h_v^{(t+1)} = U_t\!\left(h_v^{(t)}, m_v^{(t+1)}\right)$$

where $M_t$ is a learned **message function**, $U_t$ a learned **update function** (Gilmer et al. used a GRU), and $\mathcal{N}(v)$ the neighborhood of $v$. A permutation-invariant **readout** produces a graph-level embedding:

$$\hat{y} = R\left(\{h_v^{(T)} \mid v \in V\}\right)$$

The generic modern form separates *aggregation* from *combination*:

$$a_v^{(k)} = \operatorname{AGGREGATE}^{(k)}\!\left(\{h_u^{(k-1)} : u \in \mathcal{N}(v)\}\right), \qquad h_v^{(k)} = \operatorname{COMBINE}^{(k)}\!\left(h_v^{(k-1)}, a_v^{(k)}\right)$$

The aggregator must be permutation-invariant (sum, mean, max, attention-weighted sum). This is the single equation from which GCN, GraphSAGE, GAT and GIN are all special cases.

### 1.2 Graph Convolutional Networks (GCN)

Kipf & Welling (2017), *Semi-Supervised Classification with Graph Convolutional Networks*, ICLR 2017 (arXiv:1609.02907), derived a first-order localized spectral filter. Starting from spectral graph convolution $g_\theta \star x = U g_\theta(\Lambda) U^\top x$ with $L = I_N - D^{-1/2}AD^{-1/2} = U\Lambda U^\top$, they applied a first-order Chebyshev truncation and a renormalization trick, yielding the celebrated layer-wise propagation rule:

$$H^{(l+1)} = \sigma\!\left(\tilde{D}^{-\frac{1}{2}}\,\tilde{A}\,\tilde{D}^{-\frac{1}{2}}\,H^{(l)}\,W^{(l)}\right)$$

with $\tilde{A} = A + I_N$ (adjacency with self-loops), $\tilde{D}_{ii} = \sum_j \tilde{A}_{ij}$, $H^{(0)} = X$, $W^{(l)}$ the trainable weight matrix of layer $l$, and $\sigma$ typically ReLU. The **renormalization trick** ($A \to A+I$ before normalization) keeps the eigenvalues of the propagation operator bounded, preventing exploding/vanishing gradients in deep stacks. In node form:

$$h_v^{(l+1)} = \sigma\!\left(\sum_{u \in \mathcal{N}(v)\cup\{v\}} \frac{1}{\sqrt{\tilde{d}_v \tilde{d}_u}} W^{(l)} h_u^{(l)}\right)$$

The symmetric normalization $D^{-1/2}\tilde{A}D^{-1/2}$ is preferred over random-walk normalization $D^{-1}\tilde{A}$ because it keeps the operator symmetric (real spectrum) and downweights messages from high-degree hubs on both endpoints. A two-layer GCN for semi-supervised classification reads $Z = \operatorname{softmax}\!\left(\hat{A}\,\operatorname{ReLU}(\hat{A}XW^{(0)})\,W^{(1)}\right)$ with $\hat{A} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}$.

**Limitation:** GCN is transductive (needs the full adjacency at training time) and suffers **over-smoothing** — as depth grows, node representations converge to a degree-dependent stationary vector, since repeated multiplication by $\hat{A}$ is a low-pass filter.

### 1.3 GraphSAGE

Hamilton, Ying & Leskovec (2017), *Inductive Representation Learning on Large Graphs*, NeurIPS 2017 (arXiv:1706.02216), made GNNs **inductive** by learning aggregator functions over *sampled* fixed-size neighborhoods rather than the full graph:

$$h_{\mathcal{N}(v)}^{(k)} = \operatorname{AGGREGATE}_k\!\left(\{h_u^{(k-1)}, \forall u \in \mathcal{N}(v)\}\right)$$
$$h_v^{(k)} = \sigma\!\left(W^{(k)} \cdot \operatorname{CONCAT}\!\left(h_v^{(k-1)},\, h_{\mathcal{N}(v)}^{(k)}\right)\right), \qquad h_v^{(k)} \leftarrow \frac{h_v^{(k)}}{\|h_v^{(k)}\|_2}$$

The concatenation (rather than summation) of the self-vector with the neighborhood vector is a "skip connection" that preserves the node's own identity. Three aggregators were proposed:

- **Mean:** $h_v^{(k)} = \sigma\!\left(W \cdot \operatorname{mean}(\{h_v^{(k-1)}\} \cup \{h_u^{(k-1)}, u \in \mathcal{N}(v)\})\right)$ — the "GCN-inductive" variant.
- **LSTM:** apply an LSTM to a random permutation of the neighbors (not permutation-invariant, but expressive).
- **Max-pooling:** $\operatorname{AGGREGATE}^{\text{pool}} = \max\!\left(\{\sigma(W_{\text{pool}} h_u^{(k-1)} + b),\ \forall u \in \mathcal{N}(v)\}\right)$, element-wise max.

Unsupervised training uses a graph-based loss with negative sampling: $J(z_u) = -\log\!\left(\sigma(z_u^\top z_v)\right) - Q\cdot\mathbb{E}_{v_n \sim P_n(v)}\log\!\left(\sigma(-z_u^\top z_{v_n})\right)$, where $v$ co-occurs with $u$ on a fixed-length random walk.

### 1.4 Graph Attention Networks (GAT)

Veličković et al. (2018), *Graph Attention Networks*, ICLR 2018 (arXiv:1710.10903), replaced the fixed structural coefficient $1/\sqrt{\tilde d_v \tilde d_u}$ by learned attention. Unnormalized attention logits:

$$e_{ij} = \operatorname{LeakyReLU}\!\left(\vec{\mathbf{a}}^{\top}\left[\mathbf{W}\vec{h}_i \,\|\, \mathbf{W}\vec{h}_j\right]\right)$$

with $\|$ concatenation, $\mathbf{W} \in \mathbb{R}^{F' \times F}$ shared, $\vec{\mathbf{a}} \in \mathbb{R}^{2F'}$ the attention vector, and LeakyReLU slope $0.2$. Normalization is a masked softmax over the first-order neighborhood (including $i$ itself):

$$\alpha_{ij} = \operatorname{softmax}_j(e_{ij}) = \frac{\exp(e_{ij})}{\sum_{k \in \mathcal{N}_i} \exp(e_{ik})}$$

Node update, and its multi-head ($K$ heads) versions:

$$\vec{h}_i' = \sigma\!\left(\sum_{j \in \mathcal{N}_i} \alpha_{ij}\,\mathbf{W}\vec{h}_j\right), \qquad
\vec{h}_i' = \Big\|_{k=1}^{K} \sigma\!\left(\sum_{j\in\mathcal{N}_i}\alpha_{ij}^k \mathbf{W}^k \vec{h}_j\right)$$

and on the final (prediction) layer, heads are **averaged** rather than concatenated: $\vec{h}_i' = \sigma\!\left(\frac{1}{K}\sum_{k=1}^{K}\sum_{j\in\mathcal{N}_i}\alpha_{ij}^k \mathbf{W}^k \vec{h}_j\right)$.

GAT is inductive, does not require knowing the graph structure upfront, and assigns different importances to neighbors of the same node. Brody, Alon & Yahav (2022), *How Attentive are Graph Attention Networks?* (**GATv2**, ICLR 2022), showed GAT computes only *static* attention (the ranking of neighbors is shared across query nodes) and fixed it by moving the nonlinearity inside: $e_{ij} = \vec{\mathbf{a}}^\top \operatorname{LeakyReLU}\!\left(\mathbf{W}[\vec{h}_i \| \vec{h}_j]\right)$.

### 1.5 Graph Isomorphism Network (GIN)

Xu, Hu, Leskovec & Jegelka (2019), *How Powerful are Graph Neural Networks?*, ICLR 2019 (arXiv:1810.00826), proved that a message-passing GNN is **at most as discriminative as the 1-dimensional Weisfeiler-Lehman (1-WL) test**, and that this bound is attained iff the aggregation and readout functions are injective on multisets. Since sum is injective over multisets (mean and max are not), they proposed:

$$h_v^{(k)} = \operatorname{MLP}^{(k)}\!\left(\left(1 + \epsilon^{(k)}\right)\cdot h_v^{(k-1)} + \sum_{u\in\mathcal{N}(v)} h_u^{(k-1)}\right)$$

with $\epsilon^{(k)}$ a learnable or fixed scalar (GIN-$\epsilon$ vs GIN-0), which disambiguates the node's own features from its neighbors' sum. For graph-level tasks, they concatenate summed readouts across all depths to preserve both local and global structure:

$$h_G = \operatorname{CONCAT}\!\left(\operatorname{READOUT}\left(\{h_v^{(k)} \mid v \in G\}\right) \ \Big|\ k = 0,1,\ldots,K\right)$$

Mean aggregation fails to distinguish multisets with identical distributions but different multiplicities; max aggregation fails to distinguish multisets with identical support. This ordering (sum > mean > max in expressive power) is the central theoretical result of the paper.

### 1.6 Applications

- **AlphaFold 2** (Jumper et al., *Nature* 2021, "Highly accurate protein structure prediction with AlphaFold"): the Evoformer treats the pair representation as a fully-connected weighted graph over residues and enforces the **triangle inequality** through triangular multiplicative updates and triangular self-attention — a geometric, graph-structured attention mechanism. The Structure Module then performs Invariant Point Attention (IPA) in SE(3).
- **Molecular property prediction:** MPNN/GIN on QM9, ChEMBL; Stokes et al. (*Cell* 2020) discovered the antibiotic **halicin** with a directed-MPNN (Chemprop).
- **Recommender systems:** PinSAGE (Ying et al., KDD 2018), a GraphSAGE variant deployed on Pinterest's 3-billion-node graph.
- **Physics simulation:** Sanchez-Gonzalez et al. (ICML 2020), *Learning to Simulate Complex Physics with Graph Networks*.
- **Combinatorial optimization, traffic forecasting** (Google Maps ETA), **fraud detection**, **weather** (GraphCast, Lam et al., *Science* 2023).

---

## 2. Spiking Neural Networks (SNNs)

SNNs are the "third generation" of neural networks (Maass, 1997, *Networks of spiking neurons: The third generation of neural network models*, Neural Networks 10(9)). Information is carried by discrete, asynchronous events (spikes) in continuous time rather than by real-valued activations.

### 2.1 Leaky Integrate-and-Fire (LIF)

The workhorse model, tracing back to Lapicque (1907):

$$\tau_m \frac{dV(t)}{dt} = -\left(V(t) - V_{\text{rest}}\right) + R_m I(t)$$

with $\tau_m = R_m C_m$ the membrane time constant (typically 10–30 ms in cortex). When $V(t)$ crosses threshold $V_{\text{th}}$, a spike is emitted and the potential is reset: $V \leftarrow V_{\text{reset}}$, followed by an absolute refractory period $t_{\text{ref}}$. The equivalent circuit is a leaky RC compartment. The discrete-time form used in deep SNN frameworks (snnTorch, SpikingJelly, Norse):

$$V[t] = \beta V[t-1] + \sum_j W_j S_j[t] - S[t-1]\,V_{\text{th}}, \qquad S[t] = \Theta\!\left(V[t] - V_{\text{th}}\right)$$

with decay $\beta = e^{-\Delta t/\tau_m}$ and $\Theta$ the Heaviside step. The last term implements **soft reset** (subtraction) versus **hard reset** ($V \leftarrow 0$).

Because $\Theta'$ is a Dirac delta, gradient descent requires a **surrogate gradient** (Neftci, Mostafa & Zenke, 2019, *Surrogate Gradient Learning in Spiking Neural Networks*, IEEE Signal Processing Magazine), e.g. the fast sigmoid derivative $\frac{\partial S}{\partial V} \approx \frac{1}{(1 + \gamma|V - V_{\text{th}}|)^2}$ or an arctan/box surrogate. This enables BPTT training of deep SNNs.

### 2.2 Hodgkin–Huxley

Hodgkin & Huxley (1952), *A quantitative description of membrane current and its application to conduction and excitation in nerve*, J. Physiol. 117:500–544 (Nobel Prize 1963). The biophysically complete, four-dimensional model of the squid giant axon:

$$C_m \frac{dV}{dt} = I_{\text{ext}} - \bar{g}_{\text{Na}} m^3 h\,(V - E_{\text{Na}}) - \bar{g}_{\text{K}} n^4 (V - E_{\text{K}}) - \bar{g}_L (V - E_L)$$

with three gating variables $m$ (Na$^+$ activation), $h$ (Na$^+$ inactivation), $n$ (K$^+$ activation), each obeying first-order kinetics:

$$\frac{dx}{dt} = \alpha_x(V)(1 - x) - \beta_x(V)\,x = \frac{x_\infty(V) - x}{\tau_x(V)}, \qquad x \in \{m,h,n\}$$

where $x_\infty = \alpha_x/(\alpha_x + \beta_x)$ and $\tau_x = 1/(\alpha_x + \beta_x)$. Typical parameters: $\bar g_{\text{Na}} = 120$, $\bar g_{\text{K}} = 36$, $\bar g_L = 0.3$ mS/cm², $C_m = 1\,\mu$F/cm². HH is accurate but costs ~1200 FLOPs per 1 ms of simulation, making it impractical for large networks.

### 2.3 Izhikevich model

Izhikevich (2003), *Simple Model of Spiking Neurons*, IEEE Transactions on Neural Networks 14(6):1569–1572. A two-dimensional reduction (via normal-form theory of the saddle-node-on-invariant-circle bifurcation) that retains HH-like richness at ~13 FLOPs/ms:

$$\frac{dv}{dt} = 0.04v^2 + 5v + 140 - u + I, \qquad \frac{du}{dt} = a\,(bv - u)$$

with the auxiliary after-spike reset:

$$\text{if } v \geq 30\ \text{mV}, \quad \text{then } \begin{cases} v \leftarrow c \\ u \leftarrow u + d\end{cases}$$

$v$ is membrane potential (mV), $u$ a recovery variable (K$^+$ activation and Na$^+$ inactivation). The four parameters $(a,b,c,d)$ select the firing regime: regular spiking $(0.02, 0.2, -65, 8)$, intrinsically bursting $(0.02,0.2,-55,4)$, chattering $(0.02,0.2,-50,2)$, fast spiking $(0.1,0.2,-65,2)$, low-threshold spiking, resonator, etc. — reproducing ~20 documented cortical firing patterns.

### 2.4 STDP

Spike-Timing-Dependent Plasticity, characterized experimentally by Bi & Poo (1998), *J. Neurosci.* 18(24):10464–10472, and by Markram et al. (1997), *Science*. The classical pair-based, additive exponential window with $\Delta t = t_{\text{post}} - t_{\text{pre}}$:

$$\Delta w = \begin{cases} A_+ \exp\!\left(-\dfrac{\Delta t}{\tau_+}\right) & \text{if } \Delta t > 0 \quad \text{(LTP: pre before post — causal)} \\[2ex] -A_- \exp\!\left(\dfrac{\Delta t}{\tau_-}\right) & \text{if } \Delta t \leq 0 \quad \text{(LTD: post before pre)} \end{cases}$$

with $\tau_+ \approx \tau_- \approx 20$ ms and $A_\pm$ the learning rates. This is a **local, unsupervised, Hebbian** rule ("neurons that fire together, wire together" — Hebb, 1949) that implements a causality detector. Online implementation uses pre- and post-synaptic **eligibility traces** $x_{\text{pre}}, x_{\text{post}}$:

$$\tau_+ \frac{dx_{\text{pre}}}{dt} = -x_{\text{pre}} + \sum_f \delta(t - t^f_{\text{pre}}), \qquad \frac{dw}{dt} = A_+ x_{\text{pre}}\,S_{\text{post}}(t) - A_- x_{\text{post}}\,S_{\text{pre}}(t)$$

Variants include **multiplicative STDP** (weight-dependent, $A_+ \propto (w_{\max}-w)$, guaranteeing bounded weights), **triplet STDP** (Pfister & Gerstner, 2006), and **R-STDP / dopamine-modulated STDP** where a global reward signal gates the eligibility trace, yielding a biologically plausible reinforcement learning rule.

### 2.5 Neural coding

- **Rate coding:** information in the spike count over a window; simple, robust, but high latency and energy.
- **Temporal / latency coding (TTFS, time-to-first-spike):** information in the precise spike time; a single spike per neuron suffices, extremely energy-efficient.
- **Phase coding:** spike time relative to a background oscillation.
- **Population / rank-order coding** (Thorpe et al.): information in the *order* in which neurons in a population fire.

### 2.6 Neuromorphic hardware

- **IBM TrueNorth** (Merolla et al., *Science* 345:668–673, 2014): 4096 neurosynaptic cores, **1 million** digital neurons, **256 million** synapses, 5.4 billion transistors on Samsung 28 nm, running at ~**65–70 mW** with ~46 GSOPS/W. Fully event-driven, no clock in the neural fabric.
- **Intel Loihi** (Davies et al., *IEEE Micro* 38(1):82–99, 2018): 128 neuromorphic cores + 3 x86 cores, 130,000 LIF neurons, 130 million synapses, 14 nm, with **on-chip programmable learning rules** (including STDP) via a microcode "learning engine". Discretized dynamics: $u_i[t] = u_i[t-1](1 - \delta^{(u)}_i 2^{-12}) + 2^6\sum_j w_{ij}s_j[t]$, $v_i[t] = v_i[t-1](1-\delta^{(v)}_i 2^{-12}) + u_i[t] + u_{\text{bias}}$.
- **Loihi 2** (Intel, 2021): Intel 4 (pre-production 7 nm), 31 mm², 2.3 billion transistors, up to **1 million neurons** and 120 million synapses, 128 neuromorphic cores of 8192 neurons, 6 x86 cores, **programmable neuron models** (not just LIF) and **graded spikes** (spikes carry a 32-bit payload). Programmed via the open-source **Lava** framework. Hala Point (2024) assembles 1152 Loihi 2 chips for 1.15 billion neurons.
- **SpiNNaker / SpiNNaker2** (Furber et al., University of Manchester): ARM-based massively parallel simulator, 1 million cores in SpiNNaker1.
- **BrainScaleS** (Heidelberg): analog above-threshold, 1000–10000× accelerated against biological real-time.
- **Others:** Tianjic (Tsinghua, *Nature* 2019), Akida (BrainChip), DYNAP-SE (SynSense), IBM NorthPole (2023).

---

## 3. Self-Organizing Maps (Kohonen)

Kohonen (1982), *Self-Organized Formation of Topologically Correct Feature Maps*, Biological Cybernetics 43:59–69; book: *Self-Organizing Maps*, Springer, 1995/2001. An unsupervised, competitive-learning algorithm that projects a high-dimensional input space onto a low-dimensional (usually 2D) discrete lattice while **preserving topology**.

Each unit $i$ on the lattice carries a codebook (prototype) vector $m_i \in \mathbb{R}^n$. Two steps per sample $x(t)$:

**(a) Competition — Best Matching Unit (BMU):**

$$c = \arg\min_i \|x(t) - m_i(t)\| \quad \Longleftrightarrow \quad \|x - m_c\| = \min_i \|x - m_i\|$$

**(b) Cooperation & adaptation:**

$$m_i(t+1) = m_i(t) + \alpha(t)\,h_{ci}(t)\,\left[x(t) - m_i(t)\right]$$

with $\alpha(t) \in (0,1)$ a monotonically decreasing learning rate (e.g. $\alpha(t) = \alpha_0 e^{-t/\lambda}$) and $h_{ci}(t)$ the **neighborhood kernel** measured in *lattice* (not input) space:

$$h_{ci}(t) = \exp\!\left(-\frac{\|r_c - r_i\|^2}{2\sigma^2(t)}\right), \qquad \sigma(t) = \sigma_0 \exp\!\left(-\frac{t}{\lambda}\right)$$

or the "bubble" kernel $h_{ci} = \mathbb{1}[\|r_c - r_i\| \le \sigma(t)]$. The radius $\sigma(t)$ starts large (global ordering phase) and shrinks (fine-tuning / convergence phase); this annealing is what produces the topology-preserving unfolding. A **batch SOM** variant exists: $m_i = \frac{\sum_t h_{c(t)i}\,x(t)}{\sum_t h_{c(t)i}}$.

Quality is assessed with **quantization error** (mean $\|x - m_c\|$) and **topographic error** (fraction of samples whose first and second BMUs are non-adjacent). Applications: exploratory data visualization (U-matrix), WEBSOM document clustering, process monitoring, financial/macroprudential dashboards, gene expression clustering. SOMs are conceptually the ancestor of vector-quantization layers in VQ-VAE.

---

## 4. Capsule Networks

Sabour, Frosst & Hinton (2017), *Dynamic Routing Between Capsules*, NeurIPS 2017 (arXiv:1710.09829), building on Hinton, Krizhevsky & Wang (2011), *Transforming Auto-encoders*. A **capsule** is a group of neurons whose activity *vector* encodes the instantiation parameters (pose, deformation, hue, texture) of an entity, with the **length** of the vector encoding the probability that the entity is present. This addresses CNNs' loss of precise spatial relationships under max-pooling ("equivariance instead of invariance").

**Squashing nonlinearity** (vector-valued, preserves orientation, maps length into $[0,1)$):

$$\mathbf{v}_j = \frac{\|\mathbf{s}_j\|^2}{1 + \|\mathbf{s}_j\|^2}\,\frac{\mathbf{s}_j}{\|\mathbf{s}_j\|}$$

**Prediction vectors** ("votes") from lower capsule $i$ to higher capsule $j$ via a learned pose transformation matrix $\mathbf{W}_{ij}$:

$$\hat{\mathbf{u}}_{j|i} = \mathbf{W}_{ij}\,\mathbf{u}_i, \qquad \mathbf{s}_j = \sum_i c_{ij}\,\hat{\mathbf{u}}_{j|i}$$

**Coupling coefficients** by routing softmax over the output capsules:

$$c_{ij} = \frac{\exp(b_{ij})}{\sum_k \exp(b_{ik})}$$

**Routing-by-agreement** (typically $r=3$ iterations): initialize $b_{ij} \leftarrow 0$, then repeat

$$b_{ij} \leftarrow b_{ij} + \hat{\mathbf{u}}_{j|i} \cdot \mathbf{v}_j$$

i.e. the log-prior is increased when a lower capsule's vote agrees (large scalar product) with the current output of the higher capsule — a form of clustering in pose space that implements part–whole assignment.

**Margin loss** per class capsule $k$:

$$L_k = T_k \max(0,\, m^+ - \|\mathbf{v}_k\|)^2 + \lambda\,(1 - T_k)\max(0,\, \|\mathbf{v}_k\| - m^-)^2$$

with $m^+ = 0.9$, $m^- = 0.1$, $\lambda = 0.5$, $T_k = 1$ iff class $k$ is present. A reconstruction decoder adds a scaled-down ($0.0005$) MSE regularizer. CapsNet reached 0.25% error on MNIST and was notably strong on overlapping digits (MultiMNIST). Follow-up: **EM routing** with matrix capsules (Hinton, Sabour & Frosst, ICLR 2018) and **Stacked Capsule Autoencoders** (Kosiorek et al., NeurIPS 2019). Capsules remain computationally expensive and have not scaled to ImageNet-class problems.

---

## 5. Neural Ordinary Differential Equations

Chen, Rubanova, Bettencourt & Duvenaud (2018), *Neural Ordinary Differential Equations*, NeurIPS 2018 **Best Paper** (arXiv:1806.07366). Observing that a residual block $h_{t+1} = h_t + f(h_t, \theta_t)$ is an Euler discretization, they take the continuous limit:

$$\frac{d\mathbf{h}(t)}{dt} = f\!\left(\mathbf{h}(t), t, \theta\right), \qquad \mathbf{h}(t_1) = \mathbf{h}(t_0) + \int_{t_0}^{t_1} f(\mathbf{h}(t), t, \theta)\,dt = \operatorname{ODESolve}(\mathbf{h}(t_0), f, t_0, t_1, \theta)$$

The network becomes a **continuous-depth** model; the ODE solver (Dormand–Prince, adaptive Runge–Kutta) chooses the number of function evaluations, trading accuracy for compute *at test time*.

**Adjoint sensitivity method** (Pontryagin et al., 1962) gives $O(1)$ memory in depth, since no intermediate activations need storing. Define the adjoint $\mathbf{a}(t) = \partial L/\partial \mathbf{h}(t)$. It obeys a backward ODE:

$$\frac{d\mathbf{a}(t)}{dt} = -\mathbf{a}(t)^{\top}\frac{\partial f(\mathbf{h}(t), t, \theta)}{\partial \mathbf{h}}$$

and the parameter gradient is a single quadrature:

$$\frac{dL}{d\theta} = -\int_{t_1}^{t_0} \mathbf{a}(t)^{\top}\,\frac{\partial f(\mathbf{h}(t), t, \theta)}{\partial \theta}\,dt$$

In practice one integrates the **augmented state** $[\mathbf{h}, \mathbf{a}, \partial L/\partial\theta]$ backwards in time in a single solver call; the vector-Jacobian products $\mathbf{a}^\top \partial f/\partial \mathbf{h}$ are obtained by ordinary reverse-mode autodiff on $f$ alone.

**Consequences and descendants:**
- **Continuous Normalizing Flows / FFJORD**: the instantaneous change of variables $\frac{\partial \log p(\mathbf{z}(t))}{\partial t} = -\operatorname{tr}\!\left(\frac{\partial f}{\partial \mathbf{z}(t)}\right)$ replaces the $O(d^3)$ log-determinant of discrete flows by an $O(d)$ trace (Hutchinson estimator).
- **Latent ODEs / ODE-RNN** for irregularly-sampled time series.
- **Augmented Neural ODEs** (Dupont, Doucet & Teh, NeurIPS 2019): ODE flows are homeomorphisms and cannot cross trajectories, so some functions are unrepresentable; lifting to $[\mathbf{x}; \mathbf{a}]$ with extra dimensions fixes this.
- **Neural SDEs, Neural CDEs** (Kidger et al., 2020), **Hamiltonian/Lagrangian Neural Networks**.

---

## 6. Physics-Informed Neural Networks (PINNs)

Raissi, Perdikaris & Karniadakis (2019), *Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations*, Journal of Computational Physics 378:686–707 (preprints arXiv:1711.10561/10566). Antecedent: Lagaris, Likas & Fotiadis (1998).

Consider a PDE in general form on $\Omega \times [0,T]$:

$$u_t + \mathcal{N}[u;\lambda] = 0, \quad x \in \Omega,\ t\in[0,T]$$

Approximate $u(x,t) \approx u_\theta(x,t)$ by an MLP and define the **PDE residual**, computed exactly by automatic differentiation of the network with respect to its inputs:

$$r_\theta(x,t) := \frac{\partial u_\theta}{\partial t} + \mathcal{N}\!\left[u_\theta; \lambda\right]$$

The **composite loss** is a weighted sum of mean-squared terms:

$$\mathcal{L}(\theta) = \underbrace{\lambda_r \frac{1}{N_r}\sum_{i=1}^{N_r}\left|r_\theta(x_r^i, t_r^i)\right|^2}_{\text{PDE residual (collocation points)}} + \underbrace{\lambda_b \frac{1}{N_b}\sum_{i=1}^{N_b}\left|u_\theta(x_b^i,t_b^i) - g^i\right|^2}_{\text{boundary conditions}} + \underbrace{\lambda_0\frac{1}{N_0}\sum_{i=1}^{N_0}\left|u_\theta(x_0^i,0) - u_0^i\right|^2}_{\text{initial condition}} + \underbrace{\lambda_d \frac{1}{N_d}\sum_{i=1}^{N_d}\left|u_\theta(x_d^i,t_d^i) - u_d^i\right|^2}_{\text{observed data}}$$

Key properties:
- **Mesh-free**: collocation points are sampled (uniformly, Latin hypercube, or adaptively) rather than discretized on a grid; the curse of dimensionality is much milder than for finite elements.
- **Inverse problems come for free**: the unknown physical parameters $\lambda$ (e.g. viscosity, diffusivity) are simply additional trainable variables optimized jointly with $\theta$ against sparse data.
- **Training**: typically Adam followed by L-BFGS; `tanh` activations (need smooth higher derivatives).

**Known pathologies:** severe **loss-term imbalance** (gradient pathologies, Wang, Teng & Perdikaris, SIAM J. Sci. Comput. 2021 — fixed by learning-rate annealing or NTK-based weighting), **spectral bias** against high-frequency solutions, and failure on stiff/convection-dominated regimes ("failure modes", Krishnapriyan et al., NeurIPS 2021). Remedies include hard-constrained boundary conditions ($u_\theta = g + B(x)\,\text{NN}(x)$), domain decomposition (XPINN, cPINN), causal training weights, and Fourier feature embeddings. Extensions: **DeepONet** (Lu, Jin & Karniadakis, *Nature Machine Intelligence* 2021) and **Fourier Neural Operator** (Li et al., ICLR 2021), which learn operators between function spaces rather than single solutions.

---

## 7. Neural Radiance Fields (NeRF)

Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi & Ng (2020), *NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis*, ECCV 2020 **Best Paper Honorable Mention** (arXiv:2003.08934). A scene is a *single* MLP:

$$F_\Theta : (\mathbf{x}, \mathbf{d}) = (x,y,z,\theta,\phi) \longmapsto (\mathbf{c}, \sigma) = (r,g,b,\sigma)$$

with volume density $\sigma$ predicted from position only (enforcing multi-view consistent geometry) and view-dependent color $\mathbf{c}$ from position **and** viewing direction (enabling specularities).

**Volume rendering** along a camera ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$, between near and far bounds $[t_n, t_f]$ (classical emission-absorption model, Kajiya & Von Herzen 1984):

$$C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,\mathbf{c}(\mathbf{r}(t), \mathbf{d})\,dt, \qquad T(t) = \exp\!\left(-\int_{t_n}^{t}\sigma(\mathbf{r}(s))\,ds\right)$$

where $T(t)$ is the **accumulated transmittance** (probability the ray travels from $t_n$ to $t$ without being absorbed). The quadrature approximation over $N$ stratified samples with intervals $\delta_i = t_{i+1} - t_i$:

$$\hat{C}(\mathbf{r}) = \sum_{i=1}^{N} T_i\,\alpha_i\,\mathbf{c}_i, \qquad \alpha_i = 1 - \exp(-\sigma_i\delta_i), \qquad T_i = \exp\!\left(-\sum_{j=1}^{i-1}\sigma_j\delta_j\right) = \prod_{j=1}^{i-1}(1-\alpha_j)$$

This is exactly alpha compositing and is **fully differentiable**, so the whole pipeline trains by photometric loss alone: $\mathcal{L} = \sum_{\mathbf{r}\in\mathcal{R}} \left[\|\hat{C}_c(\mathbf{r}) - C(\mathbf{r})\|_2^2 + \|\hat{C}_f(\mathbf{r}) - C(\mathbf{r})\|_2^2\right]$ over the coarse and fine networks (hierarchical sampling: the coarse network's weights $w_i = T_i\alpha_i$ define a PDF used for inverse-transform sampling of the fine network).

**Positional encoding** is essential: MLPs exhibit spectral bias toward low frequencies, so raw $(x,y,z)$ inputs give blurry reconstructions. NeRF maps each scalar coordinate through:

$$\gamma(p) = \left(\sin(2^0\pi p), \cos(2^0 \pi p), \sin(2^1 \pi p), \cos(2^1\pi p), \ldots, \sin(2^{L-1}\pi p), \cos(2^{L-1}\pi p)\right)$$

with $L=10$ for $\mathbf{x}$ (60 dims) and $L=4$ for $\mathbf{d}$ (24 dims). Tancik et al. (NeurIPS 2020), *Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains*, explained this via NTK: the encoding converts the NTK into a stationary, tunable-bandwidth kernel.

Descendants: **Mip-NeRF** (anti-aliasing via integrated positional encoding of conical frustums), **Instant-NGP** (Müller et al., SIGGRAPH 2022 — multiresolution hash encoding, seconds instead of days), **Plenoxels**, **NeRF in the Wild**, **Zip-NeRF**, and the successor paradigm **3D Gaussian Splatting** (Kerbl et al., SIGGRAPH 2023), which replaces the MLP with explicit anisotropic Gaussians and rasterization for real-time rendering.

---

## 8. Implicit Neural Representations and SIREN

An **implicit neural representation** (INR) encodes a signal as a continuous function $\Phi_\theta : \mathbb{R}^n \to \mathbb{R}^m$ parameterized by network weights, rather than as a discrete grid — resolution-independent, memory scaling with signal complexity rather than resolution. Examples: **DeepSDF** (Park et al., CVPR 2019, signed distance functions), **Occupancy Networks** (Mescheder et al., CVPR 2019), NeRF.

**SIREN** — Sitzmann, Martel, Bergman, Lindell & Wetzstein (2020), *Implicit Neural Representations with Periodic Activation Functions*, NeurIPS 2020 **Oral** (arXiv:2006.09661). Replace ReLU by a sine:

$$\Phi(\mathbf{x}) = \mathbf{W}_n\left(\phi_{n-1}\circ\phi_{n-2}\circ\cdots\circ\phi_0\right)(\mathbf{x}) + \mathbf{b}_n, \qquad \phi_i(\mathbf{x}_i) = \sin\!\left(\omega_0\,\mathbf{W}_i\mathbf{x}_i + \mathbf{b}_i\right)$$

Crucially, **the derivative of a SIREN is itself a SIREN** (since $\frac{d}{dx}\sin(x) = \cos(x) = \sin(x + \pi/2)$), so all higher-order derivatives are well-behaved and nonzero — unlike ReLU networks, whose second derivative vanishes everywhere. This lets SIRENs be supervised directly on derivative constraints: solving the **eikonal equation** $\|\nabla_\mathbf{x}\Phi\| = 1$ for SDFs, the **Poisson equation** from gradients/Laplacians only, or the Helmholtz/wave equations.

**Principled initialization** is required to keep activation distributions stable across depth: draw $w_i \sim \mathcal{U}\!\left(-\sqrt{6/\text{fan\_in}},\, \sqrt{6/\text{fan\_in}}\right)$, so that the pre-activations are normally distributed and the post-activations are arcsine-distributed, with $\omega_0 = 30$ for the first layer to span the input frequency spectrum.

Related: **Fourier Feature Networks** (Tancik et al. 2020), **WIRE** (Gabor wavelet activations, CVPR 2023), **BACON**, **MFN** (multiplicative filter networks), and **functa**/**hypernetwork-conditioned INRs** for generative modeling over signals.

---

## 9. Networks for Deep Reinforcement Learning

### 9.1 Deep Q-Networks (DQN)

Mnih et al. (2015), *Human-level control through deep reinforcement learning*, *Nature* 518:529–533 (NIPS workshop version 2013). The optimal action-value function obeys the **Bellman optimality equation**:

$$Q^*(s,a) = \mathbb{E}_{s'\sim\mathcal{E}}\!\left[r + \gamma \max_{a'} Q^*(s',a') \,\Big|\, s,a\right]$$

DQN approximates $Q^*(s,a) \approx Q(s,a;\theta)$ with a CNN over raw pixels and minimizes the **TD loss**:

$$L_i(\theta_i) = \mathbb{E}_{(s,a,r,s')\sim U(\mathcal{D})}\left[\left(\underbrace{r + \gamma\max_{a'}Q(s',a';\theta^-)}_{\text{TD target } y} - Q(s,a;\theta_i)\right)^2\right]$$

Two stabilizing innovations: (i) **experience replay** — sample uniformly from a buffer $\mathcal{D}$ to break temporal correlations; (ii) a **target network** with frozen parameters $\theta^-$, synchronized every $C$ steps ($\theta^- \leftarrow \theta$), which prevents the target from chasing the prediction. Gradient: $\nabla_{\theta_i}L_i = \mathbb{E}\left[(y - Q(s,a;\theta_i))\nabla_{\theta_i}Q(s,a;\theta_i)\right]$; Huber loss is used in practice for robustness.

Extensions, consolidated in **Rainbow** (Hessel et al., AAAI 2018): **Double DQN** (van Hasselt et al., AAAI 2016 — decouple selection from evaluation, $y = r + \gamma Q(s', \arg\max_{a'}Q(s',a';\theta);\theta^-)$, curing overestimation bias); **Dueling networks** (Wang et al., ICML 2016 — $Q(s,a) = V(s) + A(s,a) - \frac{1}{|\mathcal{A}|}\sum_{a'}A(s,a')$); **Prioritized Experience Replay** (Schaul et al., ICLR 2016 — $p_i \propto |\delta_i|^\alpha$ with importance-sampling correction); **Noisy Nets**; **distributional RL / C51** (Bellemare et al., ICML 2017); **n-step returns**.

### 9.2 Policy gradients and REINFORCE

The **policy gradient theorem** (Sutton, McAllester, Singh & Mansour, NeurIPS 2000) for $J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}[R(\tau)]$:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}\left[\sum_{t=0}^{T}\nabla_\theta \log \pi_\theta(a_t|s_t)\, \Psi_t\right]$$

**REINFORCE** (Williams, 1992, *Simple statistical gradient-following algorithms for connectionist reinforcement learning*, Machine Learning 8:229–256) takes $\Psi_t = G_t = \sum_{k=t}^{T}\gamma^{k-t}r_k$, giving the update $\theta \leftarrow \theta + \alpha\,\gamma^t G_t \nabla_\theta \log\pi_\theta(a_t|s_t)$. It is unbiased but has very high variance; subtracting a state-dependent **baseline** $b(s_t)$ leaves it unbiased (since $\mathbb{E}[\nabla_\theta \log\pi_\theta(a|s)\,b(s)] = 0$) and reduces variance:

$$\nabla_\theta J = \mathbb{E}\left[\sum_t \nabla_\theta \log\pi_\theta(a_t|s_t)\left(G_t - b(s_t)\right)\right]$$

### 9.3 Actor-Critic and A3C

Setting $b(s) = V^\pi_\phi(s)$ and $\Psi_t = A^\pi(s_t,a_t) = Q^\pi(s_t,a_t) - V^\pi(s_t)$ gives **advantage actor-critic**. With the $n$-step estimator $\hat{A}_t = \sum_{i=0}^{n-1}\gamma^i r_{t+i} + \gamma^n V_\phi(s_{t+n}) - V_\phi(s_t)$, the combined objective is:

$$\mathcal{L} = -\log\pi_\theta(a_t|s_t)\,\hat{A}_t + c_v\left(V_\phi(s_t) - G_t\right)^2 - c_e\,H\!\left(\pi_\theta(\cdot|s_t)\right)$$

where $H$ is the policy entropy, encouraging exploration (typically $c_e = 0.01$, $c_v = 0.5$).

**A3C** — Mnih et al. (2016), *Asynchronous Methods for Deep Reinforcement Learning*, ICML 2016. Multiple CPU actor-learners explore in parallel and apply **Hogwild!**-style asynchronous updates to shared parameters; the decorrelation induced by parallel exploration replaces the replay buffer, allowing on-policy learning and training on CPUs alone. **A2C** is the synchronous, batched variant that dominates in practice on GPUs.

**GAE** (Schulman et al., ICLR 2016) interpolates bias/variance: $\hat{A}_t^{\text{GAE}(\gamma,\lambda)} = \sum_{l=0}^{\infty}(\gamma\lambda)^l\delta_{t+l}$ with $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$.

### 9.4 TRPO and PPO

TRPO (Schulman et al., ICML 2015) maximizes a surrogate under a KL trust region: $\max_\theta \mathbb{E}\left[\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}\hat A\right]$ s.t. $\mathbb{E}\left[D_{\text{KL}}(\pi_{\theta_{\text{old}}}\|\pi_\theta)\right] \le \delta$, requiring conjugate gradients and Fisher-vector products.

**PPO** — Schulman, Wolski, Dhariwal, Radford & Klimov (2017), *Proximal Policy Optimization Algorithms* (arXiv:1707.06347) — achieves the same effect first-order. With the probability ratio $r_t(\theta) = \dfrac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$, the **clipped surrogate objective** is:

$$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t\left[\min\Big(r_t(\theta)\,\hat{A}_t,\ \operatorname{clip}\big(r_t(\theta),\,1-\epsilon,\,1+\epsilon\big)\,\hat{A}_t\Big)\right]$$

typically $\epsilon = 0.2$. The $\min$ makes the objective a **pessimistic lower bound**: when $\hat A_t > 0$ the gain is capped at $1+\epsilon$; when $\hat A_t < 0$ the loss is capped at $1-\epsilon$; but the unclipped branch is retained when it is *worse*, so a policy that has moved too far in the wrong direction is still pulled back. The full loss combines value and entropy terms:

$$L^{\text{CLIP+VF+S}}(\theta) = \hat{\mathbb{E}}_t\left[L^{\text{CLIP}}_t(\theta) - c_1 L^{\text{VF}}_t(\theta) + c_2\,S[\pi_\theta](s_t)\right]$$

PPO allows multiple epochs of minibatch SGD on the same rollout, is simple and robust, and became the default for continuous control, Dota 2 (OpenAI Five), and **RLHF for large language models** (Ouyang et al., NeurIPS 2022, InstructGPT), where the reward is a learned preference model and a per-token KL penalty to the reference policy is added. An alternative penalized form is $L^{\text{KLPEN}} = \hat{\mathbb{E}}\left[r_t(\theta)\hat A_t - \beta\,\text{KL}[\pi_{\theta_{\text{old}}}, \pi_\theta]\right]$ with adaptive $\beta$.

Off-policy actor-critics complete the picture: **DDPG** (Lillicrap et al., ICLR 2016), **TD3** (Fujimoto et al., ICML 2018), **SAC** (Haarnoja et al., ICML 2018) with the maximum-entropy objective $J(\pi) = \sum_t \mathbb{E}\left[r_t + \alpha H(\pi(\cdot|s_t))\right]$.

### 9.5 AlphaGo / AlphaZero / MuZero

- **AlphaGo** (Silver et al., *Nature* 529:484–489, 2016): SL policy network trained on human expert moves, an RL policy network refined by self-play, a fast rollout policy, and a value network, all combined in MCTS. Defeated Lee Sedol 4–1 in 2016.
- **AlphaGo Zero** (Silver et al., *Nature* 550:354–359, 2017) and **AlphaZero** (Silver et al., *Science* 362:1140–1144, 2018): a single residual CNN $(\mathbf{p}, v) = f_\theta(s)$ outputs a move-prior vector and a scalar value, trained **tabula rasa** purely from self-play, no human data, generalized to chess and shogi.

MCTS selection uses a **PUCT** rule (Rosin 2011, adapted):

$$a_t = \arg\max_a\left(Q(s_t,a) + U(s_t,a)\right), \qquad U(s,a) = c_{\text{puct}}\,P(s,a)\,\frac{\sqrt{\sum_b N(s,b)}}{1 + N(s,a)}$$

where $P(s,a)$ is the network prior, $N(s,a)$ the visit count and $Q(s,a) = \frac{1}{N(s,a)}\sum_{s'|s,a\to s'} V(s')$ the mean value. Search acts as a **policy improvement operator**: the visit-count distribution $\pi_a \propto N(s,a)^{1/\tau}$ is stronger than the raw network prior, and the network is then trained to imitate it:

$$\ell = (z - v)^2 - \boldsymbol{\pi}^{\top}\log \mathbf{p} + c\|\theta\|^2$$

with $z \in \{-1,0,+1\}$ the self-play game outcome. Dirichlet noise is added at the root ($P(s,a) = (1-\varepsilon)p_a + \varepsilon\eta_a$, $\eta \sim \text{Dir}(0.03)$) for exploration.

**MuZero** (Schrittwieser et al., *Nature* 588:604–609, 2020) removes the need for a known simulator by learning a representation function $h$, dynamics function $g$ and prediction function $f$ in a latent space, trained only to predict reward, value and policy — a **learned model** for planning.

---

## 10. Siamese Networks and Metric Learning

Introduced by Bromley, Guyon, LeCun, Säckinger & Shah (1993), *Signature Verification using a "Siamese" Time Delay Neural Network*, NeurIPS 1993. Two (or more) identical towers with **shared weights** map inputs into an embedding space $f_\theta: \mathcal{X}\to\mathbb{R}^d$ where semantic similarity becomes geometric distance. This enables **one-shot / few-shot** recognition and open-set verification, where the class set is not fixed at training time.

**Contrastive loss** — Hadsell, Chopra & LeCun (2006), *Dimensionality Reduction by Learning an Invariant Mapping*, CVPR 2006 (see also Chopra, Hadsell & LeCun, CVPR 2005). With $Y=0$ for a similar pair and $Y=1$ for dissimilar, and $D_W = \|f_\theta(x_1) - f_\theta(x_2)\|_2$:

$$\mathcal{L}(W, Y, x_1, x_2) = (1-Y)\,\frac{1}{2}\,D_W^2 \;+\; Y\,\frac{1}{2}\left\{\max\left(0,\, m - D_W\right)\right\}^2$$

Similar pairs are pulled together with a quadratic spring; dissimilar pairs are pushed apart only until they reach margin $m$, after which they contribute no gradient (otherwise all negatives would be pushed to infinity).

**Triplet loss** — Schroff, Kalenichenko & Philbin (2015), *FaceNet: A Unified Embedding for Face Recognition and Clustering*, CVPR 2015; earlier ranking form in Weinberger & Saul (LMNN, JMLR 2009). Given an anchor $a$, positive $p$ (same class) and negative $n$, with embeddings constrained to the unit hypersphere $\|f(x)\|_2 = 1$:

$$\mathcal{L}_{\text{triplet}} = \sum_{i}^{N}\left[\left\|f(x_i^a) - f(x_i^p)\right\|_2^2 - \left\|f(x_i^a) - f(x_i^n)\right\|_2^2 + \alpha\right]_{+}$$

with $[\cdot]_+ = \max(0,\cdot)$ and margin $\alpha$ (0.2 in FaceNet). The loss is **relative** rather than absolute: it only demands the negative be farther than the positive by $\alpha$, which is a weaker and better-behaved constraint than contrastive loss's absolute distance targets.

**Mining is critical**: random triplets are quickly satisfied and give zero gradient. FaceNet uses *semi-hard* online mining within a large minibatch (~1800): negatives with $\|f_a - f_p\|^2 < \|f_a - f_n\|^2 < \|f_a - f_p\|^2 + \alpha$ — harder than the positive but still inside the margin. Selecting the *hardest* negatives causes collapse to $f(x) = 0$.

**Extensions:** Koch, Zemel & Salakhutdinov (ICML Deep Learning Workshop 2015) on one-shot Omniglot with binary cross-entropy over the L1 distance; **N-pair loss** (Sohn, NeurIPS 2016); **Lifted structured loss**; **angular margin softmax** losses — SphereFace (A-Softmax), CosFace, and **ArcFace** (Deng et al., CVPR 2019) with $\mathcal{L} = -\log\frac{e^{s\cos(\theta_{y_i}+m)}}{e^{s\cos(\theta_{y_i}+m)} + \sum_{j\ne y_i}e^{s\cos\theta_j}}$; and the **InfoNCE** loss (Oord et al. 2018) at the heart of SimCLR/MoCo/CLIP: $\mathcal{L} = -\log\frac{\exp(\text{sim}(z_i,z_j)/\tau)}{\sum_{k\ne i}\exp(\text{sim}(z_i,z_k)/\tau)}$. Non-contrastive siamese methods (BYOL, Grill et al. NeurIPS 2020; SimSiam, Chen & He CVPR 2021; **Barlow Twins**; **VICReg**) avoid negatives entirely via stop-gradient, predictor heads, or redundancy-reduction terms.

---

## 11. Other Specialized and Emerging Architectures

### 11.1 Extreme Learning Machines (ELM)

Huang, Zhu & Siew (2006), *Extreme learning machine: Theory and applications*, Neurocomputing 70(1–3):489–501 (ICNN 2004 conference version). For a single-hidden-layer feedforward network with $L$ hidden nodes, **randomly** assign input weights $\mathbf{w}_i$ and biases $b_i$ and never tune them; only the output weights $\boldsymbol{\beta}$ are learned. The hidden-layer output matrix is $H_{ij} = g(\mathbf{w}_j\cdot\mathbf{x}_i + b_j)$, and training reduces to the linear least-squares problem $\mathbf{H}\boldsymbol{\beta} = \mathbf{T}$, whose minimum-norm solution is:

$$\hat{\boldsymbol{\beta}} = \mathbf{H}^{\dagger}\mathbf{T}$$

with $\mathbf{H}^\dagger$ the **Moore–Penrose pseudoinverse**; the regularized version is $\hat{\boldsymbol{\beta}} = \left(\mathbf{H}^\top\mathbf{H} + \frac{\mathbf{I}}{C}\right)^{-1}\mathbf{H}^\top\mathbf{T}$. Training is non-iterative and orders of magnitude faster than backpropagation, with universal approximation guarantees. Criticism (notably by Wang & Li, 2017) is that ELM largely restates Schmidt et al. (1992) and the Random Vector Functional Link (Pao et al., 1994). The idea is the feedforward cousin of **reservoir computing** — Echo State Networks (Jaeger, 2001) and Liquid State Machines (Maass, Natschläger & Markram, 2002) — where a fixed random recurrent reservoir is read out linearly.

### 11.2 Deep Equilibrium Models (DEQ)

Bai, Kolter & Koltun (2019), *Deep Equilibrium Models*, NeurIPS 2019 (arXiv:1909.01377). Instead of stacking $L$ layers, directly solve for the **fixed point** of a single weight-tied layer, equivalent to an infinite-depth network:

$$\mathbf{z}^\star = f_\theta(\mathbf{z}^\star; \mathbf{x})$$

found with a black-box root solver (Broyden's method, Anderson acceleration) on $g_\theta(\mathbf{z}) = f_\theta(\mathbf{z};\mathbf{x}) - \mathbf{z} = 0$. Backpropagation uses the **implicit function theorem**, not the solver's trajectory:

$$\frac{\partial \ell}{\partial(\cdot)} = \frac{\partial \ell}{\partial \mathbf{z}^\star}\left(I - J_{f_\theta}(\mathbf{z}^\star)\right)^{-1}\frac{\partial f_\theta(\mathbf{z}^\star;\mathbf{x})}{\partial(\cdot)}, \qquad J_{f_\theta}(\mathbf{z}^\star) = \left.\frac{\partial f_\theta}{\partial \mathbf{z}}\right|_{\mathbf{z}^\star}$$

The inverse-Jacobian-vector product is itself computed by solving a linear fixed-point system, so **memory is $O(1)$ in depth** (up to 88% reduction reported), independent of the number of solver iterations. Concerns are ill-conditioning and non-existence/instability of the fixed point, addressed by **Jacobian regularization** (Bai, Koltun et al., NeurIPS 2021, penalizing $\|J\|_F$ via Hutchinson estimation) and monotone operator formulations (**monDEQ**, Winston & Kolter, NeurIPS 2020). DEQ is the discrete-implicit sibling of Neural ODEs; both belong to the family of **implicit layers** (see also OptNet, differentiable convex optimization layers).

### 11.3 HyperNetworks

Ha, Dai & Le (2017), *HyperNetworks*, ICLR 2017 (arXiv:1609.09106). A small network $g_\psi$ **generates the weights** of a larger primary network:

$$\theta^{(l)} = g_\psi\!\left(\mathbf{e}^{(l)}\right)$$

where $\mathbf{e}^{(l)}$ is a learned per-layer embedding. In the static case (CNNs) this is a form of relaxed weight sharing and drastic compression; in the dynamic case (recurrent HyperLSTM) the hypernetwork is itself recurrent and produces **input-dependent, time-varying** weights — a mechanism closely related to fast weights (Schmidhuber, 1992) and to linear attention. Modern descendants: **hypernetwork-conditioned INRs**, **HyperMorph** for medical registration, **task-conditioned hypernetworks** for continual learning and meta-learning, and the weight-generation view of **LoRA**-style adapters.

### 11.4 Neural Architecture Search (NAS)

- **RL-based**: Zoph & Le (2017), *Neural Architecture Search with Reinforcement Learning*, ICLR 2017 — an RNN controller samples architecture descriptions and is trained with REINFORCE on validation accuracy $R$: $\nabla_{\theta_c}J = \sum_t \mathbb{E}\left[\nabla_{\theta_c}\log P(a_t|a_{(t-1):1};\theta_c)(R - b)\right]$. Cost: ~800 GPUs for weeks. NASNet (Zoph et al., CVPR 2018) searched transferable cells rather than whole networks.
- **Evolutionary**: Real et al. (AAAI 2019), *Regularized Evolution for Image Classifier Architecture Search* (AmoebaNet), with age-based tournament selection.
- **Differentiable**: Liu, Simonyan & Yang (2019), *DARTS: Differentiable Architecture Search*, ICLR 2019. Relax the categorical choice of operation on edge $(i,j)$ into a softmax mixture over the operation set $\mathcal{O}$:

$$\bar{o}^{(i,j)}(x) = \sum_{o\in\mathcal{O}}\frac{\exp\left(\alpha_o^{(i,j)}\right)}{\sum_{o'\in\mathcal{O}}\exp\left(\alpha_{o'}^{(i,j)}\right)}\,o(x)$$

and solve the **bilevel** problem $\min_\alpha \mathcal{L}_{\text{val}}(w^*(\alpha), \alpha)$ s.t. $w^*(\alpha) = \arg\min_w \mathcal{L}_{\text{train}}(w,\alpha)$ with a one-step (second-order or first-order) approximation. Search cost drops to ~1 GPU-day. Discretization keeps $o^{(i,j)} = \arg\max_o \alpha^{(i,j)}_o$. Known failure mode: collapse to parameter-free skip connections; fixes include DARTS+, PC-DARTS, Fair DARTS, and early stopping on the Hessian eigenvalue.
- **One-shot / weight sharing**: ENAS, Once-for-All (Cai et al., ICLR 2020), BigNAS; **zero-cost proxies** and NAS benchmarks (NAS-Bench-101/201/301). Practical outcomes: EfficientNet (compound scaling), MnasNet, MobileNetV3.

### 11.5 Binarized and Quantized Neural Networks

**BinaryConnect** (Courbariaux, Bengio & David, NeurIPS 2015), **BinaryNet/BNN** (Hubara, Courbariaux et al., NeurIPS 2016), **XNOR-Net** (Rastegari et al., ECCV 2016). Weights and/or activations are constrained to $\{-1,+1\}$:

$$x^b = \operatorname{sign}(x) = \begin{cases} +1 & x \ge 0 \\ -1 & x < 0\end{cases}$$

so that the dot product collapses to XNOR + popcount, giving up to 32× memory compression and ~58× speedup on CPU. Since $\partial\,\text{sign}/\partial x = 0$ a.e., training uses the **straight-through estimator** (Hinton, 2012; Bengio, Léonard & Courville, 2013) with a clipping/hard-tanh window:

$$\frac{\partial \mathcal{L}}{\partial x} \approx \frac{\partial\mathcal{L}}{\partial x^b}\cdot\mathbb{1}_{|x|\le 1}$$

while a **latent real-valued** shadow copy of the weights accumulates the small gradient updates. XNOR-Net adds per-channel scaling factors $\alpha = \frac{1}{n}\|W\|_{\ell 1}$ to reduce the quantization error $\|W - \alpha B\|^2$. Modern relatives: **quantization-aware training** (Jacob et al., CVPR 2018), **LSQ** (learned step size), post-training quantization (GPTQ, AWQ), and 1-bit LLMs (**BitNet b1.58**, Ma et al. 2024, with ternary $\{-1,0,1\}$ weights).

### 11.6 Bayesian Neural Networks

Foundations: MacKay (1992, evidence framework), Neal (1995, HMC and the NN–GP correspondence). BNNs place a prior $p(\mathbf{w})$ over weights and seek the posterior $p(\mathbf{w}|\mathcal{D}) = \frac{p(\mathcal{D}|\mathbf{w})p(\mathbf{w})}{p(\mathcal{D})}$, giving calibrated **epistemic uncertainty**; prediction marginalizes: $p(y^*|x^*,\mathcal{D}) = \int p(y^*|x^*,\mathbf{w})\,p(\mathbf{w}|\mathcal{D})\,d\mathbf{w}$.

The posterior is intractable, so **variational inference** fits $q_\phi(\mathbf{w})$ (typically fully-factorized Gaussian, $\mathbf{w}\sim\mathcal{N}(\mu, \sigma^2)$) by minimizing the variational free energy / maximizing the **ELBO**:

$$\mathcal{F}(\phi) = D_{\text{KL}}\!\left(q_\phi(\mathbf{w})\,\|\,p(\mathbf{w})\right) - \mathbb{E}_{q_\phi(\mathbf{w})}\left[\log p(\mathcal{D}|\mathbf{w})\right] = -\text{ELBO}$$

**Bayes by Backprop** — Blundell, Cornebise, Kavukcuoglu & Wierstra (2015), *Weight Uncertainty in Neural Networks*, ICML 2015 — makes this trainable by the reparameterization trick $\mathbf{w} = \mu + \log(1+e^{\rho})\odot\boldsymbol{\epsilon}$, $\boldsymbol{\epsilon}\sim\mathcal{N}(0,I)$, yielding the unbiased minibatch estimator:

$$\mathcal{F} \approx \sum_{i=1}^{n}\left[\log q_\phi(\mathbf{w}^{(i)}) - \log p(\mathbf{w}^{(i)}) - \log p(\mathcal{D}|\mathbf{w}^{(i)})\right]$$

with a scale-mixture-of-Gaussians prior. Cheaper practical alternatives: **MC Dropout** (Gal & Ghahramani, ICML 2016 — dropout at test time approximates a deep GP), **Deep Ensembles** (Lakshminarayanan et al., NeurIPS 2017 — often the strongest baseline), **SWAG**, **Laplace approximation** (Ritter et al. 2018; Daxberger et al. 2021), and **SG-MCMC** (SGLD, Welling & Teh 2011; cyclical SG-MCMC, Zhang et al. 2020).

### 11.7 Liquid Neural Networks

Hasani, Lechner, Amini, Rus & Grosu (2021), *Liquid Time-constant Networks*, AAAI 2021 (arXiv:2006.04439), inspired by the *C. elegans* nervous system. A continuous-time RNN whose time constant is itself **state- and input-dependent** ("liquid"):

$$\frac{d\mathbf{x}(t)}{dt} = -\left[\frac{1}{\tau} + f\big(\mathbf{x}(t), \mathbf{I}(t), t, \theta\big)\right]\odot\mathbf{x}(t) + f\big(\mathbf{x}(t),\mathbf{I}(t),t,\theta\big)\odot A$$

so the effective time constant is $\tau_{\text{sys}} = \dfrac{\tau}{1 + \tau f(\mathbf{x},\mathbf{I},t,\theta)}$, varying with the input — this is what makes the model adaptive after training and gives it bounded dynamics and stable state ($\mathbf{x}$ bounded between $\min(0,A)$ and $\max(0,A)$). Training uses a fused (implicit/explicit) ODE solver with BPTT. LTCs were shown to have higher **expressivity** (measured by trajectory length) than classical CT-RNNs and to be causally more robust in end-to-end drone/car navigation with as few as 19 neurons (**Neural Circuit Policies**, Lechner et al., *Nature Machine Intelligence* 2020).

Because the LTC ODE has no known analytic solution, Hasani et al. (2022), *Closed-form Continuous-time Neural Networks*, *Nature Machine Intelligence* 4:992–1003 (arXiv:2106.13898), derived **CfC** — an approximate closed-form solution that removes the numerical solver entirely:

$$\mathbf{x}(t) = \sigma\big(-f(\mathbf{x},\mathbf{I};\theta_f)\,t\big)\odot g(\mathbf{x},\mathbf{I};\theta_g) + \left[1 - \sigma\big(-f(\mathbf{x},\mathbf{I};\theta_f)\,t\big)\right]\odot h(\mathbf{x},\mathbf{I};\theta_h)$$

with $\sigma$ a sigmoidal time-gate, yielding 1–5 orders of magnitude faster training and inference at comparable accuracy on irregularly sampled time series. Commercialized by **Liquid AI** (MIT spin-off, 2023), which extended the ideas to Liquid Foundation Models (LFM).

### 11.8 World Models and JEPA

**World Models** — Ha & Schmidhuber (2018), *World Models*, NeurIPS 2018 (arXiv:1803.10122). A three-component agent: **V** (a VAE compressing frames into $z_t$), **M** (an MDN-RNN predicting $p(z_{t+1}|a_t, z_t, h_t)$ as a mixture of Gaussians with temperature $\tau$), and **C** (a tiny linear controller $a_t = W_c[z_t\,h_t] + b_c$ evolved with CMA-ES). Crucially the agent can be trained **entirely inside its own hallucinated dream** (the "car racing in the dream" and "DoomTakeCover" experiments) and transferred back to the real environment. Successors: **PlaNet** (Hafner et al., ICML 2019), **Dreamer / DreamerV2 / DreamerV3** (Hafner et al., 2020/2021/*Nature* 2025), which learn latent dynamics with an RSSM and train an actor-critic on imagined rollouts, and **Genie** (Bruce et al., ICML 2024) for action-controllable video world models.

**JEPA** — LeCun (2022), *A Path Towards Autonomous Machine Intelligence*, OpenReview position paper. The core critique is that generative/reconstructive objectives waste capacity on unpredictable pixel-level detail. A **Joint-Embedding Predictive Architecture** instead predicts *in representation space*: an encoder $s_x = \text{Enc}_x(x)$, a target encoder $s_y = \text{Enc}_y(y)$, and a predictor $\hat{s}_y = \text{Pred}(s_x, z)$ conditioned on a latent variable $z$ capturing the residual unpredictability. Training minimizes an **energy** $E(x,y) = \|s_y - \hat s_y\|$ with a mechanism to prevent representation collapse (asymmetric architecture + EMA target encoder, or VICReg-style variance/covariance regularization) rather than negative sampling.

- **I-JEPA** — Assran et al. (2023), *Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture*, CVPR 2023: predict representations of several large, sufficiently-informative target blocks of an image from a single context block, using ViT encoders and an EMA target — strong semantic representations without hand-crafted data augmentations.
- **V-JEPA** (Bardes et al., 2024) and **V-JEPA 2** (Meta, 2025): feature prediction over masked spatio-temporal video patches, then action-conditioned fine-tuning for zero-shot robot planning.
- The family now spans audio (A-JEPA), point clouds, graphs, time series and scientific domains. JEPA is Meta AI's flagship bet on non-generative, planning-capable **world models** as the route to human-like machine intelligence, in explicit contrast to autoregressive LLMs.

---

**Sources:**
- [Kipf & Welling, GCN (TensorFlow implementation)](https://github.com/tkipf/gcn)
- [Veličković et al., Graph Attention Networks](https://www.emergentmind.com/topics/graph-attention-networks-gat)
- [Gilmer et al., Neural Message Passing for Quantum Chemistry (ICML 2017)](https://proceedings.mlr.press/v70/gilmer17a/gilmer17a.pdf)
- [Xu et al., How Powerful are Graph Neural Networks? (arXiv:1810.00826)](https://arxiv.org/pdf/2411.05464)
- [Orhan, The Leaky Integrate-and-Fire Neuron Model](https://www.cns.nyu.edu/~eorhan/notes/lif-neuron.pdf)
- [Izhikevich, Simple Model of Spiking Neurons](https://www.izhikevich.org/publications/spikes.pdf)
- [NESTML STDP windows tutorial](https://nestml.readthedocs.io/en/latest/tutorials/stdp_windows/stdp_windows.html)
- [Intel, Loihi 2 Technology Brief](https://download.intel.com/newsroom/2021/new-technologies/neuromorphic-computing-loihi-2-brief.pdf)
- [Open Neuromorphic, TrueNorth Deep Dive](https://open-neuromorphic.org/blog/truenorth-deep-dive-ibm-neuromorphic-chip-design/)
- [Hamarsheh, Self-Organizing Maps (Kohonen Maps)](https://www.philadelphia.edu.jo/academics/qhamarsheh/uploads/Lecture%2015_Self-Organizing%20Maps%20(Kohonen%20Maps).pdf)
- [Sabour, Frosst & Hinton, Dynamic Routing Between Capsules (arXiv:1710.09829)](https://arxiv.org/pdf/1710.09829)
- [Chen et al., Neural ODEs — adjoint method overview](https://arxiv.org/pdf/2209.06886)
- [Raissi et al., PINNs — comprehensive review](https://link.springer.com/article/10.1007/s10462-025-11322-7)
- [Mildenhall et al., NeRF (arXiv:2003.08934)](https://arxiv.org/pdf/2003.08934)
- [Sitzmann et al., Implicit Neural Representations with Periodic Activation Functions (arXiv:2006.09661)](https://arxiv.org/abs/2006.09661)
- [Schulman et al., PPO — implementation details](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/)
- [AlphaZero PUCT / neural MCTS](https://arxiv.org/pdf/2101.06619)
- [Hadsell/Chopra/LeCun contrastive vs. triplet loss analysis](https://arxiv.org/pdf/2510.02161)
- [Bai, Kolter & Koltun, Deep Equilibrium Models](http://implicit-layers-tutorial.org/deep_equilibrium_models/)
- [Ha, Dai & Le, HyperNetworks (arXiv:1609.09106)](https://deepsense.ai/wp-content/uploads/2023/03/1609.09106.pdf)
- [Liu, Simonyan & Yang, DARTS](https://www.cl.cam.ac.uk/~ey204/teaching/ACS/R244_2018_2019/papers/Liu_ArXiv_2018.pdf)
- [Straight-Through Estimators overview](https://www.emergentmind.com/topics/straight-through-estimators-ste)
- [Blundell et al., Bayes by Backprop / Bayesian RNNs](https://www.gatsby.ucl.ac.uk/~ucgtcbl/papers/ForBluVin2017a.pdf)
- [Hasani et al., Liquid Time-Constant Networks (AAAI 2021)](https://cdn.aaai.org/ojs/16936/16936-13-20430-1-2-20210518.pdf)
- [Hasani et al., Closed-form Continuous-time Neural Networks (arXiv:2106.13898)](https://arxiv.org/pdf/2106.13898)
- [Ha & Schmidhuber, World Models (arXiv:1803.10122)](https://arxiv.org/abs/1803.10122)
- [Meta AI, I-JEPA](https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/)
- [Extreme Learning Machine overview](https://www.sciencedirect.com/topics/computer-science/extreme-learning-machine)

> 📕 The full LaTeX book lives in its own repo: [artificial-neural-networks-book](https://git.spboucher.ai/artificial-neural-networks-book)