SPB Git

spb/artificial-neural-networks-book Public

Artificial Neural Networks — Methods, Equations and Graphical Representations: a complete book, every method with rigorous equations, pseudocode and native TikZ figures.

deep-learningneural-networksbooklatex
TeX 100%

Artificial Neural Networks — Methods, Equations and Graphical Representations

Complete 119-page LaTeX book: 13 chapters in 5 parts, 256 numbered
equations, 42 native TikZ/pgfplots figures, 26 estimation algorithms,
36-entry bibliography. Compiled PDF included.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Simon-Pierre Boucher committed 5 days ago (Aug 5, 2026)

Showing 18 changed files with +8,956 and −0

added .gitignore +17 −0
@@ -0,0 +1,17 @@
1 +# LaTeX build artifacts
2 +*.aux
3 +*.log
4 +*.out
5 +*.toc
6 +*.lof
7 +*.lot
8 +*.loa
9 +*.fls
10 +*.fdb_latexmk
11 +*.synctex.gz
12 +*.bbl
13 +*.blg
14 +build-*
15 +
16 +# OS
17 +.DS_Store
added CLAUDE.md +194 −0
@@ -0,0 +1,194 @@
1 +# CLAUDE.md — Mission : construire un LIVRE LaTeX légendaire sur les réseaux de neurones
2 +
3 +> **Amendements utilisateur (2026-08-05)** : format **livre** (parties,
4 +> préface, couverture) plutôt que thèse ; langue **anglaise** ; fichiers de
5 +> chapitres renommés en anglais ; environnements théorème en anglais
6 +> (`theorem`, `property`, `definition`, `remark`).
7 +>
8 +> **Extension majeure (2026-08-05)** : couverture de TOUS les grands types
9 +> de réseaux de neurones, chacun avec équations + **algorithme
10 +> d'estimation** (pseudocode) + figure TikZ, en profondeur maximale.
11 +> Structure finale : **13 chapitres en 5 parties** —
12 +> I. Foundations and Learning (01-foundations, 02-training,
13 +> 03-regularization) ; II. Core Architectures (04-cnn, 05-rnn,
14 +> 06-attention, 07-modern-transformers : ViT/MoE/efficient
15 +> attention/Mamba) ; III. Graphs, Energy and Memory (08-gnn : MPNN/GCN/
16 +> GraphSAGE/GAT/GIN, 09-energy : Hopfield/Boltzmann/RBM/DBN) ;
17 +> IV. Generative Models (10-generative : AE/VAE/GAN, 11-advanced-generative :
18 +> flows/diffusion/autorégressifs) ; V. Specialized and Emerging
19 +> (12-bio-inspired : SNN/SOM/reservoir, 13-emerging : capsules/Neural
20 +> ODE/NTM/KAN). Bibliographie étendue à 36 entrées. Cible : ~120 pages,
21 +> ≥30 figures, ≥150 équations, ≥15 algorithmes.
22 +
23 +> Ce fichier est une **spécification exécutable**. Claude : lis-le en entier,
24 +> puis construis le projet complet décrit ci-dessous dans ce dossier, sans
25 +> demander de confirmation intermédiaire. À la fin, compile et vérifie.
26 +
27 +---
28 +
29 +## 1. Objectif
30 +
31 +Produire un document LaTeX **robuste, complet et élégant** de type thèse,
32 +couvrant en détail toutes les grandes méthodes des réseaux de neurones.
33 +Exigence absolue : **chaque méthode = équations rigoureuses + représentation
34 +graphique TikZ**. Aucune figure externe (pas d'images importées) — tout en
35 +TikZ/pgfplots natif.
36 +
37 +## 2. Métadonnées
38 +
39 +| Champ | Valeur |
40 +|---|---|
41 +| Titre | Artificial Neural Networks — Methods, Equations and Graphical Representations |
42 +| Auteur | **Simon-Pierre Boucher** |
43 +| Contact | **contact@spboucher.ai** (lien `mailto:` cliquable sur la page de titre) |
44 +| Langue | Anglais (babel `english`) |
45 +| Classe | `book`, 11pt, A4 |
46 +
47 +## 3. Structure du projet
48 +
49 +```
50 +these/
51 +├── CLAUDE.md # ce fichier
52 +├── main.tex # préambule, page de titre, résumé, TOC, includes, bibliographie
53 +└── chapters/
54 + ├── 01-fondations.tex
55 + ├── 02-entrainement.tex
56 + ├── 03-regularisation.tex
57 + ├── 04-cnn.tex
58 + ├── 05-rnn.tex
59 + ├── 06-attention.tex
60 + └── 07-generatifs.tex
61 +```
62 +
63 +## 4. Contenu exigé par chapitre
64 +
65 +### Chapitre 1 — Fondations
66 +- Neurone artificiel : `z = w^T x + b`, `a = φ(z)` ; **figure TikZ** du neurone
67 + (entrées, poids, somme Σ, activation φ).
68 +- Perceptron de Rosenblatt : règle de mise à jour, théorème de convergence.
69 +- Fonctions d'activation : sigmoïde, tanh, ReLU, Leaky ReLU, GELU — équations
70 + **et dérivées** ; **courbes pgfplots** superposées.
71 +- Softmax (équation).
72 +- MLP : équations couche par couche `z^(ℓ) = W^(ℓ) a^(ℓ−1) + b^(ℓ)` ;
73 + **figure TikZ** d'un réseau entièrement connecté à 2 couches cachées.
74 +- Théorème d'approximation universelle (énoncé en environnement `propriete`).
75 +
76 +### Chapitre 2 — Entraînement
77 +- Risque empirique, MSE, entropie croisée ; gradient softmax+CE = `ŷ − y`.
78 +- **Rétropropagation** : les trois équations (δ^(L), récurrence δ^(ℓ),
79 + gradients W et b) en environnement théorème ; **figure TikZ** du graphe de
80 + calcul avec passe avant (flèches pleines) et passe arrière (flèches
81 + pointillées) ; **algorithme** en pseudocode (`algorithm`/`algpseudocode`).
82 +- Optimiseurs : SGD, Momentum, Nesterov, AdaGrad, RMSProp, **Adam complet**
83 + (avec correction de biais) ; **figure pgfplots** comparant les trajectoires
84 + SGD vs Momentum sur des contours de perte anisotropes.
85 +- Ordonnancement du taux d'apprentissage (paliers, cosinus, warmup).
86 +- Initialisation Xavier/Glorot et He/Kaiming (formules).
87 +
88 +### Chapitre 3 — Régularisation
89 +- Décomposition biais–variance (équation).
90 +- L2 (weight decay, forme de la mise à jour) et L1.
91 +- **Dropout** : équation inverted dropout ; **figure TikZ** comparant réseau
92 + complet vs réseau avec neurones désactivés (croix, pointillés).
93 +- **Batch Normalization** : les 4 équations (μ, σ², ẑ, γẑ+β) ; comportement
94 + à l'inférence. **Layer Normalization** (équation).
95 +- Arrêt précoce (**courbe pgfplots** train vs validation), augmentation de
96 + données, label smoothing, gradient clipping.
97 +
98 +### Chapitre 4 — CNN
99 +- Convolution : équation avec stride et padding ; formule de dimension de
100 + sortie ; **figure TikZ en grilles** : entrée 5×5, noyau 3×3 (valeurs
101 + affichées), sortie 3×3, fenêtre active surlignée.
102 +- Pooling max/moyenne (équations).
103 +- **Figure TikZ** d'une architecture CNN complète (blocs conv/pool de
104 + hauteurs décroissantes → aplatissement → dense → softmax, avec accolades
105 + « extraction de caractéristiques » / « classification »).
106 +- **ResNet** : équation `y = F(x) + x`, gradient à travers l'identité ;
107 + **figure TikZ** du bloc résiduel avec raccourci.
108 +- Champ réceptif (formule récursive).
109 +
110 +### Chapitre 5 — RNN / LSTM / GRU
111 +- RNN simple : équations ; **figure TikZ** forme repliée = forme dépliée.
112 +- BPTT : produit de jacobiennes, gradients évanescents/explosifs.
113 +- **LSTM** : les 6 équations (portes f, i, o, candidat, cellule, sortie) ;
114 + **figure TikZ détaillée de la cellule** — ligne d'état c_t horizontale en
115 + haut, portes σ/tanh, opérations × et + explicites. Vérifier visuellement
116 + qu'aucun nœud ne se chevauche.
117 +- **GRU** : les 4 équations.
118 +- Bidirectionnel, encodeur–décodeur seq2seq (factorisation autorégressive).
119 +
120 +### Chapitre 6 — Attention / Transformers
121 +- Scaled dot-product attention (équation matricielle) + justification du
122 + facteur 1/√d_k (argument de variance) ; **figure TikZ** du flux
123 + Q,K,V → QKᵀ → ÷√d_k → softmax → ×V.
124 +- Multi-têtes (équations head_i et concat·W^O).
125 +- Encodage positionnel sinusoïdal (les deux équations).
126 +- Bloc Transformer : Add&LayerNorm, FFN ; masque causal ; **grande figure
127 + TikZ** de l'architecture encodeur–décodeur complète (deux colonnes,
128 + connexions résiduelles, flèche K,V encodeur → attention croisée, ×N).
129 +- Complexité O(n²d) ; familles BERT / GPT / T5.
130 +
131 +### Chapitre 7 — Modèles génératifs
132 +- Auto-encodeur : équations + **figure TikZ** en sablier (goulot latent).
133 +- **VAE** : ELBO (avec accolades « reconstruction » / « régularisation »),
134 + KL en forme close pour gaussiennes diagonales, **reparamétrisation** ;
135 + **figure TikZ** du flux x → encodeur → (μ, σ) → z = μ+σ⊙ε → décodeur.
136 +- **GAN** : jeu minimax, discriminateur optimal D*, lien avec la divergence
137 + de Jensen–Shannon, perte non saturante, WGAN ; **figure TikZ**
138 + générateur/discriminateur avec gradient adverse en pointillés.
139 +- **Tableau comparatif** (booktabs) VAE / GAN / autorégressif.
140 +- Ouverture sur les modèles de diffusion (équation du bruitage).
141 +
142 +## 5. Conventions techniques (obligatoires)
143 +
144 +- **Préambule** : `inputenc utf8`, `fontenc T1`, `babel french`, `lmodern`,
145 + `microtype`, `amsmath amssymb amsthm mathtools bm`, `tikz` (libraries :
146 + `positioning, arrows.meta, calc, shapes.geometric, fit, backgrounds,
147 + decorations.pathreplacing`), `pgfplots` (`compat=1.17`), `algorithm` +
148 + `algpseudocode`, `geometry`, `fancyhdr`, `booktabs`, `hyperref` (liens
149 + colorés).
150 +- **Macros** : `\vect{x}` (vecteur gras via `\bm`), `\mat{W}`, `\Loss`,
151 + `\E`, `\R`, `\softmax`, `\argmin`, `\argmax`. Notation cohérente partout ;
152 + produit de Hadamard `\odot`.
153 +- **Styles TikZ globaux** dans `main.tex` : `neuron`, `ninput` (bleu),
154 + `nhidden` (orange), `noutput` (rouge), `bloc`, `fleche` (Stealth), `op` ;
155 + palette nommée : `cinput` bleu, `chidden` orange, `coutput` rouge,
156 + `cgate` vert, `cmem` violet. **Toutes** les figures réutilisent ces styles.
157 +- **Environnements théorème** : `definition`, `remarque`, `propriete`
158 + numérotés par chapitre.
159 +- Toute équation référencée porte un `\label{eq:...}` et est citée par
160 + `\eqref`. Toute figure : `figure` + `\caption` + `\label{fig:...}`.
161 +- **Page de titre** : titre, sous-titre, mini-réseau TikZ décoratif, auteur
162 + en italique, e-mail cliquable, date.
163 +- **Résumé** + mots-clés avant la table des matières.
164 +- **Bibliographie** `thebibliography` inline dans `main.tex` (pas de bibtex) :
165 + Rosenblatt 1958, Rumelhart 1986, LeCun 1998, Hochreiter 1997, Cho 2014,
166 + Srivastava 2014, Ioffe 2015, Kingma & Ba 2015, He 2016, Vaswani 2017,
167 + Kingma & Welling 2014, Goodfellow 2014, Goodfellow et al. (livre) 2016 —
168 + toutes citées au moins une fois dans le texte.
169 +
170 +## 6. Construction et vérification (workflow imposé)
171 +
172 +1. Vérifier les dépendances TeX (TeX Live complet requis : pgf/pgfplots,
173 + babel-french, algorithms).
174 +2. Créer `main.tex` puis chaque chapitre.
175 +3. Compiler : `pdflatex -interaction=nonstopmode main.tex` **deux fois**
176 + (TOC + références croisées).
177 +4. **Zéro tolérance** : aucune erreur `!`, aucune `Undefined reference`,
178 + aucune citation non résolue. Corriger et recompiler jusqu'à propreté.
179 +5. **Contrôle visuel obligatoire** : rasteriser les pages contenant les
180 + figures complexes (`pdftoppm -png`) et inspecter au minimum : page de
181 + titre, cellule LSTM, architecture Transformer, convolution en grilles.
182 + Corriger tout chevauchement de nœuds ou flèche mal routée, puis
183 + recompiler.
184 +6. Nettoyer les artefacts (`.aux`, `.log`, `.out`, `.toc`, PNG de contrôle).
185 +7. Livrer : `main.pdf` + le dossier source complet (zip).
186 +
187 +## 7. Critères de « légendaire »
188 +
189 +- ~25 pages ou plus, densité soutenue mais lisible.
190 +- Au moins **12 figures TikZ/pgfplots** et **50 équations numérotées**.
191 +- Cohérence de notation absolue d'un chapitre à l'autre.
192 +- Chaque affirmation non triviale adossée à une équation, une figure ou une
193 + référence bibliographique.
194 +- Le document doit pouvoir servir de support de cours autonome.
added README.md +103 −0
@@ -0,0 +1,103 @@
1 +# Artificial Neural Networks — Methods, Equations and Graphical Representations
2 +
3 +**A complete, self-contained book on neural networks — every method presented with rigorous equations, an estimation algorithm in pseudocode, and a native TikZ figure.**
4 +
5 +*Simon-Pierre Boucher* · [contact@spboucher.ai](mailto:contact@spboucher.ai)
6 +
7 +---
8 +
9 +## 📖 About
10 +
11 +This repository contains the full LaTeX source and compiled PDF of a 119-page book covering **all major families of artificial neural networks**, from Rosenblatt's perceptron (1958) to Kolmogorov–Arnold networks (2024).
12 +
13 +The book is built on one strict organizing principle:
14 +
15 +> **Every architecture = rigorous equations + estimation/training algorithm + faithful graphical representation.**
16 +
17 +Every single figure is drawn natively in **TikZ/pgfplots** — no imported images — so each diagram is exactly as precise as the equations it illustrates. Notation is unified across all chapters (bold lowercase vectors, bold uppercase matrices, Hadamard product `⊙`, shared color palette for inputs/hidden/output/gates/memory).
18 +
19 +## 📊 At a glance
20 +
21 +| Metric | Count |
22 +|---|---:|
23 +| Pages | 119 |
24 +| Chapters | 13 (in 5 parts) |
25 +| Numbered equations | 256 |
26 +| Native TikZ/pgfplots figures | 42 |
27 +| Estimation algorithms (pseudocode) | 26 |
28 +| Comparison tables | 7 |
29 +| Bibliography entries (all cited) | 36 |
30 +
31 +## 🗂 Contents
32 +
33 +### Part I — Foundations and Learning
34 +1. **Foundations of Neural Networks** — artificial neuron, perceptron (Novikoff convergence theorem + learning algorithm), activation functions with derivatives, softmax, MLP, universal approximation theorem, XOR geometry, RBF contrast.
35 +2. **Training Neural Networks** — empirical risk, MSE/cross-entropy, the softmax+CE gradient, the three backpropagation equations (with computational-graph figure and full pseudocode), SGD/Momentum/Nesterov/AdaGrad/RMSProp/**Adam** with bias correction, learning-rate schedules, Xavier/He initialization.
36 +3. **Regularization** — bias–variance decomposition, L2/L1, inverted dropout, Batch & Layer Normalization, early stopping, label smoothing, gradient clipping.
37 +
38 +### Part II — Core Architectures
39 +4. **Convolutional Neural Networks** — convolution with stride/padding (worked 5×5 ⊛ 3×3 grid figure), pooling, canonical architecture, **ResNet** residual block and its gradient, receptive fields, conv forward-pass algorithm.
40 +5. **Recurrent Networks: RNN, LSTM and GRU** — folded/unfolded views, BPTT (Jacobian products, vanishing/exploding gradients, algorithm), the six **LSTM** equations with a detailed cell diagram, GRU, bidirectional RNNs, seq2seq.
41 +6. **Attention and Transformers** — scaled dot-product attention with the 1/√d_k variance argument, multi-head attention, sinusoidal positional encoding, Add&Norm/FFN/causal masking, full **encoder–decoder architecture figure**, greedy decoding algorithm, BERT/GPT/T5.
42 +7. **Modern Transformer Variants** — **Vision Transformer** (patches, [CLS], training algorithm), **Mixture of Experts** (noisy top-k routing, load balancing), efficient/linear attention, **state-space models and Mamba** (ZOH discretization, selective scan), scaling laws.
43 +
44 +### Part III — Graphs, Energy and Memory
45 +8. **Graph Neural Networks** — message-passing framework, **GCN** (spectral derivation), **GraphSAGE** (minibatch estimation algorithm), **GAT**, **GIN** with the 1-WL expressiveness theorem, task heads.
46 +9. **Energy-Based and Associative-Memory Networks** — **Hopfield** networks (energy descent theorem, capacity, recall algorithm), modern Hopfield ≈ attention, **Boltzmann machines**, **RBM** with **contrastive divergence (CD-k)**, deep belief networks with greedy layer-wise pretraining.
47 +
48 +### Part IV — Generative Models
49 +10. **Autoencoders, VAEs and GANs** — hourglass autoencoder, **VAE** (ELBO with reconstruction/regularization decomposition, closed-form Gaussian KL, reparameterization trick, training algorithm), **GAN** (minimax game, optimal discriminator, Jensen–Shannon link, non-saturating loss, WGAN, alternating training algorithm), VAE/GAN/autoregressive comparison table.
50 +11. **Normalizing Flows, Diffusion and Autoregressive Models** — change of variables, RealNVP coupling layers, **DDPM** (forward/reverse processes, variational bound, ε-parameterization, training *and* sampling algorithms), DDIM, classifier-free guidance, **WaveNet** dilated causal convolutions, energy-based models.
51 +
52 +### Part V — Specialized and Emerging Architectures
53 +12. **Biologically Inspired Networks** — **spiking neurons** (LIF dynamics, f–I curve, STDP window, surrogate gradients), **self-organizing maps** (BMU, neighborhood kernel, training loop), **reservoir computing / echo state networks** (echo-state property, ridge readout).
54 +13. **Emerging Architectures** — **capsule networks** (dynamic routing-by-agreement), **Neural ODEs** (adjoint method), **Neural Turing Machines** (full addressing pipeline), **Kolmogorov–Arnold networks** (superposition theorem, learnable edge functions).
55 +
56 +## 🔧 Building from source
57 +
58 +Requires a full TeX Live installation (pgf/pgfplots, `babel-english`, `algorithm`/`algpseudocode`, `booktabs`, `microtype`, `hyperref`).
59 +
60 +```bash
61 +cd these # repository root
62 +pdflatex -interaction=nonstopmode main.tex
63 +pdflatex -interaction=nonstopmode main.tex # resolve TOC + cross-references
64 +pdflatex -interaction=nonstopmode main.tex # settle part/page numbers
65 +```
66 +
67 +The build is clean by construction: **zero errors, zero undefined references, zero unresolved citations**.
68 +
69 +## 📁 Repository layout
70 +
71 +```
72 +.
73 +├── README.md
74 +├── CLAUDE.md # executable build specification of the book
75 +├── main.pdf # compiled book (119 pages)
76 +├── main.tex # preamble, cover, preface, abstract, TOC, bibliography
77 +└── chapters/
78 + ├── 01-foundations.tex
79 + ├── 02-training.tex
80 + ├── 03-regularization.tex
81 + ├── 04-cnn.tex
82 + ├── 05-rnn.tex
83 + ├── 06-attention.tex
84 + ├── 07-modern-transformers.tex
85 + ├── 08-gnn.tex
86 + ├── 09-energy.tex
87 + ├── 10-generative.tex
88 + ├── 11-advanced-generative.tex
89 + ├── 12-bio-inspired.tex
90 + └── 13-emerging.tex
91 +```
92 +
93 +All global TikZ styles, the named color palette, notation macros and theorem environments are defined once in `main.tex` and reused by every chapter — guaranteeing visual and mathematical consistency across the whole book.
94 +
95 +## ✍️ Author
96 +
97 +**Simon-Pierre Boucher**
98 +📧 [contact@spboucher.ai](mailto:contact@spboucher.ai)
99 +
100 +## 📜 License
101 +
102 +© 2026 Simon-Pierre Boucher. All rights reserved.
103 +The source is published for reading and personal study; please contact the author for any other use.
added chapters/01-foundations.tex +706 −0
@@ -0,0 +1,706 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 1 : Foundations of Neural Networks (chapters/01-foundations.tex)
6 +% ============================================================================
7 +
8 +\chapter{Foundations of Neural Networks}\label{chap:foundations}
9 +
10 +Artificial neural networks rest on a small number of ideas of remarkable
11 +economy: a neuron is an affine map followed by a nonlinearity, a network is
12 +a composition of such maps, and learning is the adjustment of the affine
13 +parameters from examples. This chapter develops these ideas in their
14 +historical and mathematical order. We begin with the artificial neuron as a
15 +radical abstraction of its biological counterpart, formalize the perceptron
16 +of Rosenblatt~\cite{rosenblatt1958} together with its learning rule and
17 +convergence guarantee, examine the activation functions that give networks
18 +their expressive power, and assemble these components into the multilayer
19 +perceptron (MLP). The chapter closes with the universal approximation
20 +theorem, which explains \emph{why} such compositions can, in principle,
21 +represent essentially any continuous function.
22 +
23 +Throughout the book, vectors are bold lowercase ($\vect{x} \in \R^n$),
24 +matrices bold uppercase ($\mat{W} \in \R^{m \times n}$), scalars italic,
25 +and the Hadamard (elementwise) product is written $\odot$. The loss is
26 +$\Loss$ and expectation is $\E$.
27 +
28 +% ----------------------------------------------------------------------------
29 +\section{From the Biological Neuron to the Artificial Neuron}
30 +% ----------------------------------------------------------------------------
31 +
32 +A biological neuron receives electrochemical signals through its
33 +\emph{dendrites}, integrates them in the cell body, and — if the aggregated
34 +membrane depolarization crosses a threshold — emits an action potential
35 +that propagates along the axon to synapses connecting to downstream
36 +neurons. Two properties of this system survive in the mathematical
37 +abstraction: synaptic transmission is \emph{weighted} (a synapse may be
38 +excitatory or inhibitory, strong or weak), and firing is \emph{nonlinear
39 +and threshold-based}. Everything else — spike timing, refractory periods,
40 +dendritic compartmentalization, neuromodulation — is discarded. The
41 +artificial neuron is therefore best understood as a computational primitive
42 +\emph{inspired by}, not a model of, neurophysiology.
43 +
44 +The founding formalization is due to McCulloch and Pitts (1943). Their
45 +neuron takes Boolean inputs $x_i \in \{0,1\}$ and produces a Boolean output
46 +through a threshold (Heaviside) function:
47 +\begin{equation}
48 + y \;=\; \Theta\!\left(\sum_{i=1}^{n} w_i x_i - \theta\right),
49 + \qquad
50 + \Theta(u) \;=\;
51 + \begin{cases}
52 + 1 & \text{if } u \geq 0,\\
53 + 0 & \text{if } u < 0.
54 + \end{cases}
55 + \label{eq:found-mp}
56 +\end{equation}
57 +Networks of such units can implement any Boolean function — AND is obtained
58 +with $\theta = n$, OR with $\theta = 1$, NOT via inhibition — and, when
59 +cycles are allowed, any finite-state automaton. The decisive limitation of
60 +the McCulloch--Pitts unit is that it does not \emph{learn}: the weights
61 +$w_i$ and the threshold $\theta$ must be set by the designer. The history
62 +of neural networks is, in large part, the history of removing this
63 +limitation.
64 +
65 +% ----------------------------------------------------------------------------
66 +\section{The Artificial Neuron}
67 +% ----------------------------------------------------------------------------
68 +
69 +\begin{definition}[Artificial neuron]\label{def:found-neuron}
70 +An \emph{artificial neuron} with weights $\vect{w} \in \R^n$, bias
71 +$b \in \R$ and activation function
72 +$\varphi : \R \to \R$ is the map that assigns to an input
73 +$\vect{x} \in \R^n$ the \emph{pre-activation} $z$ and the
74 +\emph{activation} $a$ defined by
75 +\begin{equation}
76 + z \;=\; \vect{w}\transp \vect{x} + b \;=\; \sum_{i=1}^{n} w_i x_i + b,
77 + \qquad
78 + a \;=\; \varphi(z).
79 + \label{eq:found-neuron}
80 +\end{equation}
81 +\end{definition}
82 +
83 +The computation~\eqref{eq:found-neuron} decomposes into three stages,
84 +represented graphically in Figure~\ref{fig:found-neuron}: a weighted
85 +aggregation of the inputs, the addition of a bias, and a pointwise
86 +nonlinearity. The bias can be absorbed into the weights by augmenting the
87 +input, $\vect{x} \leftarrow (\vect{x}, 1)$ and
88 +$\vect{w} \leftarrow (\vect{w}, b)$, a convention we use freely whenever it
89 +lightens notation.
90 +
91 +\begin{figure}[htbp]
92 + \centering
93 + \begin{tikzpicture}
94 + % --- inputs -------------------------------------------------------------
95 + \node[ninput] (x1) at (0, 2.2) {$x_1$};
96 + \node[ninput] (x2) at (0, 1.1) {$x_2$};
97 + \node[ninput] (x3) at (0, 0.0) {$x_3$};
98 + \node (xd) at (0,-0.85) {$\vdots$};
99 + \node[ninput] (xn) at (0,-1.7) {$x_n$};
100 + % --- bias ---------------------------------------------------------------
101 + \node[neuron, fill=black!8] (bias) at (3.2, 2.3) {$1$};
102 + % --- sum and activation ---------------------------------------------------
103 + \node[op, minimum size=9mm] (sum) at (3.2, 0.25) {$\Sigma$};
104 + \node[nhidden, minimum size=9mm] (phi) at (5.5, 0.25) {$\varphi$};
105 + % --- output ---------------------------------------------------------------
106 + \node (out) at (7.9, 0.25) {$a = \varphi(z)$};
107 + % --- edges with weight labels ---------------------------------------------
108 + \draw[fleche] (x1) -- node[etiquette, above, pos=0.40] {$w_1$} (sum);
109 + \draw[fleche] (x2) -- node[etiquette, above, pos=0.45] {$w_2$} (sum);
110 + \draw[fleche] (x3) -- node[etiquette, above, pos=0.50] {$w_3$} (sum);
111 + \draw[fleche] (xn) -- node[etiquette, below, pos=0.45] {$w_n$} (sum);
112 + \draw[fleche] (bias) -- node[etiquette, right, pos=0.45] {$b$} (sum);
113 + \draw[fleche] (sum) -- node[etiquette, above] {$z$} (phi);
114 + \draw[fleche] (phi) -- (out);
115 + \end{tikzpicture}
116 + \caption{The artificial neuron of
117 + Definition~\ref{def:found-neuron}. Inputs $x_1, \dots, x_n$ are
118 + weighted by $w_1, \dots, w_n$ and aggregated by the summation node
119 + $\Sigma$ together with the bias $b$; the pre-activation
120 + $z = \vect{w}\transp\vect{x} + b$ then passes through the activation
121 + function $\varphi$ to produce the output $a = \varphi(z)$.}
122 + \label{fig:found-neuron}
123 +\end{figure}
124 +
125 +\begin{remark}[Geometry of a single neuron]
126 +The level set $\{\vect{x} : \vect{w}\transp\vect{x} + b = 0\}$ is a
127 +hyperplane with normal vector $\vect{w}$, at signed distance
128 +$-b/\lVert\vect{w}\rVert$ from the origin. A single neuron with a monotone
129 +activation therefore responds along one direction of input space only: it
130 +is a \emph{linear feature detector} followed by a nonlinear read-out.
131 +Everything a deep network does can be traced back to compositions of these
132 +elementary half-space responses.
133 +\end{remark}
134 +
135 +% ----------------------------------------------------------------------------
136 +\section{The Rosenblatt Perceptron}
137 +% ----------------------------------------------------------------------------
138 +
139 +Rosenblatt's perceptron~\cite{rosenblatt1958} is historically the first
140 +neuron that \emph{learns}: its weights are adjusted from labelled examples
141 +rather than hand-designed. For binary classification with labels
142 +$y \in \{-1, +1\}$, the perceptron predicts with the sign of the
143 +pre-activation,
144 +\begin{equation}
145 + \hat{y} \;=\; \operatorname{sign}\!\left(\vect{w}\transp\vect{x} + b\right)
146 + \;=\;
147 + \begin{cases}
148 + +1 & \text{if } \vect{w}\transp\vect{x} + b \geq 0,\\
149 + -1 & \text{otherwise,}
150 + \end{cases}
151 + \label{eq:found-perceptron-decision}
152 +\end{equation}
153 +so that the decision boundary is the hyperplane
154 +$\vect{w}\transp\vect{x} + b = 0$. Learning proceeds \emph{online}: the
155 +examples $(\vect{x}^{(k)}, y^{(k)})$ are presented one at a time, and each
156 +misclassified example triggers the update
157 +\begin{equation}
158 + \vect{w} \;\leftarrow\; \vect{w} + \eta\, y^{(k)} \vect{x}^{(k)},
159 + \qquad
160 + b \;\leftarrow\; b + \eta\, y^{(k)},
161 + \label{eq:found-perceptron-update}
162 +\end{equation}
163 +with learning rate $\eta > 0$; correctly classified examples produce no
164 +update. Rule~\eqref{eq:found-perceptron-update} is intuitive — it rotates
165 +the hyperplane toward each mistake — and it requires no differentiability.
166 +Indeed, the step function of~\eqref{eq:found-perceptron-decision} has zero
167 +derivative almost everywhere, so the perceptron rule is \emph{not}
168 +gradient descent on the misclassification error. It is, however,
169 +exactly stochastic subgradient descent on the \emph{perceptron criterion}
170 +\begin{equation}
171 + \Loss(\vect{w}, b)
172 + \;=\;
173 + \max\!\left(0,\; -y\left(\vect{w}\transp\vect{x} + b\right)\right),
174 + \label{eq:found-perceptron-criterion}
175 +\end{equation}
176 +which vanishes on correct classifications and grows linearly with the
177 +margin of error otherwise — an early ancestor of the hinge loss. The
178 +complete estimation procedure, which simply cycles through the data
179 +applying~\eqref{eq:found-perceptron-update} until an error-free pass,
180 +is summarized in Algorithm~\ref{alg:found-perceptron}.
181 +
182 +\begin{algorithm}[htbp]
183 + \caption{Perceptron learning (Rosenblatt)}
184 + \label{alg:found-perceptron}
185 + \begin{algorithmic}[1]
186 + \Require training set $\{(\vect{x}^{(k)}, y^{(k)})\}_{k=1}^{m}$ with
187 + $y^{(k)} \in \{-1,+1\}$, learning rate $\eta > 0$
188 + \State $\vect{w} \gets \vect{0}$, \quad $b \gets 0$
189 + \Repeat
190 + \State $\mathit{errors} \gets 0$
191 + \For{$k = 1, \dots, m$}
192 + \If{$y^{(k)}\bigl(\vect{w}\transp\vect{x}^{(k)} + b\bigr) \leq 0$}
193 + \Comment{misclassified example}
194 + \State $\vect{w} \gets \vect{w} + \eta\, y^{(k)}\vect{x}^{(k)}$
195 + \State $b \gets b + \eta\, y^{(k)}$
196 + \State $\mathit{errors} \gets \mathit{errors} + 1$
197 + \EndIf
198 + \EndFor
199 + \Until{$\mathit{errors} = 0$}
200 + \State \Return $\vect{w},\, b$
201 + \end{algorithmic}
202 +\end{algorithm}
203 +
204 +The perceptron admits one of the cleanest guarantees in machine learning,
205 +formalized by Novikoff (1962).
206 +
207 +\begin{theorem}[Perceptron convergence]\label{thm:found-convergence}
208 +Suppose the training set
209 +$\{(\vect{x}^{(k)}, y^{(k)})\}_{k=1}^{m}$ is linearly separable with
210 +margin $\gamma > 0$: there exists a unit vector $\vect{w}^{\star}$,
211 +$\lVert\vect{w}^{\star}\rVert = 1$, such that
212 +$y^{(k)}\,\vect{w}^{\star\top}\vect{x}^{(k)} \geq \gamma$ for all $k$.
213 +Let $R = \max_k \lVert\vect{x}^{(k)}\rVert$. Then the perceptron
214 +algorithm~\eqref{eq:found-perceptron-update}, started from
215 +$\vect{w} = \vect{0}$, makes at most
216 +\begin{equation}
217 + T \;\leq\; \left(\frac{R}{\gamma}\right)^{2}
218 + \label{eq:found-novikoff}
219 +\end{equation}
220 +updates before finding a separating hyperplane.
221 +\end{theorem}
222 +
223 +\begin{remark}
224 +The bound~\eqref{eq:found-novikoff} is independent of the input dimension
225 +$n$ and of the number of examples $m$: only the \emph{normalized margin}
226 +$\gamma / R$ matters. This is an early instance of a recurring theme —
227 +the difficulty of a learning problem is governed by geometric quantities,
228 +not by raw dimensionality.
229 +\end{remark}
230 +
231 +Shortly after the perceptron, Widrow and Hoff (1960) introduced ADALINE,
232 +which differs in one crucial respect: the error is computed on the
233 +\emph{linear} pre-activation $z = \vect{w}\transp\vect{x} + b$ rather than
234 +on the thresholded output. Minimizing the squared error
235 +$\tfrac{1}{2}(d - z)^2$ against a target $d$ gives the \emph{delta rule}
236 +(or LMS, least-mean-squares, rule)
237 +\begin{equation}
238 + \vect{w} \;\leftarrow\; \vect{w} + \eta\,(d - z)\,\vect{x},
239 + \label{eq:found-adaline}
240 +\end{equation}
241 +which, unlike~\eqref{eq:found-perceptron-update}, \emph{is} genuine
242 +stochastic gradient descent on a differentiable objective. The delta rule
243 +keeps improving even after the data are separated, driving the weights
244 +toward the minimum mean-squared-error solution, and it does not diverge on
245 +non-separable data. Gradient-based learning of exactly this kind — extended
246 +to many layers — is the subject of the next chapter.
247 +
248 +% ----------------------------------------------------------------------------
249 +\subsection{The Limits of Linear Separation: XOR}
250 +% ----------------------------------------------------------------------------
251 +
252 +The perceptron can only realize decision boundaries that are hyperplanes.
253 +Minsky and Papert (1969) analyzed this restriction with mathematical
254 +precision, and their canonical counterexample is the exclusive-or function
255 +XOR, shown in Figure~\ref{fig:found-xor}: the positive examples
256 +$(0,1)$ and $(1,0)$ sit on one diagonal of the unit square, the negative
257 +examples $(0,0)$ and $(1,1)$ on the other, and no straight line separates
258 +two diagonals. The proof is elementary. If a separating line
259 +$w_1 x_1 + w_2 x_2 + b = 0$ existed, then $(0,0) \mapsto 0$ forces
260 +$b < 0$, while the positive cases force $w_1 + b \geq 0$ and
261 +$w_2 + b \geq 0$; adding these, $w_1 + w_2 + 2b \geq 0$, hence
262 +$w_1 + w_2 + b \geq -b > 0$, which would classify $(1,1)$ as positive — a
263 +contradiction.
264 +
265 +\begin{figure}[htbp]
266 + \centering
267 + \begin{tikzpicture}[scale=2.6]
268 + % axes
269 + \draw[fleche, black!60] (-0.30, 0) -- (1.55, 0)
270 + node[below, font=\small] {$x_1$};
271 + \draw[fleche, black!60] (0, -0.30) -- (0, 1.55)
272 + node[left, font=\small] {$x_2$};
273 + % the two hidden-unit hyperplanes
274 + \draw[cgate, dashed, thick] (-0.15, 0.65) -- (0.65, -0.15)
275 + node[etiquette, below, text=cgate!60!black]
276 + {$x_1 + x_2 = \tfrac{1}{2}$};
277 + \draw[cgate, dashed, thick] (0.38, 1.18) -- (1.18, 0.38)
278 + node[etiquette, below right, text=cgate!60!black]
279 + {$x_1 + x_2 = \tfrac{3}{2}$};
280 + % class 0 (blue) and class 1 (red) points
281 + \node[circle, fill=cinput, draw=cinput!60!black, inner sep=2.4pt,
282 + label={[etiquette]below left:$(0,0)$}] at (0,0) {};
283 + \node[circle, fill=cinput, draw=cinput!60!black, inner sep=2.4pt,
284 + label={[etiquette]above right:$(1,1)$}] at (1,1) {};
285 + \node[circle, fill=coutput, draw=coutput!60!black, inner sep=2.4pt,
286 + label={[etiquette]above left:$(0,1)$}] at (0,1) {};
287 + \node[circle, fill=coutput, draw=coutput!60!black, inner sep=2.4pt,
288 + label={[etiquette]below right:$(1,0)$}] at (1,0) {};
289 + \end{tikzpicture}
290 + \caption{The XOR problem. Negative examples (blue) occupy one diagonal
291 + of the unit square, positive examples (red) the other; no single line
292 + separates the two classes. The two dashed lines show the resolution by
293 + a hidden layer: two threshold units implementing
294 + $x_1 + x_2 \geq \tfrac{1}{2}$ (OR) and $x_1 + x_2 \leq \tfrac{3}{2}$
295 + (NAND) carve out the strip between the lines, and an AND unit on top
296 + of them computes XOR exactly.}
297 + \label{fig:found-xor}
298 +\end{figure}
299 +
300 +Crucially, XOR \emph{is} solvable with one hidden layer:
301 +$\mathrm{XOR}(x_1, x_2) = \mathrm{OR}(x_1, x_2) \wedge
302 +\neg\,\mathrm{AND}(x_1, x_2)$, i.e.\ two threshold units feeding a third.
303 +Minsky and Papert's pessimism about such multilayer extensions — for which
304 +no training algorithm was then known — is commonly cited as a trigger of
305 +the first connectionist winter, which lasted until the mid-1980s. The
306 +missing algorithm, backpropagation, is derived in the next chapter; the
307 +representational question of what multilayer networks \emph{can} express
308 +is answered at the end of this one.
309 +
310 +% ----------------------------------------------------------------------------
311 +\section{Activation Functions}
312 +% ----------------------------------------------------------------------------
313 +
314 +The nonlinearity $\varphi$ is not a detail: without it, a network of any
315 +depth collapses. If every layer applied only an affine map, their
316 +composition would again be affine, and the deepest network would be
317 +exactly as expressive as a single linear layer. The choice of $\varphi$
318 +also governs how gradients flow backward through the network — as we will
319 +see in the next chapter, training multiplies derivatives $\varphi'$ across
320 +layers, so activations that \emph{saturate} (have near-zero derivative
321 +over most of their domain) starve deep networks of learning signal. We now
322 +review the five activations that dominate practice, giving for each the
323 +definition and the derivative; Figure~\ref{fig:found-activations} plots
324 +them side by side.
325 +
326 +\paragraph{Sigmoid.} The logistic sigmoid maps $\R$ onto $(0,1)$ and was
327 +historically the default choice:
328 +\begin{equation}
329 + \sigma(z) \;=\; \frac{1}{1 + e^{-z}},
330 + \qquad
331 + \sigma'(z) \;=\; \sigma(z)\bigl(1 - \sigma(z)\bigr).
332 + \label{eq:found-sigmoid}
333 +\end{equation}
334 +Its derivative is bounded by $\sigma'(0) = \tfrac{1}{4}$ and decays
335 +exponentially for $\lvert z \rvert \gtrsim 5$: the function
336 +\emph{saturates}, and its outputs are not zero-centred, which induces
337 +correlated gradient signs within a layer. It survives today mainly in
338 +gates and binary output units.
339 +
340 +\paragraph{Hyperbolic tangent.} A rescaled sigmoid,
341 +$\tanh(z) = 2\sigma(2z) - 1$, with range $(-1, 1)$:
342 +\begin{equation}
343 + \tanh(z) \;=\; \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}},
344 + \qquad
345 + \frac{d}{dz}\tanh(z) \;=\; 1 - \tanh^{2}(z).
346 + \label{eq:found-tanh}
347 +\end{equation}
348 +Zero-centred and with maximal derivative $1$ at the origin, it is better
349 +conditioned than the sigmoid for hidden layers, though still saturating;
350 +it remains standard inside recurrent cells.
351 +
352 +\paragraph{ReLU.} The rectified linear unit, repopularized around
353 +2010--2012, is the piecewise-linear map
354 +\begin{equation}
355 + \mathrm{ReLU}(z) \;=\; \max(0, z),
356 + \qquad
357 + \mathrm{ReLU}'(z) \;=\;
358 + \begin{cases}
359 + 1 & z > 0,\\
360 + 0 & z < 0,
361 + \end{cases}
362 + \label{eq:found-relu}
363 +\end{equation}
364 +with the convention $\mathrm{ReLU}'(0) = 0$ at the (measure-zero)
365 +non-differentiable point. For positive inputs the gradient is exactly $1$,
366 +so ReLU does not saturate on its active half — the property that unlocked
367 +the training of genuinely deep networks. Its failure mode is the
368 +\emph{dying ReLU}: a unit pushed into the negative regime for every input
369 +receives zero gradient forever and never recovers.
370 +
371 +\paragraph{Leaky ReLU.} A minimal repair of the dying-unit problem replaces
372 +the zero slope with a small $\alpha > 0$ (typically $\alpha = 0.01$):
373 +\begin{equation}
374 + \mathrm{LReLU}(z) \;=\; \max(\alpha z,\, z)
375 + \;=\;
376 + \begin{cases}
377 + z & z \geq 0,\\
378 + \alpha z & z < 0,
379 + \end{cases}
380 + \qquad
381 + \mathrm{LReLU}'(z) \;=\;
382 + \begin{cases}
383 + 1 & z > 0,\\
384 + \alpha & z < 0.
385 + \end{cases}
386 + \label{eq:found-leaky}
387 +\end{equation}
388 +The parametric variant (PReLU) learns $\alpha$ per channel at negligible
389 +cost.
390 +
391 +\paragraph{GELU.} The Gaussian error linear unit weights its input by the
392 +probability that a standard Gaussian falls below it. With $\Phi$ and
393 +$\phi$ the standard normal cumulative distribution function and density,
394 +\begin{equation}
395 + \mathrm{GELU}(z) \;=\; z\,\Phi(z)
396 + \;=\; \frac{z}{2}\left[1 + \operatorname{erf}\!\left(
397 + \frac{z}{\sqrt{2}}\right)\right],
398 + \qquad
399 + \frac{d}{dz}\mathrm{GELU}(z) \;=\; \Phi(z) + z\,\phi(z).
400 + \label{eq:found-gelu}
401 +\end{equation}
402 +GELU can be read as a deterministic version of stochastic gating: instead
403 +of dropping a unit with probability $1 - \Phi(z)$, it scales the unit by
404 +the expected mask. It is smooth, non-monotonic (a shallow negative dip
405 +near $z \approx -0.75$), and is the default activation in Transformer
406 +architectures. Implementations commonly use the tanh approximation
407 +\begin{equation}
408 + \mathrm{GELU}(z) \;\approx\; \tfrac{1}{2}\, z \left(1 + \tanh\!\left[
409 + \sqrt{\tfrac{2}{\pi}}\left(z + 0.044715\, z^{3}\right)
410 + \right]\right).
411 + \label{eq:found-gelu-approx}
412 +\end{equation}
413 +
414 +\begin{figure}[htbp]
415 + \centering
416 + \begin{tikzpicture}
417 + \begin{axis}[
418 + width=0.50\textwidth, height=6.2cm,
419 + title={\small (a) Activation functions},
420 + xlabel={$z$}, ylabel={$\varphi(z)$},
421 + xmin=-4, xmax=4, ymin=-1.6, ymax=4,
422 + grid=major, grid style={black!12},
423 + legend pos=north west,
424 + legend style={font=\scriptsize, fill=white, fill opacity=0.85,
425 + text opacity=1, draw=black!30},
426 + every axis plot/.append style={thick},
427 + samples=200, domain=-4:4,
428 + ]
429 + \addplot[cinput] {1/(1+exp(-x))};
430 + \addlegendentry{sigmoid}
431 + \addplot[chidden] {tanh(x)};
432 + \addlegendentry{$\tanh$}
433 + \addplot[coutput] {max(0,x)};
434 + \addlegendentry{ReLU}
435 + \addplot[cgate, dashed] {max(0,x) + 0.1*min(0,x)};
436 + \addlegendentry{Leaky ReLU}
437 + \addplot[cmem]
438 + {0.5*x*(1 + tanh(0.7978845608*(x + 0.044715*x^3)))};
439 + \addlegendentry{GELU}
440 + \end{axis}
441 + \end{tikzpicture}\hfill
442 + \begin{tikzpicture}
443 + \begin{axis}[
444 + width=0.50\textwidth, height=6.2cm,
445 + title={\small (b) Derivatives},
446 + xlabel={$z$}, ylabel={$\varphi'(z)$},
447 + xmin=-4, xmax=4, ymin=-0.25, ymax=1.35,
448 + grid=major, grid style={black!12},
449 + legend pos=north west,
450 + legend style={font=\scriptsize, fill=white, fill opacity=0.85,
451 + text opacity=1, draw=black!30},
452 + every axis plot/.append style={thick},
453 + samples=200, domain=-4:4,
454 + ]
455 + \addplot[cinput] {exp(-x)/((1+exp(-x))^2)};
456 + \addlegendentry{sigmoid$'$}
457 + \addplot[chidden] {1 - tanh(x)^2};
458 + \addlegendentry{$\tanh'$}
459 + \addplot[coutput] coordinates {(-4,0) (0,0)};
460 + \addlegendentry{ReLU$'$}
461 + \addplot[coutput, forget plot] coordinates {(0,1) (4,1)};
462 + \addplot[cgate, dashed] coordinates {(-4,0.1) (0,0.1)};
463 + \addlegendentry{Leaky ReLU$'$}
464 + \addplot[cgate, dashed, forget plot] coordinates {(0,1) (4,1)};
465 + \addplot[cmem]
466 + {0.5*(1 + tanh(0.7978845608*(x + 0.044715*x^3)))
467 + + x*0.3989422804*exp(-x^2/2)};
468 + \addlegendentry{GELU$'$}
469 + \end{axis}
470 + \end{tikzpicture}
471 + \caption{The five standard activation functions
472 + \eqref{eq:found-sigmoid}--\eqref{eq:found-gelu} (left) and their
473 + derivatives (right). Leaky ReLU is drawn with $\alpha = 0.1$ for
474 + visibility (in practice $\alpha = 0.01$ is typical). Note the
475 + saturation of sigmoid and $\tanh$ — their derivatives vanish for
476 + $\lvert z\rvert \gtrsim 4$ — against the constant unit slope of the
477 + ReLU family on the positive half-line, and the smooth non-monotonic
478 + profile of GELU.}
479 + \label{fig:found-activations}
480 +\end{figure}
481 +
482 +% ----------------------------------------------------------------------------
483 +\subsection{The Softmax Function}
484 +% ----------------------------------------------------------------------------
485 +
486 +Multiclass classification requires a vector-valued output layer that maps
487 +$K$ real \emph{logits} to a probability distribution over $K$ classes.
488 +This is the role of the softmax:
489 +\begin{equation}
490 + \softmax(\vect{z})_i \;=\; \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}},
491 + \qquad i = 1, \dots, K,
492 + \label{eq:found-softmax}
493 +\end{equation}
494 +whose outputs are positive and sum to one. Its Jacobian has the compact
495 +form
496 +\begin{equation}
497 + \frac{\partial\, \softmax(\vect{z})_i}{\partial z_j}
498 + \;=\;
499 + \softmax(\vect{z})_i \left(\delta_{ij} - \softmax(\vect{z})_j\right),
500 + \label{eq:found-softmax-jacobian}
501 +\end{equation}
502 +with $\delta_{ij}$ the Kronecker delta — an expression that will combine
503 +particularly gracefully with the cross-entropy loss in the next chapter.
504 +
505 +\begin{remark}[Shift invariance and temperature]
506 +Softmax is invariant to adding a constant to every logit,
507 +$\softmax(\vect{z} + c\vect{1}) = \softmax(\vect{z})$; numerically stable
508 +implementations exploit this by subtracting $\max_j z_j$ before
509 +exponentiating. Dividing the logits by a \emph{temperature} $T > 0$
510 +interpolates between a hard maximum ($T \to 0$) and the uniform
511 +distribution ($T \to \infty$).
512 +\end{remark}
513 +
514 +% ----------------------------------------------------------------------------
515 +\section{The Multilayer Perceptron}
516 +% ----------------------------------------------------------------------------
517 +
518 +A single neuron detects one linear feature; the multilayer perceptron
519 +composes entire \emph{layers} of them, each layer feeding the next.
520 +
521 +\begin{definition}[Multilayer perceptron]\label{def:found-mlp}
522 +An MLP with $L$ layers of widths $n_1, \dots, n_L$ on inputs of dimension
523 +$n_0$ is defined by weight matrices
524 +$\mat{W}^{(\ell)} \in \R^{n_\ell \times n_{\ell-1}}$ and bias vectors
525 +$\vect{b}^{(\ell)} \in \R^{n_\ell}$, $\ell = 1, \dots, L$. Setting
526 +$\vect{a}^{(0)} = \vect{x}$, the \emph{forward propagation} computes, for
527 +$\ell = 1, \dots, L$,
528 +\begin{align}
529 + \vect{z}^{(\ell)} &= \mat{W}^{(\ell)} \vect{a}^{(\ell-1)}
530 + + \vect{b}^{(\ell)},
531 + \label{eq:found-mlp-z}\\
532 + \vect{a}^{(\ell)} &= \varphi^{(\ell)}\!\left(\vect{z}^{(\ell)}\right),
533 + \label{eq:found-mlp-a}
534 +\end{align}
535 +where $\varphi^{(\ell)}$ acts componentwise, and the network output is
536 +$\hat{\vect{y}} = \vect{a}^{(L)}$.
537 +\end{definition}
538 +
539 +Componentwise, \eqref{eq:found-mlp-z} reads
540 +\begin{equation}
541 + z^{(\ell)}_j \;=\; \sum_{i=1}^{n_{\ell-1}}
542 + W^{(\ell)}_{ji}\, a^{(\ell-1)}_i + b^{(\ell)}_j,
543 + \label{eq:found-mlp-component}
544 +\end{equation}
545 +so that each unit of layer $\ell$ is exactly an artificial neuron in the
546 +sense of Definition~\ref{def:found-neuron}, whose inputs are the
547 +activations of the previous layer. The final activation
548 +$\varphi^{(L)}$ is chosen to match the task: identity for regression,
549 +sigmoid for binary classification, softmax~\eqref{eq:found-softmax} for
550 +multiclass classification, while hidden layers use one of the
551 +nonlinearities of Figure~\ref{fig:found-activations}. A network with two
552 +hidden layers is drawn in Figure~\ref{fig:found-mlp}.
553 +
554 +\begin{figure}[htbp]
555 + \centering
556 + \begin{tikzpicture}
557 + % --- input layer --------------------------------------------------------
558 + \foreach \i in {1,2,3}
559 + \node[ninput] (x\i) at (0, 2.2-\i*1.1) {$x_{\i}$};
560 + % --- hidden layer 1 -----------------------------------------------------
561 + \foreach \j in {1,...,4}
562 + \node[nhidden] (h1\j) at (2.7, 2.75-\j*1.1) {};
563 + % --- hidden layer 2 -----------------------------------------------------
564 + \foreach \j in {1,...,4}
565 + \node[nhidden] (h2\j) at (5.4, 2.75-\j*1.1) {};
566 + % --- output layer -------------------------------------------------------
567 + \foreach \k in {1,2}
568 + \node[noutput] (o\k) at (8.1, 1.1-\k*1.1) {};
569 + % --- connections ----------------------------------------------------------
570 + \foreach \i in {1,2,3} \foreach \j in {1,...,4}
571 + \draw[black!35, semithick] (x\i) -- (h1\j);
572 + \foreach \i in {1,...,4} \foreach \j in {1,...,4}
573 + \draw[black!35, semithick] (h1\i) -- (h2\j);
574 + \foreach \i in {1,...,4} \foreach \k in {1,2}
575 + \draw[black!35, semithick] (h2\i) -- (o\k);
576 + % --- outputs ---------------------------------------------------------------
577 + \draw[fleche] (o1) -- ++(1.3,0) node[right] {$\hat{y}_1$};
578 + \draw[fleche] (o2) -- ++(1.3,0) node[right] {$\hat{y}_2$};
579 + % --- weight-matrix labels ---------------------------------------------------
580 + \node[etiquette] at (1.35, 2.55) {$\mat{W}^{(1)}, \vect{b}^{(1)}$};
581 + \node[etiquette] at (4.05, 2.55) {$\mat{W}^{(2)}, \vect{b}^{(2)}$};
582 + \node[etiquette] at (6.75, 2.55) {$\mat{W}^{(3)}, \vect{b}^{(3)}$};
583 + % --- layer labels ---------------------------------------------------------
584 + \node[etiquette, align=center] at (0, -2.6)
585 + {input layer\\ $\vect{a}^{(0)} = \vect{x}$};
586 + \node[etiquette, align=center] at (2.7, -2.6)
587 + {hidden layer 1\\ $\vect{a}^{(1)}$};
588 + \node[etiquette, align=center] at (5.4, -2.6)
589 + {hidden layer 2\\ $\vect{a}^{(2)}$};
590 + \node[etiquette, align=center] at (8.1, -2.6)
591 + {output layer\\ $\hat{\vect{y}} = \vect{a}^{(3)}$};
592 + \end{tikzpicture}
593 + \caption{A fully connected network with two hidden layers
594 + ($n_0 = 3$, $n_1 = n_2 = 4$, $n_3 = 2$). Each edge carries one entry
595 + of a weight matrix $\mat{W}^{(\ell)}$; each column of neurons applies
596 + the affine map~\eqref{eq:found-mlp-z} followed by the pointwise
597 + nonlinearity~\eqref{eq:found-mlp-a}.}
598 + \label{fig:found-mlp}
599 +\end{figure}
600 +
601 +\begin{remark}[Batched computation]
602 +For a mini-batch of $m$ examples stacked as rows of
603 +$\mat{X} \in \R^{m \times n_0}$, forward propagation becomes
604 +$\mat{Z}^{(\ell)} = \mat{A}^{(\ell-1)} \mat{W}^{(\ell)\top}
605 ++ \vect{1}_m \vect{b}^{(\ell)\top}$ — a chain of dense matrix products.
606 +This is why MLPs map so efficiently onto modern hardware: the entire
607 +network is a sequence of GEMM primitives.
608 +\end{remark}
609 +
610 +\begin{remark}[Why the nonlinearity is essential]
611 +If $\varphi^{(\ell)} = \mathrm{id}$ for all $\ell$, then
612 +$\hat{\vect{y}} = \mat{W}^{(L)} \cdots \mat{W}^{(1)} \vect{x} + \vect{c}$
613 +for some constant $\vect{c}$: the composition of affine maps is affine,
614 +and depth buys nothing. All the expressive power of the MLP resides in the
615 +interleaving of~\eqref{eq:found-mlp-z} with the
616 +nonlinearity~\eqref{eq:found-mlp-a}.
617 +\end{remark}
618 +
619 +% ----------------------------------------------------------------------------
620 +\subsection{Universal Approximation}
621 +% ----------------------------------------------------------------------------
622 +
623 +How expressive is the MLP? The classical answer, due independently to
624 +Cybenko (1989) and to Hornik, Stinchcombe and White (1989), is that
625 +\emph{one} hidden layer already suffices, provided it is wide enough.
626 +
627 +\begin{property}[Universal approximation]\label{prop:found-uat}
628 +Let $\varphi$ be a continuous sigmoidal function
629 +($\varphi(t) \to 1$ as $t \to +\infty$ and $\varphi(t) \to 0$ as
630 +$t \to -\infty$). Then finite sums of the form
631 +\begin{equation}
632 + G(\vect{x}) \;=\; \sum_{j=1}^{N} \alpha_j\,
633 + \varphi\!\left(\vect{w}_j\transp \vect{x} + \theta_j\right)
634 + \label{eq:found-uat}
635 +\end{equation}
636 +are dense in $C([0,1]^{n})$ for the uniform norm: for every continuous
637 +$f : [0,1]^n \to \R$ and every $\varepsilon > 0$, there exist $N$ and
638 +parameters $\{\alpha_j, \vect{w}_j, \theta_j\}$ such that
639 +$\lvert G(\vect{x}) - f(\vect{x})\rvert < \varepsilon$ for all
640 +$\vect{x} \in [0,1]^n$. More generally (Leshno et al., 1993), a
641 +one-hidden-layer network with a locally bounded, piecewise-continuous
642 +activation is a universal approximator \emph{if and only if} the
643 +activation is not a polynomial.
644 +\end{property}
645 +
646 +The ``if and only if'' clause explains why ReLU — which is not sigmoidal —
647 +is nevertheless universal, and why a purely linear network is not. Two
648 +caveats temper the theorem's optimism, and both shape the rest of this
649 +book~\cite{goodfellow2016book}. First,
650 +Property~\ref{prop:found-uat} is an \emph{existence} result: it is silent
651 +on how many hidden units are required (the width $N$ may grow
652 +exponentially with the input dimension $n$) and on whether any learning
653 +algorithm will \emph{find} the approximating weights. Second, it concerns
654 +shallow networks only; depth-separation results show that certain
655 +functions computable by a deep network with polynomially many units
656 +require exponentially many units at bounded depth. Approximation theory
657 +thus motivates depth, but it is the training machinery of the next
658 +chapter — loss functions, backpropagation, and stochastic optimization —
659 +that makes depth usable in practice.
660 +
661 +% ----------------------------------------------------------------------------
662 +\section{A Contrast: Radial Basis Function Networks}
663 +% ----------------------------------------------------------------------------
664 +
665 +The MLP is not the only way to combine simple units into a universal
666 +approximator, and a brief look at its classical alternative sharpens our
667 +understanding of what makes the MLP distinctive. A \emph{radial basis
668 +function} (RBF) network, introduced by Broomhead and Lowe (1988) and
669 +refined by Moody and Darken (1989), has exactly one hidden layer of $J$
670 +\emph{locally tuned} units, each defined by a centre
671 +$\vect{\mu}_j \in \R^n$ and a width $\sigma_j > 0$, followed by a linear
672 +output layer:
673 +\begin{equation}
674 + f(\vect{x}) \;=\; \sum_{j=1}^{J} w_j\,
675 + \varphi_j(\vect{x}) + b,
676 + \qquad
677 + \varphi_j(\vect{x}) \;=\;
678 + \exp\!\left(
679 + -\frac{\lVert \vect{x} - \vect{\mu}_j \rVert^{2}}{2\sigma_j^{2}}
680 + \right).
681 + \label{eq:found-rbf}
682 +\end{equation}
683 +The contrast with Definition~\ref{def:found-neuron} is fundamental. An
684 +MLP unit computes an \emph{inner product} $\vect{w}\transp\vect{x}$ and
685 +responds along a hyperplane — a global, distributed representation. An
686 +RBF unit computes a \emph{distance} $\lVert\vect{x} - \vect{\mu}_j\rVert$
687 +and responds only in a localized neighbourhood of its centre — a local,
688 +spherical receptive field. Locality makes RBF networks fast to train:
689 +once the centres are placed (by random subsampling or $k$-means
690 +clustering), the model is \emph{linear} in the output weights, and with
691 +the design matrix $\Phi_{ij} = \varphi_j(\vect{x}^{(i)})$ and targets
692 +$\vect{y}$, the ridge-regularized least-squares solution is closed-form,
693 +\begin{equation}
694 + \vect{w} \;=\;
695 + \left(\mat{\Phi}\transp \mat{\Phi} + \lambda \mat{I}\right)^{-1}
696 + \mat{\Phi}\transp \vect{y}.
697 + \label{eq:found-rbf-ls}
698 +\end{equation}
699 +The price of locality is the curse of dimensionality: covering a
700 +high-dimensional input space with local bumps requires exponentially many
701 +centres, whereas the global half-space features of the MLP can be shared
702 +and composed. This trade-off — local interpolation versus global,
703 +composable features — anticipates a pattern that recurs throughout the
704 +book, and it is the composable option that deep learning has embraced.
705 +Everything now hinges on one question: how are the weights of a
706 +multilayer network actually learned? The next chapter answers it.
added chapters/02-training.tex +554 −0
@@ -0,0 +1,554 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 2 : Training Neural Networks (chapters/02-training.tex)
6 +% ============================================================================
7 +\chapter{Training Neural Networks}\label{chap:training}
8 +
9 +The previous chapter established what a feed-forward network computes; this
10 +chapter establishes how its parameters are found. Training is cast as the
11 +minimization of an empirical risk, and virtually every modern network is
12 +trained by the same recipe: compute the gradient of the loss with respect to
13 +every parameter by \emph{backpropagation}~\cite{rumelhart1986}, then descend
14 +along that gradient with a first-order update rule. We derive the
15 +backpropagation equations in full, present the algorithm in pseudocode, and
16 +then survey the family of optimizers — from plain stochastic gradient
17 +descent to Adam~\cite{kingma2015adam} — together with the learning-rate
18 +schedules and weight-initialization schemes that make deep training stable
19 +in practice.
20 +
21 +% ============================================================================
22 +\section{The Learning Problem}
23 +
24 +\begin{definition}[Empirical risk minimization]\label{def:train-erm}
25 +Let $f(\cdot\,;\vect{\theta})$ be a network with parameters $\vect{\theta}$,
26 +let $\Loss(\hat{\vect{y}},\vect{y})$ be a per-example loss, and let
27 +$\mathcal{D}$ be the data distribution. The \emph{(true) risk} and its
28 +Monte-Carlo estimate on a training set
29 +$\{(\vect{x}_i,\vect{y}_i)\}_{i=1}^{n}$, the \emph{empirical risk}, are
30 +\begin{equation}\label{eq:train-risk}
31 + R(\vect{\theta})
32 + = \E_{(\vect{x},\vect{y})\sim\mathcal{D}}
33 + \bigl[\Loss\bigl(f(\vect{x};\vect{\theta}),\vect{y}\bigr)\bigr],
34 + \qquad
35 + \widehat{R}(\vect{\theta})
36 + = \frac{1}{n}\sum_{i=1}^{n}
37 + \Loss\bigl(f(\vect{x}_i;\vect{\theta}),\vect{y}_i\bigr).
38 +\end{equation}
39 +Training solves
40 +$\vect{\theta}^{\star} \in \argmin_{\vect{\theta}} \widehat{R}(\vect{\theta})$.
41 +\end{definition}
42 +
43 +The choice of $\Loss$ encodes the task. We review the two workhorses —
44 +squared error for regression, cross-entropy for classification — and the
45 +single most useful gradient identity in deep learning.
46 +
47 +\subsection{Regression: mean squared error}
48 +
49 +For scalar targets $y_i \in \R$ and predictions $\hat{y}_i$, the
50 +\emph{mean squared error} and its gradient are
51 +\begin{equation}\label{eq:train-mse}
52 + \Loss_{\mathrm{MSE}}
53 + = \frac{1}{n}\sum_{i=1}^{n}\bigl(y_i - \hat{y}_i\bigr)^{2},
54 + \qquad
55 + \frac{\partial \Loss_{\mathrm{MSE}}}{\partial \hat{y}_i}
56 + = -\frac{2}{n}\,\bigl(y_i - \hat{y}_i\bigr).
57 +\end{equation}
58 +Minimizing~\eqref{eq:train-mse} is maximum-likelihood estimation under
59 +Gaussian observation noise; the quadratic growth makes it sensitive to
60 +outliers, which motivates robust alternatives such as the mean absolute
61 +error $\frac{1}{n}\sum_i |y_i-\hat y_i|$ (Laplace likelihood, whose
62 +minimizer is the conditional median) and the Huber loss, quadratic for
63 +small residuals and linear in the tails.
64 +
65 +\subsection{Classification: cross-entropy}
66 +
67 +For binary labels $y\in\{0,1\}$ with $\hat{y}=\sigma(z)$ produced by a
68 +sigmoid over the logit $z$, the \emph{binary cross-entropy} is
69 +\begin{equation}\label{eq:train-bce}
70 + \Loss_{\mathrm{BCE}}
71 + = -\frac{1}{n}\sum_{i=1}^{n}
72 + \Bigl[\,y_i\ln\hat{y}_i + (1-y_i)\ln\bigl(1-\hat{y}_i\bigr)\Bigr],
73 + \qquad
74 + \frac{\partial \Loss_{\mathrm{BCE}}}{\partial z} = \hat{y}-y .
75 +\end{equation}
76 +For $K$-way classification with one-hot target $\vect{y}$ and
77 +$\hat{\vect{y}}=\softmax(\vect{z})$, the \emph{categorical cross-entropy}
78 +is
79 +\begin{equation}\label{eq:train-ce}
80 + \Loss_{\mathrm{CE}}
81 + = -\sum_{k=1}^{K} y_k \ln \hat{y}_k ,
82 +\end{equation}
83 +which equals, up to the (constant) entropy of $\vect{y}$, the
84 +Kullback--Leibler divergence $\KL(\vect{y}\,\|\,\hat{\vect{y}})$.
85 +
86 +\begin{property}[Softmax--cross-entropy gradient]\label{prop:train-ce-grad}
87 +With $\hat{\vect{y}}=\softmax(\vect{z})$ and $\Loss_{\mathrm{CE}}$ as
88 +in~\eqref{eq:train-ce},
89 +\begin{equation}\label{eq:train-ce-grad}
90 + \frac{\partial \Loss_{\mathrm{CE}}}{\partial \vect{z}}
91 + \;=\; \hat{\vect{y}} - \vect{y}.
92 +\end{equation}
93 +\end{property}
94 +
95 +\begin{proof}[Sketch]
96 +The softmax Jacobian is
97 +$\partial\hat{y}_i/\partial z_j = \hat{y}_i(\delta_{ij}-\hat{y}_j)$ with
98 +$\delta_{ij}$ the Kronecker delta. Chaining it against
99 +$\partial\Loss/\partial\hat{y}_i = -y_i/\hat{y}_i$ gives
100 +$\partial\Loss/\partial z_j
101 + = \sum_i (-y_i/\hat{y}_i)\,\hat{y}_i(\delta_{ij}-\hat{y}_j)
102 + = -y_j + \hat{y}_j\sum_i y_i = \hat{y}_j - y_j$,
103 +since $\sum_i y_i = 1$.
104 +\end{proof}
105 +
106 +\begin{remark}
107 +The cancellation in~\eqref{eq:train-ce-grad} is not a coincidence: it holds
108 +for every matched pair of a canonical link and its exponential-family
109 +negative log-likelihood (sigmoid with binary cross-entropy
110 +in~\eqref{eq:train-bce}, identity with MSE). The practical consequence is
111 +the absence of saturating $\sigma'$ factors at the output layer: a
112 +confidently wrong prediction still receives a large gradient.
113 +\end{remark}
114 +
115 +% ============================================================================
116 +\section{Backpropagation}
117 +
118 +Consider the $L$-layer network of Chapter~\ref{chap:foundations},
119 +\begin{equation}\label{eq:train-forward}
120 + \vect{z}^{(\ell)} = \mat{W}^{(\ell)}\vect{a}^{(\ell-1)}+\vect{b}^{(\ell)},
121 + \qquad
122 + \vect{a}^{(\ell)} = \varphi\bigl(\vect{z}^{(\ell)}\bigr),
123 + \qquad \ell = 1,\dots,L,
124 +\end{equation}
125 +with $\vect{a}^{(0)}=\vect{x}$ and a scalar loss
126 +$\Loss\bigl(\vect{a}^{(L)},\vect{y}\bigr)$. Backpropagation is reverse-mode
127 +automatic differentiation applied to this composition: it computes
128 +$\partial\Loss/\partial\vect{\theta}$ for \emph{all} parameters at the cost
129 +of a constant number of forward passes. The central object is the
130 +\emph{error signal} of layer $\ell$,
131 +\begin{equation}\label{eq:train-delta-def}
132 + \vect{\delta}^{(\ell)}
133 + \;\equiv\; \frac{\partial \Loss}{\partial \vect{z}^{(\ell)}}
134 + \;\in\; \R^{n_\ell}.
135 +\end{equation}
136 +
137 +\begin{theorem}[Backpropagation equations~\cite{rumelhart1986}]
138 +\label{thm:train-backprop}
139 +For the network~\eqref{eq:train-forward}, the error signals and parameter
140 +gradients satisfy
141 +\begin{align}
142 + \vect{\delta}^{(L)}
143 + &= \nabla_{\vect{a}^{(L)}}\Loss
144 + \,\odot\, \varphi'\bigl(\vect{z}^{(L)}\bigr),
145 + \label{eq:train-bp-out}\\[2pt]
146 + \vect{\delta}^{(\ell)}
147 + &= \Bigl(\mat{W}^{(\ell+1)\transp}\,\vect{\delta}^{(\ell+1)}\Bigr)
148 + \odot \varphi'\bigl(\vect{z}^{(\ell)}\bigr),
149 + \quad \ell = L-1,\dots,1,
150 + \label{eq:train-bp-rec}\\[2pt]
151 + \frac{\partial \Loss}{\partial \mat{W}^{(\ell)}}
152 + &= \vect{\delta}^{(\ell)}\,\vect{a}^{(\ell-1)\transp},
153 + \qquad
154 + \frac{\partial \Loss}{\partial \vect{b}^{(\ell)}}
155 + = \vect{\delta}^{(\ell)} .
156 + \label{eq:train-bp-w}
157 +\end{align}
158 +\end{theorem}
159 +
160 +\begin{proof}[Sketch]
161 +Equation~\eqref{eq:train-bp-out} is the chain rule through
162 +$\vect{a}^{(L)}=\varphi(\vect{z}^{(L)})$. For~\eqref{eq:train-bp-rec},
163 +each $z^{(\ell+1)}_k$ depends on $z^{(\ell)}_j$ through
164 +$W^{(\ell+1)}_{kj}\varphi'(z^{(\ell)}_j)$, so summing over downstream
165 +paths,
166 +\[
167 +\delta^{(\ell)}_j
168 + = \sum_k \delta^{(\ell+1)}_k W^{(\ell+1)}_{kj}\,
169 + \varphi'\bigl(z^{(\ell)}_j\bigr),
170 +\]
171 +which is~\eqref{eq:train-bp-rec} in matrix form.
172 +For~\eqref{eq:train-bp-w}, note
173 +$\partial z^{(\ell)}_j / \partial W^{(\ell)}_{ji} = a^{(\ell-1)}_i$ and
174 +$\partial z^{(\ell)}_j / \partial b^{(\ell)}_j = 1$.
175 +\end{proof}
176 +
177 +The name of the algorithm is visible in~\eqref{eq:train-bp-rec}: the error
178 +is propagated \emph{backwards} through the transposes of the forward weight
179 +matrices. With softmax output and cross-entropy loss,
180 +Property~\ref{prop:train-ce-grad} replaces~\eqref{eq:train-bp-out} directly
181 +by $\vect{\delta}^{(L)} = \hat{\vect{y}}-\vect{y}$.
182 +Figure~\ref{fig:train-compgraph} shows the flow of both passes on the
183 +computational graph, and Algorithm~\ref{alg:train-backprop} states the full
184 +procedure for one mini-batch.
185 +
186 +\begin{figure}[htbp]
187 + \centering
188 + \begin{tikzpicture}
189 + % ---- forward row -------------------------------------------------
190 + \node[blocinput, minimum width=1.15cm] (x) at (0,0) {$\vect{x}$};
191 + \node[blochidden, minimum width=1.15cm] (z1) at (2.35,0) {$\vect{z}^{(1)}$};
192 + \node[blochidden, minimum width=1.15cm] (a1) at (4.70,0) {$\vect{a}^{(1)}$};
193 + \node[blochidden, minimum width=1.15cm] (z2) at (7.05,0) {$\vect{z}^{(2)}$};
194 + \node[blocoutput, minimum width=1.15cm] (a2) at (9.40,0) {$\hat{\vect{y}}$};
195 + \node[bloc, minimum width=1.15cm] (Ls) at (11.75,0){$\Loss$};
196 + % ---- parameter and target nodes ------------------------------------
197 + \node[mem, minimum width=1.5cm] (W1) at (2.35,1.9)
198 + {$\mat{W}^{(1)},\vect{b}^{(1)}$};
199 + \node[mem, minimum width=1.5cm] (W2) at (7.05,1.9)
200 + {$\mat{W}^{(2)},\vect{b}^{(2)}$};
201 + \node[blocinput, minimum width=1.15cm] (y) at (11.75,1.9) {$\vect{y}$};
202 + % ---- forward arrows ------------------------------------------------
203 + \draw[fleche] (x) -- (z1);
204 + \draw[fleche] (z1) -- node[above, etiquette] {$\varphi$} (a1);
205 + \draw[fleche] (a1) -- (z2);
206 + \draw[fleche] (z2) -- node[above, etiquette] {$\varphi$} (a2);
207 + \draw[fleche] (a2) -- (Ls);
208 + \draw[fleche] (W1) -- (z1);
209 + \draw[fleche] (W2) -- (z2);
210 + \draw[fleche] (y) -- (Ls);
211 + % ---- backward (dashed) arrows, routed below ------------------------
212 + \draw[flechep, coutput] (Ls.south) to[bend left=35]
213 + node[below, etiquette] {$\nabla_{\hat{\vect{y}}}\Loss$} (a2.south);
214 + \draw[flechep, coutput] (a2.south) to[bend left=35]
215 + node[below, etiquette] {$\vect{\delta}^{(2)}$} (z2.south);
216 + \draw[flechep, coutput] (z2.south) to[bend left=35]
217 + node[below, etiquette] {$\mat{W}^{(2)\transp}\vect{\delta}^{(2)}$}
218 + (a1.south);
219 + \draw[flechep, coutput] (a1.south) to[bend left=35]
220 + node[below, etiquette] {$\vect{\delta}^{(1)}$} (z1.south);
221 + % ---- gradients to parameters ---------------------------------------
222 + \draw[flechep, coutput] (z1.north east) to[bend right=30]
223 + node[right, etiquette, xshift=2pt]
224 + {$\vect{\delta}^{(1)}\vect{a}^{(0)\transp}$} (W1.east);
225 + \draw[flechep, coutput] (z2.north east) to[bend right=30]
226 + node[right, etiquette, xshift=2pt]
227 + {$\vect{\delta}^{(2)}\vect{a}^{(1)\transp}$} (W2.east);
228 + \end{tikzpicture}
229 + \caption{Computational graph of a two-layer network. Solid arrows:
230 + forward pass~\eqref{eq:train-forward}. Dashed red arrows: backward
231 + pass — the error signals $\vect{\delta}^{(\ell)}$ of
232 + Theorem~\ref{thm:train-backprop} flow from the loss back through the
233 + layers, branching off to the parameter
234 + gradients~\eqref{eq:train-bp-w}.}
235 + \label{fig:train-compgraph}
236 +\end{figure}
237 +
238 +\begin{algorithm}[htbp]
239 + \caption{Backpropagation with mini-batch gradient descent}
240 + \label{alg:train-backprop}
241 + \begin{algorithmic}[1]
242 + \Require mini-batch $\{(\vect{x}_i,\vect{y}_i)\}_{i=1}^{m}$, parameters
243 + $\{\mat{W}^{(\ell)},\vect{b}^{(\ell)}\}_{\ell=1}^{L}$, learning rate
244 + $\eta$
245 + \For{$i = 1$ \textbf{to} $m$} \Comment{forward pass}
246 + \State $\vect{a}^{(0)} \gets \vect{x}_i$
247 + \For{$\ell = 1$ \textbf{to} $L$}
248 + \State $\vect{z}^{(\ell)} \gets
249 + \mat{W}^{(\ell)}\vect{a}^{(\ell-1)} + \vect{b}^{(\ell)}$;
250 + \quad $\vect{a}^{(\ell)} \gets \varphi(\vect{z}^{(\ell)})$
251 + \EndFor
252 + \State $\vect{\delta}^{(L)} \gets
253 + \nabla_{\vect{a}^{(L)}}\Loss \odot \varphi'(\vect{z}^{(L)})$
254 + \Comment{backward pass; $\hat{\vect{y}}-\vect{y}$ for softmax+CE}
255 + \For{$\ell = L-1$ \textbf{down to} $1$}
256 + \State $\vect{\delta}^{(\ell)} \gets
257 + \bigl(\mat{W}^{(\ell+1)\transp}\vect{\delta}^{(\ell+1)}\bigr)
258 + \odot \varphi'(\vect{z}^{(\ell)})$
259 + \EndFor
260 + \State accumulate
261 + $\Delta\mat{W}^{(\ell)} \mathrel{+}=
262 + \vect{\delta}^{(\ell)}\vect{a}^{(\ell-1)\transp}$,
263 + \; $\Delta\vect{b}^{(\ell)} \mathrel{+}= \vect{\delta}^{(\ell)}$
264 + \textbf{for all} $\ell$
265 + \EndFor
266 + \For{$\ell = 1$ \textbf{to} $L$} \Comment{gradient step}
267 + \State $\mat{W}^{(\ell)} \gets
268 + \mat{W}^{(\ell)} - \dfrac{\eta}{m}\,\Delta\mat{W}^{(\ell)}$;
269 + \quad
270 + $\vect{b}^{(\ell)} \gets
271 + \vect{b}^{(\ell)} - \dfrac{\eta}{m}\,\Delta\vect{b}^{(\ell)}$
272 + \EndFor
273 + \end{algorithmic}
274 +\end{algorithm}
275 +
276 +\begin{remark}[Cost]
277 +One backward pass costs the same order as one forward pass,
278 +$O\bigl(\sum_\ell n_\ell n_{\ell-1}\bigr)$: the full gradient of a scalar
279 +with respect to $P$ parameters is obtained for $O(1)$ — not $O(P)$ —
280 +forward-pass equivalents. This efficiency of reverse-mode differentiation
281 +is what makes deep learning computationally feasible.
282 +\end{remark}
283 +
284 +\begin{remark}[Vanishing and exploding gradients]\label{rem:train-vanish}
285 +The recursion~\eqref{eq:train-bp-rec} multiplies a $\varphi'$ factor and a
286 +weight matrix at every layer. With sigmoid activations,
287 +$\sigma'(z)\le 1/4$, so error signals shrink at least geometrically with
288 +depth; with large weights they can instead grow without bound. This
289 +vanishing/exploding behaviour motivates ReLU-family activations, careful
290 +initialization (Section~\ref{sec:train-init}), normalization layers and
291 +residual connections, treated in later chapters.
292 +\end{remark}
293 +
294 +% ============================================================================
295 +\section{First-Order Optimizers}
296 +
297 +Throughout this section $\vect{\theta}_t$ denotes the parameters at step
298 +$t$, $\vect{g}_t = \nabla_{\vect{\theta}}\Loss(\vect{\theta}_t)$ the
299 +mini-batch gradient, $\eta$ the learning rate, and all operations on
300 +vectors are elementwise.
301 +
302 +\subsection{Stochastic gradient descent and momentum}
303 +
304 +\emph{Stochastic gradient descent} (SGD) applies the elementary update
305 +\begin{equation}\label{eq:train-sgd}
306 + \vect{\theta}_{t+1} = \vect{\theta}_t - \eta\,\vect{g}_t .
307 +\end{equation}
308 +Classical stochastic-approximation theory guarantees convergence when the
309 +step sizes satisfy $\sum_t \eta_t = \infty$ and $\sum_t \eta_t^2 < \infty$
310 +\cite{goodfellow2016book}.
311 +
312 +\emph{Momentum} (Polyak's heavy ball) accumulates an exponentially weighted
313 +velocity,
314 +\begin{equation}\label{eq:train-momentum}
315 + \vect{v}_t = \beta\,\vect{v}_{t-1} + \vect{g}_t ,
316 + \qquad
317 + \vect{\theta}_{t+1} = \vect{\theta}_t - \eta\,\vect{v}_t ,
318 +\end{equation}
319 +with $\beta \approx 0.9$. Directions in which successive gradients agree
320 +are amplified by up to $1/(1-\beta)$, while oscillating components cancel —
321 +precisely the geometry of Figure~\ref{fig:train-trajectories}.
322 +\emph{Nesterov's accelerated gradient} evaluates the gradient at a
323 +look-ahead point,
324 +\begin{equation}\label{eq:train-nesterov}
325 + \vect{v}_t = \beta\,\vect{v}_{t-1}
326 + + \nabla_{\vect{\theta}}
327 + \Loss\bigl(\vect{\theta}_t - \eta\beta\,\vect{v}_{t-1}\bigr),
328 + \qquad
329 + \vect{\theta}_{t+1} = \vect{\theta}_t - \eta\,\vect{v}_t ,
330 +\end{equation}
331 +letting the update ``see'' where it is heading and correct in advance; for
332 +smooth convex objectives it attains the optimal $O(1/t^2)$ convergence
333 +rate.
334 +
335 +\begin{figure}[htbp]
336 + \centering
337 + \begin{tikzpicture}
338 + \begin{axis}[
339 + width=0.88\textwidth, height=6.2cm,
340 + xmin=-10, xmax=1.5, ymin=-2.4, ymax=2.4,
341 + xlabel={$\theta_1$}, ylabel={$\theta_2$},
342 + xlabel near ticks, ylabel near ticks,
343 + tick label style={font=\scriptsize},
344 + label style={font=\small},
345 + legend style={font=\scriptsize, at={(0.98,0.04)},
346 + anchor=south east, draw=black!30},
347 + legend cell align=left,
348 + ]
349 + % loss contours of f = x^2/20 + y^2 (anisotropic bowl)
350 + \addplot[domain=0:360, samples=91, smooth, black!25, forget plot]
351 + ({2.828*cos(x)}, {0.632*sin(x)});
352 + \addplot[domain=0:360, samples=91, smooth, black!25, forget plot]
353 + ({4.899*cos(x)}, {1.095*sin(x)});
354 + \addplot[domain=0:360, samples=91, smooth, black!25, forget plot]
355 + ({6.928*cos(x)}, {1.549*sin(x)});
356 + \addplot[domain=0:360, samples=91, smooth, black!25, forget plot]
357 + ({8.944*cos(x)}, {2.000*sin(x)});
358 + \addplot[domain=0:360, samples=91, smooth, black!25, forget plot]
359 + ({11.662*cos(x)}, {2.608*sin(x)});
360 + % SGD trajectory (zigzag)
361 + \addplot[coutput, thick, mark=*, mark size=1.1pt] coordinates {
362 + (-8.500,1.800) (-7.777,-1.260) (-7.116,0.882) (-6.512,-0.617)
363 + (-5.958,0.432) (-5.452,-0.303) (-4.988,0.212) (-4.564,-0.148)
364 + (-4.176,0.104) (-3.821,-0.073) (-3.496,0.051) (-3.199,-0.036)
365 + (-2.927,0.025) (-2.679,-0.017) (-2.451,0.012) (-2.243,-0.009)
366 + (-2.052,0.006)
367 + };
368 + \addlegendentry{SGD~\eqref{eq:train-sgd}}
369 + % Momentum trajectory (smooth)
370 + \addplot[cinput, thick, mark=*, mark size=1.1pt] coordinates {
371 + (-8.500,1.800) (-8.381,1.296) (-8.157,0.480) (-7.840,-0.390)
372 + (-7.446,-1.063) (-6.987,-1.371) (-6.476,-1.264) (-5.925,-0.815)
373 + (-5.347,-0.182) (-4.751,0.439) (-4.149,0.874) (-3.548,1.022)
374 + (-2.959,0.868) (-2.386,0.487) (-1.838,0.007) (-1.318,-0.426)
375 + (-0.832,-0.697)
376 + };
377 + \addlegendentry{Momentum~\eqref{eq:train-momentum}}
378 + % optimum
379 + \addplot[only marks, mark=star, mark size=3.2pt, black]
380 + coordinates {(0,0)};
381 + \addlegendentry{minimum $\vect{\theta}^{\star}$}
382 + \end{axis}
383 + \end{tikzpicture}
384 + \caption{Sixteen steps of SGD and momentum on the anisotropic quadratic
385 + $\Loss(\vect{\theta}) = \theta_1^2/20 + \theta_2^2$ (grey level sets),
386 + both computed numerically from the same starting point. SGD (red)
387 + oscillates across the narrow valley while creeping along the shallow
388 + direction; momentum (blue) damps the oscillation and accelerates along
389 + the valley floor.}
390 + \label{fig:train-trajectories}
391 +\end{figure}
392 +
393 +\subsection{Adaptive methods: AdaGrad, RMSProp, Adam}
394 +
395 +Adaptive methods give every coordinate its own effective learning rate,
396 +scaled by the history of gradient magnitudes. \emph{AdaGrad} accumulates
397 +the squared gradients,
398 +\begin{equation}\label{eq:train-adagrad}
399 + \vect{G}_t = \vect{G}_{t-1} + \vect{g}_t^{2},
400 + \qquad
401 + \vect{\theta}_{t+1}
402 + = \vect{\theta}_t
403 + - \frac{\eta}{\sqrt{\vect{G}_t}+\epsilon}\odot\vect{g}_t ,
404 +\end{equation}
405 +so rarely active (sparse) coordinates receive large steps. Because
406 +$\vect{G}_t$ grows monotonically, however, the effective step size decays
407 +to zero. \emph{RMSProp} repairs this by replacing the sum with an
408 +exponential moving average,
409 +\begin{equation}\label{eq:train-rmsprop}
410 + \E[\vect{g}^2]_t
411 + = \rho\,\E[\vect{g}^2]_{t-1} + (1-\rho)\,\vect{g}_t^{2},
412 + \qquad
413 + \vect{\theta}_{t+1}
414 + = \vect{\theta}_t
415 + - \frac{\eta}{\sqrt{\E[\vect{g}^2]_t}+\epsilon}\odot\vect{g}_t ,
416 +\end{equation}
417 +with $\rho\approx 0.9$, so that stale gradients are forgotten.
418 +
419 +\emph{Adam}~\cite{kingma2015adam} (\emph{adaptive moment estimation})
420 +combines the momentum idea~\eqref{eq:train-momentum} — an exponential
421 +moving average of the gradient, the first moment — with the RMSProp
422 +idea~\eqref{eq:train-rmsprop} — an exponential moving average of its
423 +square, the second moment — and corrects the initialization bias of both:
424 +\begin{align}
425 + \vect{m}_t &= \beta_1\,\vect{m}_{t-1} + (1-\beta_1)\,\vect{g}_t ,
426 + \qquad
427 + \vect{v}_t = \beta_2\,\vect{v}_{t-1} + (1-\beta_2)\,\vect{g}_t^{2},
428 + \label{eq:train-adam-moments}\\[2pt]
429 + \hat{\vect{m}}_t &= \frac{\vect{m}_t}{1-\beta_1^{\,t}} ,
430 + \qquad
431 + \hat{\vect{v}}_t = \frac{\vect{v}_t}{1-\beta_2^{\,t}} ,
432 + \label{eq:train-adam-bias}\\[2pt]
433 + \vect{\theta}_{t+1}
434 + &= \vect{\theta}_t
435 + - \eta\,\frac{\hat{\vect{m}}_t}{\sqrt{\hat{\vect{v}}_t}+\epsilon} .
436 + \label{eq:train-adam-update}
437 +\end{align}
438 +The standard defaults are $\eta = 10^{-3}$, $\beta_1 = 0.9$,
439 +$\beta_2 = 0.999$ and $\epsilon = 10^{-8}$.
440 +
441 +\begin{remark}[Why the bias correction matters]
442 +Since $\vect{m}_0=\vect{v}_0=\vect{0}$, the raw averages
443 +in~\eqref{eq:train-adam-moments} are biased toward zero for small $t$:
444 +$\E[\vect{m}_t]\approx(1-\beta_1^{\,t})\,\E[\vect{g}_t]$. Dividing by
445 +$(1-\beta_1^{\,t})$ and $(1-\beta_2^{\,t})$
446 +in~\eqref{eq:train-adam-bias} removes this bias exactly; without it the
447 +first updates would be far too small — dramatically so for
448 +$\vect{v}_t$, whose decay rate $\beta_2 = 0.999$ makes the bias persist
449 +for roughly a thousand steps. The ratio
450 +$\hat{\vect{m}}_t/\sqrt{\hat{\vect{v}}_t}$ acts as a per-coordinate
451 +signal-to-noise estimate, and the magnitude of each update is bounded by
452 +approximately $\eta$ regardless of the gradient scale, making
453 +Adam invariant to gradient rescaling. The decoupled-weight-decay variant
454 +AdamW, which applies the $L_2$ shrinkage outside the adaptive rescaling,
455 +is the default optimizer for modern Transformer models.
456 +\end{remark}
457 +
458 +% ============================================================================
459 +\section{Learning-Rate Schedules}
460 +
461 +The learning rate is the single most important hyperparameter of the
462 +updates above, and it is rarely held constant. Writing $\eta_t$ for the
463 +rate at step $t$ over a horizon of $T$ steps, the three standard schedules
464 +are
465 +\begin{align}
466 + \eta_t &= \eta_0\,\gamma^{\lfloor t/s \rfloor},
467 + \qquad 0<\gamma<1
468 + &&\text{(step decay, factor $\gamma$ every $s$ steps)},
469 + \label{eq:train-step-decay}\\[2pt]
470 + \eta_t &= \frac{\eta_0}{2}
471 + \Bigl(1+\cos\frac{\pi t}{T}\Bigr)
472 + &&\text{(cosine annealing)},
473 + \label{eq:train-cosine}\\[2pt]
474 + \eta_t &=
475 + \begin{cases}
476 + \eta_0\, t/T_w & t \le T_w\\[2pt]
477 + \dfrac{\eta_0}{2}
478 + \Bigl(1+\cos\dfrac{\pi (t-T_w)}{T-T_w}\Bigr) & t > T_w
479 + \end{cases}
480 + &&\text{(linear warmup, then cosine)}.
481 + \label{eq:train-warmup}
482 +\end{align}
483 +Warmup~\eqref{eq:train-warmup} protects the early phase of training — when
484 +Adam's second-moment estimate $\hat{\vect{v}}_t$ is still noisy — from
485 +destructively large steps, and is standard practice for Transformers; the
486 +three profiles are compared in Figure~\ref{fig:train-schedules}.
487 +
488 +\begin{figure}[htbp]
489 + \centering
490 + \begin{tikzpicture}
491 + \begin{axis}[
492 + width=0.8\textwidth, height=5.2cm,
493 + xmin=0, xmax=100, ymin=0, ymax=0.115,
494 + xlabel={training step $t$ (\% of horizon $T$)},
495 + ylabel={$\eta_t$},
496 + xlabel near ticks, ylabel near ticks,
497 + tick label style={font=\scriptsize},
498 + label style={font=\small},
499 + legend style={font=\scriptsize, draw=black!30},
500 + legend cell align=left,
501 + ]
502 + \addplot[coutput, thick, const plot, domain=0:100, samples=201]
503 + {0.1 * 0.5^(floor(x/30))};
504 + \addlegendentry{step decay~\eqref{eq:train-step-decay}}
505 + \addplot[cinput, thick, domain=0:100, samples=201]
506 + {0.05*(1+cos(deg(pi*x/100)))};
507 + \addlegendentry{cosine annealing~\eqref{eq:train-cosine}}
508 + \addplot[cgate, thick, domain=0:10, samples=21, forget plot]
509 + {0.1*x/10};
510 + \addplot[cgate, thick, domain=10:100, samples=181]
511 + {0.05*(1+cos(deg(pi*(x-10)/90)))};
512 + \addlegendentry{warmup + cosine~\eqref{eq:train-warmup}}
513 + \end{axis}
514 + \end{tikzpicture}
515 + \caption{Learning-rate schedules with $\eta_0=0.1$: step decay
516 + ($\gamma=0.5$, $s=0.3\,T$), cosine annealing, and linear warmup over
517 + the first $10\%$ of training followed by cosine annealing.}
518 + \label{fig:train-schedules}
519 +\end{figure}
520 +
521 +% ============================================================================
522 +\section{Weight Initialization}\label{sec:train-init}
523 +
524 +Remark~\ref{rem:train-vanish} showed that signals are multiplied by a
525 +weight matrix at every layer; initialization must therefore keep the
526 +variance of activations and of backpropagated gradients approximately
527 +constant with depth~\cite{goodfellow2016book}. For a layer with
528 +$n_{\mathrm{in}}$ inputs and $n_{\mathrm{out}}$ outputs,
529 +\emph{Xavier/Glorot} initialization — appropriate for symmetric, roughly
530 +linear-around-zero activations such as $\tanh$ — balances both passes:
531 +\begin{equation}\label{eq:train-xavier}
532 + \operatorname{Var}\bigl(W_{ij}\bigr)
533 + = \frac{2}{n_{\mathrm{in}}+n_{\mathrm{out}}},
534 + \qquad\text{e.g.}\quad
535 + W_{ij} \sim
536 + \mathcal{U}\!\left[
537 + -\sqrt{\tfrac{6}{n_{\mathrm{in}}+n_{\mathrm{out}}}},\;
538 + \sqrt{\tfrac{6}{n_{\mathrm{in}}+n_{\mathrm{out}}}}
539 + \right].
540 +\end{equation}
541 +\emph{He/Kaiming} initialization corrects for the fact that ReLU zeroes
542 +half of its inputs, which halves the activation variance at each layer:
543 +\begin{equation}\label{eq:train-he}
544 + \operatorname{Var}\bigl(W_{ij}\bigr) = \frac{2}{n_{\mathrm{in}}},
545 + \qquad
546 + W_{ij} \sim
547 + \mathcal{N}\!\Bigl(0,\; \tfrac{2}{n_{\mathrm{in}}}\Bigr),
548 +\end{equation}
549 +and is the default for ReLU-family networks. Biases are initialized to
550 +zero in both schemes. Together with the schedules of
551 +Figure~\ref{fig:train-schedules} and an adaptive optimizer such as
552 +Adam~\eqref{eq:train-adam-moments}--\eqref{eq:train-adam-update}, these
553 +choices form the standard modern training recipe on which the
554 +regularization techniques of the next chapter are layered.
added chapters/03-regularization.tex +407 −0
@@ -0,0 +1,407 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 3 : Regularization (chapters/03-regularization.tex)
6 +% ============================================================================
7 +\chapter{Regularization}\label{chap:regularization}
8 +
9 +A network that fits its training set perfectly is not, in general, a good
10 +network: what we actually care about is its behaviour on data it has never
11 +seen. Deep networks are typically over-parameterized — they possess enough
12 +capacity to memorize their training set outright — so the gap between
13 +training error and test error must be controlled deliberately.
14 +\emph{Regularization} is the collective name for the techniques that
15 +control this gap. Following Goodfellow, Bengio and
16 +Courville~\cite{goodfellow2016book}, we use the term broadly: a
17 +regularizer is any modification of the model, the objective or the
18 +training procedure whose purpose is to reduce generalization error, even
19 +at the price of a higher training error.
20 +
21 +\begin{definition}[Regularization]\label{def:reg}
22 +Let $\Loss_{\mathrm{data}}(\vect{\theta})$ denote the empirical loss of a
23 +model with parameters $\vect{\theta}$. A regularization technique is any
24 +alteration of the learning problem — an additive penalty
25 +$\Loss_{\mathrm{data}} + \lambda\,\Omega(\vect{\theta})$, a stochastic
26 +perturbation of the architecture, a normalization of intermediate
27 +activations, or a constraint on the training trajectory — designed to
28 +decrease the expected loss on unseen data drawn from the same
29 +distribution.
30 +\end{definition}
31 +
32 +\section{The Bias--Variance Decomposition}
33 +
34 +Why should reducing capacity ever help? The classical answer is the
35 +bias--variance decomposition. Consider a regression setting with targets
36 +generated as $y = f(\vect{x}) + \epsilon$, where $\epsilon$ is zero-mean
37 +noise of variance $\sigma^2$, and let $\hat{f}(\vect{x}; \mathcal{D})$ be
38 +the predictor learned from a random training set $\mathcal{D}$. The
39 +expected squared error at a point $\vect{x}$, averaged over draws of the
40 +training set and of the noise, splits into three terms:
41 +\begin{equation}\label{eq:reg-biasvariance}
42 +\E_{\mathcal{D},\epsilon}\!\left[\bigl(y - \hat{f}(\vect{x};\mathcal{D})\bigr)^2\right]
43 +=
44 +\underbrace{\bigl(f(\vect{x}) - \E_{\mathcal{D}}[\hat{f}(\vect{x};\mathcal{D})]\bigr)^2}_{\text{bias}^2}
45 ++
46 +\underbrace{\E_{\mathcal{D}}\!\left[\bigl(\hat{f}(\vect{x};\mathcal{D}) - \E_{\mathcal{D}}[\hat{f}(\vect{x};\mathcal{D})]\bigr)^2\right]}_{\text{variance}}
47 ++
48 +\underbrace{\sigma^2}_{\text{noise}} .
49 +\end{equation}
50 +The last term is irreducible: no learner can predict the noise. The first
51 +two trade off against each other. A rigid model (high bias) misses
52 +structure in the data; a flexible model fitted to a finite sample (high
53 +variance) tracks the accidents of that particular sample. Every technique
54 +in this chapter can be read as a way of purchasing a large reduction in
55 +variance at the cost of a small increase in bias, so that the sum
56 +in~\eqref{eq:reg-biasvariance} decreases.
57 +
58 +\section{Norm Penalties: $L_2$ and $L_1$}
59 +
60 +\subsection{$L_2$ regularization (weight decay)}
61 +
62 +The most venerable regularizer adds the squared Frobenius norm of the
63 +weight matrices to the data loss:
64 +\begin{equation}\label{eq:reg-l2loss}
65 +\Loss_{\mathrm{total}}(\vect{\theta})
66 += \Loss_{\mathrm{data}}(\vect{\theta})
67 ++ \frac{\lambda}{2} \sum_{\ell} \bigl\lVert \mat{W}^{(\ell)} \bigr\rVert_F^2 ,
68 +\qquad
69 +\frac{\partial \Loss_{\mathrm{total}}}{\partial \mat{W}^{(\ell)}}
70 += \frac{\partial \Loss_{\mathrm{data}}}{\partial \mat{W}^{(\ell)}}
71 ++ \lambda \mat{W}^{(\ell)} .
72 +\end{equation}
73 +Substituting the penalized gradient into the SGD step with learning rate
74 +$\eta$ reveals why the method is called \emph{weight decay}:
75 +\begin{equation}\label{eq:reg-l2update}
76 +\mat{W}^{(\ell)} \;\leftarrow\; (1 - \eta\lambda)\,\mat{W}^{(\ell)}
77 +\;-\; \eta\, \frac{\partial \Loss_{\mathrm{data}}}{\partial \mat{W}^{(\ell)}} .
78 +\end{equation}
79 +Each update first shrinks every weight multiplicatively by the factor
80 +$(1-\eta\lambda) < 1$, then applies the usual data-driven correction.
81 +Weights that the data does not actively support are steadily pulled toward
82 +zero. Biases are conventionally excluded from the penalty: they do not
83 +multiply inputs, so penalizing them adds bias without reducing variance.
84 +
85 +\begin{remark}[Bayesian reading]\label{rem:reg-bayes}
86 +Minimizing~\eqref{eq:reg-l2loss} is maximum a posteriori estimation under
87 +a zero-mean Gaussian prior on the weights; the $L_1$ penalty
88 +of~\eqref{eq:reg-l1loss} corresponds in the same way to a Laplace prior.
89 +The regularization strength $\lambda$ plays the role of the inverse prior
90 +variance: the stronger our prior belief that weights are small, the harder
91 +we shrink.
92 +\end{remark}
93 +
94 +\subsection{$L_1$ regularization and sparsity}
95 +
96 +Replacing the squared norm by the absolute-value norm changes the
97 +character of the solution:
98 +\begin{equation}\label{eq:reg-l1loss}
99 +\Loss_{\mathrm{total}}(\vect{\theta})
100 += \Loss_{\mathrm{data}}(\vect{\theta})
101 ++ \lambda \sum_{\ell} \bigl\lVert \mat{W}^{(\ell)} \bigr\rVert_1 ,
102 +\qquad
103 +\frac{\partial \Loss_{\mathrm{total}}}{\partial \mat{W}^{(\ell)}}
104 += \frac{\partial \Loss_{\mathrm{data}}}{\partial \mat{W}^{(\ell)}}
105 ++ \lambda \operatorname{sign}\bigl(\mat{W}^{(\ell)}\bigr) .
106 +\end{equation}
107 +The penalty gradient has \emph{constant magnitude} $\lambda$ regardless of
108 +how small a weight already is, so weights whose data gradient cannot
109 +sustain them are driven exactly to zero rather than merely toward it. The
110 +result is a sparse network — an implicit form of feature selection. The
111 +two penalties are often combined (the \emph{elastic net}),
112 +$\lambda_1 \lVert \mat{W} \rVert_1 + \tfrac{\lambda_2}{2}\lVert \mat{W} \rVert_2^2$,
113 +retaining the sparsity of $L_1$ and the grouping stability of $L_2$.
114 +
115 +\section{Dropout}
116 +
117 +Norm penalties act on parameters; \emph{dropout}, introduced by Srivastava
118 +et al.~\cite{srivastava2014}, acts on the architecture itself. During
119 +training, each unit of a layer is deleted independently at random, so that
120 +no unit can rely on the presence of any particular other unit — the
121 +phenomenon the authors call \emph{co-adaptation} is thereby suppressed.
122 +
123 +Formally, let $\vect{a}^{(\ell)}$ be the activation vector of layer $\ell$
124 +and let $p$ be the dropout rate (the probability of deletion). A binary
125 +mask is sampled anew for every training example, and the surviving
126 +activations are rescaled — the \emph{inverted dropout} convention:
127 +\begin{equation}\label{eq:reg-dropout-mask}
128 +m_j^{(\ell)} \sim \mathrm{Bernoulli}(1-p),
129 +\qquad
130 +\tilde{\vect{a}}^{(\ell)}
131 += \frac{\vect{m}^{(\ell)} \odot \vect{a}^{(\ell)}}{1-p} ,
132 +\end{equation}
133 +and the next layer consumes $\tilde{\vect{a}}^{(\ell)}$ in place of
134 +$\vect{a}^{(\ell)}$. The division by $1-p$ keeps the expected input to
135 +each downstream unit unchanged,
136 +$\E[\tilde{a}_j^{(\ell)}] = a_j^{(\ell)}$, so that at test time the
137 +network is used \emph{without any modification}:
138 +\begin{equation}\label{eq:reg-dropout-test}
139 +\tilde{\vect{a}}^{(\ell)} = \vect{a}^{(\ell)}
140 +\qquad \text{(inference: no mask, no rescaling).}
141 +\end{equation}
142 +Typical rates are $p = 0.5$ for hidden layers and $p \approx 0.2$ for
143 +inputs. Figure~\ref{fig:reg-dropout} contrasts a standard fully connected
144 +network with one realization of its dropout-thinned counterpart.
145 +
146 +\begin{figure}[htbp]
147 + \centering
148 + \begin{tikzpicture}[scale=0.82, transform shape]
149 + % ---------- (a) standard network ----------
150 + \begin{scope}
151 + \foreach \i in {1,2,3}
152 + \node[ninput] (ai\i) at (0, 1.0-\i*1.0) {};
153 + \foreach \j in {1,...,4}
154 + \node[nhidden] (ah1\j) at (1.9, 1.5-\j*1.0) {};
155 + \foreach \j in {1,...,4}
156 + \node[nhidden] (ah2\j) at (3.8, 1.5-\j*1.0) {};
157 + \foreach \k in {1,2}
158 + \node[noutput] (ao\k) at (5.7, 0.5-\k*1.0) {};
159 + \foreach \i in {1,2,3} \foreach \j in {1,...,4}
160 + \draw[black!40, semithick] (ai\i) -- (ah1\j);
161 + \foreach \i in {1,...,4} \foreach \j in {1,...,4}
162 + \draw[black!40, semithick] (ah1\i) -- (ah2\j);
163 + \foreach \i in {1,...,4} \foreach \k in {1,2}
164 + \draw[black!40, semithick] (ah2\i) -- (ao\k);
165 + \node[etiquette, align=center] at (2.85, -3.6)
166 + {(a) standard network};
167 + \end{scope}
168 + % ---------- (b) after dropout ----------
169 + \begin{scope}[xshift=8.6cm]
170 + % surviving units: all inputs, h1 = {1,3}, h2 = {2,4}, all outputs
171 + \foreach \i in {1,2,3}
172 + \node[ninput] (bi\i) at (0, 1.0-\i*1.0) {};
173 + \foreach \j in {1,3}
174 + \node[nhidden] (bh1\j) at (1.9, 1.5-\j*1.0) {};
175 + \foreach \j in {2,4}
176 + \node[nhidden] (bh2\j) at (3.8, 1.5-\j*1.0) {};
177 + % dropped units: dashed outline, pale fill
178 + \foreach \j in {2,4}
179 + \node[neuron, dashed, draw=black!45, fill=black!5]
180 + (bh1\j) at (1.9, 1.5-\j*1.0) {};
181 + \foreach \j in {1,3}
182 + \node[neuron, dashed, draw=black!45, fill=black!5]
183 + (bh2\j) at (3.8, 1.5-\j*1.0) {};
184 + \foreach \k in {1,2}
185 + \node[noutput] (bo\k) at (5.7, 0.5-\k*1.0) {};
186 + % edges touching a dropped unit: faint and dashed
187 + \foreach \i in {1,2,3} \foreach \j in {2,4}
188 + \draw[black!20, dashed] (bi\i) -- (bh1\j);
189 + \foreach \i in {1,3} \foreach \j in {1,3}
190 + \draw[black!20, dashed] (bh1\i) -- (bh2\j);
191 + \foreach \i in {2,4} \foreach \j in {1,...,4}
192 + \draw[black!20, dashed] (bh1\i) -- (bh2\j);
193 + \foreach \i in {1,3} \foreach \k in {1,2}
194 + \draw[black!20, dashed] (bh2\i) -- (bo\k);
195 + % active edges between surviving units
196 + \foreach \i in {1,2,3} \foreach \j in {1,3}
197 + \draw[black!40, semithick] (bi\i) -- (bh1\j);
198 + \foreach \i in {1,3} \foreach \j in {2,4}
199 + \draw[black!40, semithick] (bh1\i) -- (bh2\j);
200 + \foreach \i in {2,4} \foreach \k in {1,2}
201 + \draw[black!40, semithick] (bh2\i) -- (bo\k);
202 + % crosses on dropped units
203 + \foreach \n in {bh12, bh14, bh21, bh23}{
204 + \draw[black!60, thick] (\n.north east) -- (\n.south west);
205 + \draw[black!60, thick] (\n.north west) -- (\n.south east);
206 + }
207 + \node[etiquette, align=center] at (2.85, -3.6)
208 + {(b) after dropout ($p = 0.5$ on hidden layers)};
209 + \end{scope}
210 + \end{tikzpicture}
211 + \caption{Dropout as stochastic architecture perturbation. (a)~The full
212 + network. (b)~One training-time realization: each hidden unit is deleted
213 + independently with probability $p$ (crossed out, dashed), together with
214 + all of its incoming and outgoing connections; surviving activations are
215 + rescaled by $1/(1-p)$ as in~\eqref{eq:reg-dropout-mask}. A different
216 + subnetwork is sampled for every example.}
217 + \label{fig:reg-dropout}
218 +\end{figure}
219 +
220 +\begin{remark}[Ensemble interpretation]\label{rem:reg-ensemble}
221 +A network with $N$ droppable units defines $2^N$ thinned subnetworks
222 +sharing one set of weights. Dropout training optimizes the expected loss
223 +over this exponential ensemble, and inference
224 +with~\eqref{eq:reg-dropout-test} approximates the ensemble's geometric-mean
225 +prediction with a single forward pass~\cite{srivastava2014}. The original
226 +paper recommends pairing dropout with a max-norm constraint
227 +$\lVert \vect{w}_j \rVert_2 \le c$ on incoming weight vectors.
228 +\end{remark}
229 +
230 +\section{Normalization Layers}
231 +
232 +\subsection{Batch Normalization}
233 +
234 +Batch Normalization (BN), due to Ioffe and Szegedy~\cite{ioffe2015},
235 +standardizes each pre-activation over the current mini-batch and then
236 +restores expressive freedom through two learned parameters. For a given
237 +unit, let $\mathcal{B} = \{z_1, \dots, z_m\}$ be the values it takes over
238 +a mini-batch of size $m$. The transform is defined by four equations:
239 +\begin{align}
240 +\mu_{\mathcal{B}} &= \frac{1}{m} \sum_{i=1}^{m} z_i
241 + && \text{(mini-batch mean)} \label{eq:reg-bn-mean}\\
242 +\sigma_{\mathcal{B}}^2 &= \frac{1}{m} \sum_{i=1}^{m}
243 + \bigl(z_i - \mu_{\mathcal{B}}\bigr)^2
244 + && \text{(mini-batch variance)} \label{eq:reg-bn-var}\\
245 +\hat{z}_i &= \frac{z_i - \mu_{\mathcal{B}}}
246 + {\sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}}
247 + && \text{(normalize)} \label{eq:reg-bn-norm}\\
248 +y_i &= \gamma\, \hat{z}_i + \beta
249 + && \text{(scale and shift)} \label{eq:reg-bn-scale}
250 +\end{align}
251 +where $\epsilon > 0$ is a small constant for numerical stability. The
252 +learned pair $(\gamma, \beta)$ ensures BN never destroys capacity:
253 +setting $\gamma = \sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}$ and
254 +$\beta = \mu_{\mathcal{B}}$ recovers the identity map. In convolutional
255 +networks the statistics are computed per channel, jointly over the batch
256 +and all spatial positions, with one pair $(\gamma_c, \beta_c)$ per
257 +channel, so as to respect the weight sharing of the convolution.
258 +
259 +At \emph{inference} no mini-batch is available. Exponential moving
260 +averages of the training statistics are accumulated,
261 +$\hat{\mu} \leftarrow \mathrm{EMA}(\mu_{\mathcal{B}})$ and
262 +$\hat{\sigma}^2 \leftarrow \mathrm{EMA}(\sigma_{\mathcal{B}}^2)$, and the
263 +whole layer becomes a fixed affine map,
264 +\begin{equation}\label{eq:reg-bn-inference}
265 +y = \gamma\, \frac{z - \hat{\mu}}{\sqrt{\hat{\sigma}^2 + \epsilon}} + \beta ,
266 +\end{equation}
267 +which can be folded into the preceding linear or convolutional layer at
268 +no runtime cost. In practice BN permits substantially larger learning
269 +rates, reduces sensitivity to initialization, and contributes a mild
270 +regularizing effect through the noise of mini-batch statistics — which is
271 +why it appears in this chapter. Its main weaknesses are its dependence on
272 +a sufficiently large batch size and its awkwardness in recurrent or
273 +online settings.
274 +
275 +\subsection{Layer Normalization}
276 +
277 +Layer Normalization (LN) transposes the computation: the statistics are
278 +taken over the \emph{features of a single example} rather than over the
279 +batch. For a vector $\vect{z} \in \R^d$,
280 +\begin{equation}\label{eq:reg-ln}
281 +\mu = \frac{1}{d} \sum_{j=1}^{d} z_j ,
282 +\qquad
283 +\sigma^2 = \frac{1}{d} \sum_{j=1}^{d} (z_j - \mu)^2 ,
284 +\qquad
285 +\mathrm{LN}(\vect{z})
286 += \vect{\gamma} \odot \frac{\vect{z} - \mu}{\sqrt{\sigma^2 + \epsilon}}
287 ++ \vect{\beta} ,
288 +\end{equation}
289 +with learned vectors $\vect{\gamma}, \vect{\beta} \in \R^d$.
290 +
291 +\begin{remark}[BN versus LN: the normalization axis]\label{rem:reg-bnln}
292 +BN normalizes each feature across examples
293 +(equations~\eqref{eq:reg-bn-mean}--\eqref{eq:reg-bn-scale} run over the
294 +batch index $i$); LN normalizes each example across features
295 +(equation~\eqref{eq:reg-ln} runs over the feature index $j$). Because LN
296 +is independent of the batch, it behaves identically at training and test
297 +time, works at batch size one, and handles variable-length sequences —
298 +the reasons it, rather than BN, is the standard choice in recurrent
299 +networks and Transformers.
300 +\end{remark}
301 +
302 +\section{Regularizing the Training Procedure}
303 +
304 +\subsection{Early stopping}
305 +
306 +The simplest regularizer costs nothing: monitor the loss on a held-out
307 +validation set and stop training when it ceases to improve. The training
308 +loss decreases essentially monotonically, but the validation loss
309 +typically traces a U-shape — beyond its minimum, further optimization
310 +fits sampling noise rather than structure. Figure~\ref{fig:reg-earlystop}
311 +illustrates the regime change; in practice one checkpoints the parameters
312 +at each validation improvement and restores the best checkpoint when a
313 +\emph{patience} budget of non-improving epochs is exhausted.
314 +
315 +\begin{figure}[htbp]
316 + \centering
317 + \begin{tikzpicture}
318 + \begin{axis}[
319 + width=0.78\textwidth, height=6.2cm,
320 + xlabel={epoch}, ylabel={loss},
321 + xmin=0, xmax=100, ymin=0, ymax=1.15,
322 + axis lines=left,
323 + legend style={draw=none, fill=none, at={(0.97,0.95)},
324 + anchor=north east, font=\small},
325 + domain=0:100, samples=200,
326 + ]
327 + \addplot[cinput, thick] {0.10 + 0.90*exp(-x/15)};
328 + \addlegendentry{training loss}
329 + \addplot[coutput, thick] {0.21 + 0.79*exp(-x/15) + 0.004*x};
330 + \addlegendentry{validation loss}
331 + \draw[black!60, dashed, thick] (axis cs:37.9,0) -- (axis cs:37.9,1.08);
332 + \node[etiquette, anchor=south, rotate=90, text=black!75]
333 + at (axis cs:37.9,0.72) {early stopping point};
334 + \addplot[coutput, only marks, mark=*, mark size=1.8pt]
335 + coordinates {(37.9,0.425)};
336 + \end{axis}
337 + \end{tikzpicture}
338 + \caption{Early stopping. The training loss (blue) decreases throughout,
339 + while the validation loss (red) reaches a minimum and then rises as the
340 + network begins to overfit. Training is halted at the dashed line and
341 + the parameters of the best validation checkpoint are retained.}
342 + \label{fig:reg-earlystop}
343 +\end{figure}
344 +
345 +\begin{property}[Early stopping as implicit weight decay]\label{prop:reg-es}
346 +For a quadratic loss optimized by gradient descent from
347 +$\vect{\theta}_0 = \vect{0}$, halting after $\tau$ steps with learning
348 +rate $\eta$ yields a solution equivalent to fully optimizing the
349 +$L_2$-penalized objective~\eqref{eq:reg-l2loss} with regularization
350 +strength $\lambda \approx 1/(\eta\tau)$: a trajectory of limited length
351 +cannot travel far along low-curvature directions, exactly as weight decay
352 +suppresses them~\cite{goodfellow2016book}.
353 +\end{property}
354 +
355 +\subsection{Data augmentation}
356 +
357 +Rather than constraining the model, one can enlarge the data. If
358 +$\mathcal{T}$ is a family of label-preserving transformations —
359 +translations, crops, horizontal flips, small rotations, photometric
360 +jitter for images; time-stretching or noise injection for audio —
361 +training on transformed samples replaces the empirical loss by
362 +$\E_{t \sim \mathcal{T}}\bigl[\Loss\bigl(f(t(\vect{x})), \vect{y}\bigr)\bigr]$,
363 +encoding the invariances of the task directly into the training
364 +distribution. Augmentation is often the single most effective
365 +regularizer available for perception tasks, because it injects genuine
366 +prior knowledge rather than generic smoothness.
367 +
368 +\subsection{Label smoothing}
369 +
370 +Hard one-hot targets push the softmax toward infinite logit gaps and
371 +overconfident predictions. Label smoothing blends the one-hot vector
372 +$\vect{y}$ with the uniform distribution over the $K$ classes:
373 +\begin{equation}\label{eq:reg-labelsmooth}
374 +\vect{y}^{\mathrm{smooth}} = (1 - \varepsilon)\, \vect{y}
375 ++ \frac{\varepsilon}{K} \, \vect{1} ,
376 +\qquad \varepsilon \in [0, 1),
377 +\end{equation}
378 +with $\varepsilon = 0.1$ a common choice. The correct class keeps
379 +probability $1 - \varepsilon + \varepsilon/K$; every other class receives
380 +$\varepsilon/K$. The cross-entropy gradient then stops rewarding
381 +unbounded confidence, which improves calibration and often test accuracy.
382 +
383 +\subsection{Gradient clipping}
384 +
385 +Finally, the optimization trajectory itself can be regularized. When a
386 +loss surface contains cliffs — as in recurrent networks — a single large
387 +gradient can catapult the parameters out of a good basin. Gradient
388 +clipping rescales the gradient whenever its norm exceeds a threshold
389 +$\tau$:
390 +\begin{equation}\label{eq:reg-clip}
391 +\vect{g} \;\leftarrow\;
392 +\begin{cases}
393 +\vect{g} & \text{if } \lVert \vect{g} \rVert_2 \le \tau, \\[2pt]
394 +\dfrac{\tau}{\lVert \vect{g} \rVert_2}\, \vect{g} & \text{otherwise,}
395 +\end{cases}
396 +\end{equation}
397 +preserving the direction of the update while bounding its size. Clipping
398 +does not change the location of minima; it changes which minima the
399 +trajectory can reach, and is standard practice whenever exploding
400 +gradients are a risk.
401 +
402 +\medskip
403 +Taken together, the techniques of this chapter form a layered defence:
404 +penalties shape the hypothesis space, dropout and augmentation randomize
405 +what the network sees, normalization conditions the optimization, and
406 +early stopping bounds how far the fit is allowed to proceed. Modern
407 +practice combines several of them almost by default.
added chapters/04-cnn.tex +464 −0
@@ -0,0 +1,464 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 4 : Convolutional Neural Networks (chapters/04-cnn.tex)
6 +% ============================================================================
7 +\chapter{Convolutional Neural Networks}\label{chap:cnn}
8 +
9 +Fully connected networks treat every input coordinate as unrelated to every
10 +other: a pixel in the top-left corner of an image and its immediate
11 +neighbour are, as far as the architecture is concerned, as distant as two
12 +arbitrary pixels. Convolutional neural networks (CNNs) build the spatial
13 +structure of the data directly into the architecture through three
14 +mechanisms: \emph{local connectivity} (each unit sees only a small window
15 +of the input), \emph{weight sharing} (the same filter is applied at every
16 +position, so a feature detector learned in one place is available
17 +everywhere), and \emph{hierarchical composition} (stacked layers assemble
18 +edges into textures, textures into parts, parts into objects). This chapter
19 +develops the convolution and pooling operations, assembles them into the
20 +canonical convolutional architecture of LeCun et
21 +al.~\cite{lecun1998}, analyses the residual connection of He et
22 +al.~\cite{he2016} that made very deep networks trainable, and closes with
23 +the notion of receptive field and a brief genealogy of modern
24 +architectures.
25 +
26 +% ----------------------------------------------------------------------------
27 +\section{From the Visual Cortex to the Convolutional Layer}
28 +
29 +The lineage of the CNN begins in neurophysiology. Recording from the cat's
30 +visual cortex, Hubel and Wiesel identified \emph{simple cells}, which
31 +respond to oriented edges at specific retinal positions, and \emph{complex
32 +cells}, which respond to the same features with tolerance to small shifts.
33 +Fukushima's \emph{Neocognitron} (1980) translated this two-cell hierarchy
34 +into a network that alternates feature-extracting S-cells — local
35 +receptive fields with shared weights, the conceptual ancestor of the
36 +convolutional layer — with position-tolerant C-cells, the ancestor of
37 +pooling. What the Neocognitron lacked was end-to-end supervised training:
38 +that ingredient was supplied when backpropagation was applied to
39 +weight-shared architectures, culminating in LeNet-5~\cite{lecun1998}, the
40 +first widely deployed CNN, which read millions of bank cheques using the
41 +canonical pattern
42 +$[\text{conv} \to \text{pool}] \times N \to \text{dense} \to
43 +\text{output}$ that this chapter formalizes.
44 +
45 +% ----------------------------------------------------------------------------
46 +\section{The Convolution Operation}
47 +
48 +\subsection{Definition}
49 +
50 +\begin{definition}[Discrete two-dimensional convolution]\label{def:cnn-conv}
51 +Let $I$ be an input map and $K \in \R^{k_h \times k_w}$ a kernel (filter).
52 +The discrete convolution of $I$ by $K$ is
53 +\begin{equation}\label{eq:cnn-conv2d}
54 + S(i,j) \;=\; (I * K)(i,j)
55 + \;=\; \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} I(i-m,\, j-n)\, K(m,n).
56 +\end{equation}
57 +\end{definition}
58 +
59 +\begin{remark}[Convolution versus cross-correlation]\label{rem:cnn-crosscorr}
60 +Deep learning frameworks implement \emph{cross-correlation} — the same sum
61 +without flipping the kernel, i.e.\ with indices $I(i+m,\,j+n)$. Since the
62 +kernel entries are learned, the two operations are equivalent up to a
63 +re-parameterization of the weights, and the term ``convolution'' is used
64 +for both. All equations below follow the cross-correlation convention.
65 +\end{remark}
66 +
67 +In a convolutional \emph{layer}, the input is a stack of $C_{\mathrm{in}}$
68 +channels $\vect{x} \in \R^{C_{\mathrm{in}} \times H \times W}$, and each of
69 +the $C_{\mathrm{out}}$ output channels is produced by its own
70 +three-dimensional filter plus a bias. With stride $s$ and zero-padding $p$
71 +(writing $\tilde{\vect{x}}$ for the input bordered by $p$ zeros on each
72 +side), the output channel $c_{\mathrm{out}}$ is
73 +\begin{equation}\label{eq:cnn-convlayer}
74 + y_{c_{\mathrm{out}}}(i,j)
75 + \;=\; b_{c_{\mathrm{out}}}
76 + \;+\; \sum_{c=1}^{C_{\mathrm{in}}} \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1}
77 + W_{c_{\mathrm{out}},c}(m,n)\;
78 + \tilde{x}_{c}\big(s\,i + m,\; s\,j + n\big),
79 +\end{equation}
80 +where $i,j$ index the output positions. Three properties of
81 +\eqref{eq:cnn-convlayer} explain the effectiveness of the operation. Each
82 +output value depends only on a $k_h \times k_w$ window of the input
83 +(\emph{local connectivity}); the same weights
84 +$W_{c_{\mathrm{out}},c}$ are used at every position $(i,j)$ (\emph{weight
85 +sharing}), which makes the layer \emph{equivariant} to translation — a
86 +shifted input produces a correspondingly shifted output — and reduces the
87 +parameter count dramatically; and stacking such layers composes local
88 +detectors into progressively more global and abstract
89 +ones~\cite{goodfellow2016book}. Algorithm~\ref{alg:cnn-forward} spells out
90 +the layer's forward pass as literal loops over
91 +\eqref{eq:cnn-convlayer} — production implementations replace the loops
92 +with im2col-plus-GEMM or FFT-based routines, but compute exactly the same
93 +map.
94 +
95 +\begin{algorithm}[htbp]
96 + \caption{Forward pass of a convolutional layer
97 + (stride $s$, padding $p$)}
98 + \label{alg:cnn-forward}
99 + \begin{algorithmic}[1]
100 + \Require input $\vect{x} \in \R^{C_{\mathrm{in}} \times H \times W}$,
101 + weights $W \in \R^{C_{\mathrm{out}} \times C_{\mathrm{in}} \times
102 + k_h \times k_w}$, biases $\vect{b}$, activation $\varphi$
103 + \State $\tilde{\vect{x}} \gets$ zero-pad $\vect{x}$ with a border of
104 + $p$ zeros on each spatial side
105 + \State $H_{\mathrm{out}} \gets \lfloor (H + 2p - k_h)/s \rfloor + 1$;
106 + \quad $W_{\mathrm{out}} \gets \lfloor (W + 2p - k_w)/s \rfloor + 1$
107 + \Comment{\eqref{eq:cnn-outsize}}
108 + \For{$c_{\mathrm{out}} = 1, \dots, C_{\mathrm{out}}$}
109 + \For{$i = 0, \dots, H_{\mathrm{out}}-1$;\;
110 + $j = 0, \dots, W_{\mathrm{out}}-1$}
111 + \State $u \gets b_{c_{\mathrm{out}}}$
112 + \For{$c = 1, \dots, C_{\mathrm{in}}$;\;
113 + $m = 0, \dots, k_h - 1$;\;
114 + $n = 0, \dots, k_w - 1$}
115 + \State $u \gets u + W_{c_{\mathrm{out}},c}(m,n)\,
116 + \tilde{x}_c(s\,i + m,\; s\,j + n)$
117 + \Comment{windowed sum \eqref{eq:cnn-convlayer}}
118 + \EndFor
119 + \State $y_{c_{\mathrm{out}}}(i,j) \gets \varphi(u)$
120 + \EndFor
121 + \EndFor
122 + \State \Return $\vect{y} \in \R^{C_{\mathrm{out}} \times
123 + H_{\mathrm{out}} \times W_{\mathrm{out}}}$
124 + \end{algorithmic}
125 +\end{algorithm}
126 +
127 +\subsection{Stride, Padding and Output Geometry}
128 +
129 +The \emph{stride} $s$ is the step of the sliding window: $s>1$ downsamples
130 +the output. The \emph{padding} $p$ adds a border of zeros; ``valid''
131 +convolution uses $p=0$, while ``same'' padding, $p = \lfloor k/2 \rfloor$
132 +for odd $k$ and $s=1$, preserves the spatial size. For an input of height
133 +$H$, kernel size $k$, padding $p$ and stride $s$, the output height is
134 +\begin{equation}\label{eq:cnn-outsize}
135 + H_{\mathrm{out}}
136 + \;=\; \left\lfloor \frac{H + 2p - k}{s} \right\rfloor + 1,
137 +\end{equation}
138 +and symmetrically for the width. (With \emph{dilation} $d$, which inserts
139 +$d-1$ gaps between kernel taps, $k$ in \eqref{eq:cnn-outsize} is replaced
140 +by the effective size $k_{\mathrm{eff}} = d(k-1)+1$; dilation enlarges
141 +receptive fields without adding parameters.)
142 +
143 +Figure~\ref{fig:cnn-grid} traces \eqref{eq:cnn-convlayer} on a concrete
144 +example: a $5 \times 5$ input, a $3 \times 3$ vertical-edge kernel, stride
145 +$s=1$ and padding $p=0$, so that \eqref{eq:cnn-outsize} gives
146 +$H_{\mathrm{out}} = \lfloor (5 + 0 - 3)/1 \rfloor + 1 = 3$.
147 +
148 +\begin{figure}[t]
149 + \centering
150 + \begin{tikzpicture}
151 + % -------- input grid 5x5 (6 mm cells), top-left corner at (0,0)
152 + \fill[cinput!30] (0,0) rectangle (1.8,-1.8);
153 + \draw[black!60] (0,-3) grid[step=6mm] (3,0);
154 + \draw[cinput!80!black, very thick] (0,0) rectangle (1.8,-1.8);
155 + \foreach \v [count=\k from 0] in
156 + {1,2,0,1,3, 0,1,2,3,1, 1,0,2,2,0, 2,1,0,1,2, 0,2,1,0,1}{
157 + \pgfmathtruncatemacro{\r}{int(\k/5)}
158 + \pgfmathtruncatemacro{\c}{mod(\k,5)}
159 + \node[font=\small] at (\c*0.6+0.3, -\r*0.6-0.3) {$\v$};
160 + }
161 + \node[etiquette, font=\small] at (1.5,-3.45) {input $5\times5$};
162 + % -------- operator
163 + \node[font=\large] at (3.9,-1.5) {$\ast$};
164 + % -------- kernel 3x3, vertically centred on the input
165 + \begin{scope}[shift={(4.8,-0.6)}]
166 + \fill[cgate!15] (0,0) rectangle (1.8,-1.8);
167 + \draw[black!60] (0,-1.8) grid[step=6mm] (1.8,0);
168 + \foreach \v [count=\k from 0] in {1,0,-1, 1,0,-1, 1,0,-1}{
169 + \pgfmathtruncatemacro{\r}{int(\k/3)}
170 + \pgfmathtruncatemacro{\c}{mod(\k,3)}
171 + \node[font=\small] at (\c*0.6+0.3, -\r*0.6-0.3) {$\v$};
172 + }
173 + \node[etiquette, font=\small] at (0.9,-2.25) {kernel $3\times3$};
174 + \end{scope}
175 + % -------- operator
176 + \node[font=\large] at (7.2,-1.5) {$=$};
177 + % -------- output grid 3x3
178 + \begin{scope}[shift={(7.8,-0.6)}]
179 + \fill[coutput!30] (0,0) rectangle (0.6,-0.6);
180 + \draw[black!60] (0,-1.8) grid[step=6mm] (1.8,0);
181 + \draw[coutput!80!black, very thick] (0,0) rectangle (0.6,-0.6);
182 + \foreach \v [count=\k from 0] in {-2,-3,0, -1,-4,1, 0,0,0}{
183 + \pgfmathtruncatemacro{\r}{int(\k/3)}
184 + \pgfmathtruncatemacro{\c}{mod(\k,3)}
185 + \node[font=\small] at (\c*0.6+0.3, -\r*0.6-0.3) {$\v$};
186 + }
187 + \node[etiquette, font=\small] at (0.9,-2.25) {output $3\times3$};
188 + \end{scope}
189 + % -------- window -> output cell
190 + \draw[flechep, cinput!80!black] (0.9,0.12) to[bend left=18] (8.1,-0.5);
191 + \end{tikzpicture}
192 + \caption{Convolution (cross-correlation) of a $5\times5$ input with a
193 + $3\times3$ vertical-edge kernel, stride $s=1$, padding $p=0$. The
194 + highlighted window produces the highlighted output cell:
195 + $1\cdot1 + 2\cdot0 + 0\cdot(-1) + 0\cdot1 + 1\cdot0 + 2\cdot(-1)
196 + + 1\cdot1 + 0\cdot0 + 2\cdot(-1) = -2$. Sliding the window over all
197 + nine valid positions fills the $3\times3$ output map, in accordance
198 + with \eqref{eq:cnn-outsize}.}
199 + \label{fig:cnn-grid}
200 +\end{figure}
201 +
202 +\subsection{Parameter Efficiency}
203 +
204 +The number of parameters of the layer \eqref{eq:cnn-convlayer} is
205 +\begin{equation}\label{eq:cnn-params}
206 + \#\text{params}
207 + \;=\; C_{\mathrm{out}} \left( C_{\mathrm{in}}\, k_h\, k_w + 1 \right),
208 +\end{equation}
209 +\emph{independent of the spatial resolution} $H \times W$ — the decisive
210 +advantage over a dense layer, whose parameter count grows with the square
211 +of the image size. A $3\times3$ convolution mapping $64$ channels to $64$
212 +channels costs $36{,}928$ parameters whether the image is $32\times32$ or
213 +$1024\times1024$.
214 +
215 +% ----------------------------------------------------------------------------
216 +\section{Pooling}
217 +
218 +Pooling summarizes each local neighbourhood of a feature map by a single
219 +value, providing a small amount of translation \emph{invariance} (on top of
220 +the equivariance of convolution) and reducing the spatial resolution. Let
221 +$\mathcal{R}_{ij}$ denote the $k \times k$ window of input positions
222 +associated with output position $(i,j)$ (with stride $s$, typically
223 +$k=s=2$). \emph{Max pooling} keeps the strongest activation,
224 +\begin{equation}\label{eq:cnn-maxpool}
225 + y_c(i,j) \;=\; \max_{(m,n)\,\in\,\mathcal{R}_{ij}} x_c(m,n),
226 +\end{equation}
227 +whereas \emph{average pooling} keeps the mean response,
228 +\begin{equation}\label{eq:cnn-avgpool}
229 + y_c(i,j) \;=\; \frac{1}{\lvert \mathcal{R}_{ij} \rvert}
230 + \sum_{(m,n)\,\in\,\mathcal{R}_{ij}} x_c(m,n).
231 +\end{equation}
232 +The limiting case of \eqref{eq:cnn-avgpool}, \emph{global average pooling},
233 +collapses each channel's entire $H \times W$ map to one scalar,
234 +\begin{equation}\label{eq:cnn-gap}
235 + y_c \;=\; \frac{1}{HW} \sum_{i=1}^{H} \sum_{j=1}^{W} x_c(i,j),
236 +\end{equation}
237 +and is the modern replacement for the large dense layers that dominated
238 +the parameter budget of early CNNs: it contributes no parameters, acts as
239 +a structural regularizer, and lets the same network accept inputs of any
240 +size.
241 +
242 +% ----------------------------------------------------------------------------
243 +\section{The Canonical Convolutional Architecture}
244 +
245 +Assembling the two operations yields the canonical architecture of
246 +Figure~\ref{fig:cnn-archi}: a \emph{feature-extraction} stage alternating
247 +convolution (+ nonlinearity) and pooling, in which the spatial resolution
248 +decreases while the channel count — the richness of the learned
249 +vocabulary — increases; then a \emph{classification} stage, in which the
250 +final maps are flattened (or globally pooled) into a vector, passed
251 +through dense layers, and mapped by a softmax to class probabilities. This
252 +is exactly the structure of LeNet-5~\cite{lecun1998}, which stacked two
253 +conv--pool pairs and two dense layers ($\sim$60k parameters) and
254 +established that such a pipeline can be trained end to end by gradient
255 +descent on raw pixels.
256 +
257 +\begin{figure}[t]
258 + \centering
259 + \begin{tikzpicture}[node distance=0.55cm]
260 + \node[blocinput, minimum width=1.05cm, minimum height=2.6cm,
261 + font=\scriptsize] (in) {input\\image};
262 + \node[blochidden, minimum width=1.05cm, minimum height=2.6cm,
263 + font=\scriptsize, right=of in] (c1) {conv\\ReLU};
264 + \node[bloc, minimum width=0.95cm, minimum height=2.0cm,
265 + font=\scriptsize, right=of c1] (p1) {pool};
266 + \node[blochidden, minimum width=1.05cm, minimum height=2.0cm,
267 + font=\scriptsize, right=of p1] (c2) {conv\\ReLU};
268 + \node[bloc, minimum width=0.95cm, minimum height=1.4cm,
269 + font=\scriptsize, right=of c2] (p2) {pool};
270 + \node[bloc, minimum width=1.0cm, minimum height=2.2cm,
271 + font=\scriptsize, right=of p2] (fl) {flatten};
272 + \node[blochidden, minimum width=1.0cm, minimum height=1.7cm,
273 + font=\scriptsize, right=of fl] (fc) {dense\\ReLU};
274 + \node[blocoutput, minimum width=1.1cm, minimum height=1.2cm,
275 + font=\scriptsize, right=of fc] (sm) {softmax};
276 + \foreach \a/\b in {in/c1, c1/p1, p1/c2, c2/p2, p2/fl, fl/fc, fc/sm}
277 + \draw[fleche] (\a) -- (\b);
278 + % grouping braces
279 + \draw[decorate, decoration={brace, mirror, amplitude=5pt}]
280 + (c1.west |- 0,-1.55) -- (p2.east |- 0,-1.55)
281 + node[midway, below=7pt, etiquette, font=\small]{feature extraction};
282 + \draw[decorate, decoration={brace, mirror, amplitude=5pt}]
283 + (fl.west |- 0,-1.55) -- (sm.east |- 0,-1.55)
284 + node[midway, below=7pt, etiquette, font=\small]{classification};
285 + \end{tikzpicture}
286 + \caption{The canonical convolutional architecture in the lineage of
287 + LeNet-5~\cite{lecun1998}. In the feature-extraction stage the spatial
288 + resolution shrinks (decreasing block heights) while the channel count
289 + grows; the classification stage flattens the final maps and applies
290 + dense layers followed by a softmax.}
291 + \label{fig:cnn-archi}
292 +\end{figure}
293 +
294 +The historical turning point came when this recipe met large datasets and
295 +GPU computing: AlexNet's 2012 ImageNet victory — five convolutional and
296 +three dense layers, ReLU activations and dropout — cut the top-5 error
297 +from 26.2\% to 15.3\% and opened the modern era of deep
298 +learning~\cite{goodfellow2016book}.
299 +
300 +% ----------------------------------------------------------------------------
301 +\section{Residual Learning}
302 +
303 +Depth is the currency of representational power, yet naively stacking
304 +layers eventually makes even the \emph{training} error worse — the
305 +\emph{degradation problem}. A network of $50$ plain layers underperforms
306 +its $20$-layer counterpart not because of overfitting but because the
307 +optimizer fails to find a solution as good as ``the shallow network plus
308 +identity layers'', although one exists by construction. Residual
309 +learning~\cite{he2016} removes this obstacle by making the identity the
310 +default behaviour of every block: instead of asking a block to learn a
311 +mapping $\mathcal{H}(\vect{x})$ directly, one asks it to learn only the
312 +\emph{residual} $\mathcal{F}(\vect{x}) = \mathcal{H}(\vect{x}) - \vect{x}$
313 +and adds the input back through a shortcut connection,
314 +\begin{equation}\label{eq:cnn-residual}
315 + \vect{y} \;=\; \mathcal{F}(\vect{x}, \{\mat{W}_i\}) + \vect{x},
316 +\end{equation}
317 +where typically $\mathcal{F}$ consists of two convolution--batch-norm
318 +stages with a ReLU in between (Figure~\ref{fig:cnn-resblock}), and a
319 +learned projection $\mat{W}_s \vect{x}$ replaces the identity when
320 +dimensions change. If the optimal mapping is close to the identity, the
321 +block merely needs to drive $\mathcal{F}$ towards zero — far easier than
322 +reproducing the identity through a stack of nonlinear layers.
323 +
324 +\begin{figure}[t]
325 + \centering
326 + \begin{tikzpicture}
327 + \node (xl) at (0,0) {$\vect{x}$};
328 + \node[blochidden, right=0.95cm of xl, minimum width=2.3cm]
329 + (f1) {conv $3{\times}3$\\BN, ReLU};
330 + \node[blochidden, right=0.7cm of f1, minimum width=2.3cm]
331 + (f2) {conv $3{\times}3$\\BN};
332 + \node[op, right=0.95cm of f2] (plus) {$+$};
333 + \node[bloc, right=0.7cm of plus, minimum width=1.1cm] (relu) {ReLU};
334 + \node[right=0.7cm of relu] (y) {$\vect{y}$};
335 + \draw[fleche] (xl) -- (f1);
336 + \draw[fleche] (f1) -- (f2);
337 + \draw[fleche] (f2) -- (plus);
338 + \draw[fleche] (plus) -- (relu);
339 + \draw[fleche] (relu) -- (y);
340 + % identity shortcut
341 + \coordinate (tap) at ($(xl.east)+(0.45,0)$);
342 + \fill (tap) circle (1.3pt);
343 + \draw[fleche, cmem!80!black] (tap) -- ++(0,1.35) -| (plus.north);
344 + \node[etiquette, font=\small, text=cmem!80!black]
345 + at ($(f1.north)!0.5!(f2.north)+(0,1.15)$) {identity shortcut $\vect{x}$};
346 + \node[etiquette, font=\small]
347 + at ($(f1.south)!0.5!(f2.south)+(0,-0.35)$)
348 + {residual branch $\mathcal{F}(\vect{x})$};
349 + \end{tikzpicture}
350 + \caption{The residual block of ResNet~\cite{he2016}. The main path
351 + computes the residual $\mathcal{F}(\vect{x})$ through two
352 + convolution--batch-norm stages; the shortcut carries $\vect{x}$
353 + unchanged to the addition node, implementing
354 + \eqref{eq:cnn-residual}.}
355 + \label{fig:cnn-resblock}
356 +\end{figure}
357 +
358 +\begin{property}[Gradient flow through the identity
359 + shortcut]\label{prop:cnn-resgrad}
360 +Consider a stack of residual blocks
361 +$\vect{x}_{\ell+1} = \vect{x}_\ell + \mathcal{F}(\vect{x}_\ell)$. Unrolling
362 +from layer $\ell$ to any deeper layer $L$ gives
363 +\begin{equation}\label{eq:cnn-unroll}
364 + \vect{x}_L \;=\; \vect{x}_\ell
365 + \;+\; \sum_{i=\ell}^{L-1} \mathcal{F}(\vect{x}_i),
366 +\end{equation}
367 +and the chain rule then yields
368 +\begin{equation}\label{eq:cnn-resgrad}
369 + \frac{\partial \Loss}{\partial \vect{x}_\ell}
370 + \;=\;
371 + \frac{\partial \Loss}{\partial \vect{x}_L}
372 + \left( \mat{I} \;+\;
373 + \frac{\partial}{\partial \vect{x}_\ell}
374 + \sum_{i=\ell}^{L-1} \mathcal{F}(\vect{x}_i) \right).
375 +\end{equation}
376 +The additive identity term $\mat{I}$ in \eqref{eq:cnn-resgrad} provides a
377 +direct gradient path from any layer to any shallower layer: the gradient
378 +cannot vanish even when the Jacobian of the residual branch is small,
379 +because it is never forced through a long product of weight matrices.
380 +\end{property}
381 +
382 +This single architectural idea allowed networks of $152$ layers — eight
383 +times deeper than the deepest previous mainstream design yet cheaper in
384 +floating-point operations — to win the 2015 ImageNet competition with a
385 +3.57\% top-5 ensemble error~\cite{he2016}; the same shortcut structure
386 +reappears in virtually every subsequent deep architecture, including the
387 +Transformer.
388 +
389 +% ----------------------------------------------------------------------------
390 +\section{The Receptive Field}
391 +
392 +The \emph{receptive field} of a unit is the region of the input image that
393 +can influence its value. For a stack of layers where layer $\ell$ has
394 +kernel size $k_\ell$ and stride $s_\ell$, the receptive field $r_\ell$
395 +grows according to the recursion
396 +\begin{equation}\label{eq:cnn-receptive}
397 + r_\ell \;=\; r_{\ell-1} \;+\; (k_\ell - 1) \prod_{i=1}^{\ell-1} s_i,
398 + \qquad r_0 = 1,
399 +\end{equation}
400 +where the product accounts for the cumulative downsampling in front of
401 +layer $\ell$. Two consequences guide architecture design. First, small
402 +kernels compose efficiently: two stacked $3\times3$ convolutions reach the
403 +same $5\times5$ receptive field as one $5\times5$ kernel with fewer
404 +parameters ($18C^2$ versus $25C^2$) and one extra nonlinearity — the
405 +observation that underlies VGG's uniform $3\times3$ design. Second,
406 +strides and pooling are multiplicative in \eqref{eq:cnn-receptive}: early
407 +downsampling is the cheapest way to grow the receptive field, which is why
408 +deep units eventually see the whole image and can encode global,
409 +object-level structure.
410 +
411 +% ----------------------------------------------------------------------------
412 +\section{A Brief Genealogy of Architectures}
413 +
414 +Table~\ref{tab:cnn-genealogy} summarizes the milestones that separate
415 +LeNet-5 from today's networks. Beyond ResNet, three ideas deserve mention.
416 +\emph{Multi-branch design} (GoogLeNet's Inception module) processes each
417 +position at several kernel sizes in parallel and concatenates the results,
418 +using $1\times1$ ``bottleneck'' convolutions to keep the cost low —
419 +22 layers with only $\sim$7M parameters. \emph{Dense connectivity}
420 +(DenseNet) generalizes the shortcut of \eqref{eq:cnn-residual} by
421 +concatenating, rather than adding, the features of \emph{all} preceding
422 +layers, maximizing feature reuse and gradient flow. \emph{Factorized
423 +convolution} (MobileNet) splits a standard convolution into a per-channel
424 +(depthwise) spatial filter followed by a $1\times1$ (pointwise) channel
425 +mixer; on a feature map with $M$ input channels, $N$ output channels and a
426 +$D_K \times D_K$ kernel, the cost ratio relative to the standard layer is
427 +\begin{equation}\label{eq:cnn-separable}
428 + \frac{\text{separable}}{\text{standard}}
429 + \;=\; \frac{1}{N} + \frac{1}{D_K^2},
430 +\end{equation}
431 +about an $8$--$9\times$ saving for $3\times3$ kernels — the enabling
432 +arithmetic of mobile and embedded vision.
433 +
434 +\begin{table}[t]
435 + \centering
436 + \caption{Milestones of convolutional architecture design.}
437 + \label{tab:cnn-genealogy}
438 + \begin{tabular}{@{}llll@{}}
439 + \toprule
440 + Year & Architecture & Key idea & Scale \\
441 + \midrule
442 + 1980 & Neocognitron & S/C-cell hierarchy, weight sharing & --- \\
443 + 1998 & LeNet-5~\cite{lecun1998} & end-to-end conv--pool--dense & 60k \\
444 + 2012 & AlexNet & ReLU, dropout, GPU training & 60M \\
445 + 2014 & VGG & uniform $3\times3$ depth & 138M \\
446 + 2014 & GoogLeNet & Inception multi-branch, $1\times1$ bottlenecks & 7M \\
447 + 2015 & ResNet~\cite{he2016} & identity shortcut \eqref{eq:cnn-residual} & 25M \\
448 + 2017 & DenseNet & concatenated dense connectivity & 8M \\
449 + 2017 & MobileNet & depthwise separable conv.\ \eqref{eq:cnn-separable} & 4M \\
450 + 2019 & EfficientNet & compound depth/width/resolution scaling & 5--66M \\
451 + 2022 & ConvNeXt & modernized ResNet, $7\times7$ depthwise & 29M+ \\
452 + \bottomrule
453 + \end{tabular}
454 +\end{table}
455 +
456 +The through-line of this genealogy is that every leap either improved
457 +\emph{gradient flow} (ReLU, batch normalization, the shortcut of
458 +\eqref{eq:cnn-resgrad}) or improved the \emph{allocation of computation}
459 +(bottlenecks, separable filters, compound scaling). The convolutional
460 +prior itself — locality, weight sharing, hierarchy — has remained intact
461 +from the Neocognitron to ConvNeXt, and it transfers beyond images: 1D
462 +convolutions over sequences and 3D convolutions over videos and volumes
463 +are the direct analogues of \eqref{eq:cnn-convlayer} with one fewer or one
464 +more spatial index~\cite{goodfellow2016book}.
added chapters/05-rnn.tex +501 −0
@@ -0,0 +1,501 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 5 : Recurrent Networks — RNN, LSTM and GRU (chapters/05-rnn.tex)
6 +% ============================================================================
7 +
8 +\chapter{Recurrent Networks: RNN, LSTM and GRU}
9 +\label{chap:rnn}
10 +
11 +Feedforward architectures map a fixed-size input to a fixed-size output;
12 +they have no mechanism for processing sequences of arbitrary length, nor
13 +any notion of order or memory. Recurrent neural networks (RNNs) remove this
14 +limitation by maintaining a \emph{hidden state} that is updated at every
15 +time step and acts as a compressed summary of everything the network has
16 +seen so far. This chapter develops the vanilla recurrent network and its
17 +training algorithm, backpropagation through time; analyzes why gradients
18 +vanish or explode over long horizons; and presents the two gated
19 +architectures — the long short-term memory (LSTM) of Hochreiter and
20 +Schmidhuber \cite{hochreiter1997} and the gated recurrent unit (GRU) of Cho
21 +et al.\ \cite{cho2014} — that made learning long-range dependencies
22 +practical. We close with bidirectional networks and the encoder--decoder
23 +paradigm for sequence-to-sequence learning.
24 +
25 +% ----------------------------------------------------------------------------
26 +\section{The simple recurrent network}
27 +\label{sec:rnn-simple}
28 +
29 +\begin{definition}[Recurrent neural network]
30 +A recurrent neural network processes a sequence
31 +$\vect{x}_1, \vect{x}_2, \dots, \vect{x}_T$, with
32 +$\vect{x}_t \in \R^{d}$, by maintaining a hidden state
33 +$\vect{h}_t \in \R^{n}$ computed from the current input and the previous
34 +state, $\vect{h}_t = f(\vect{h}_{t-1}, \vect{x}_t; \theta)$, where the
35 +parameters $\theta$ are \emph{shared across all time steps}.
36 +\end{definition}
37 +
38 +The standard (Elman) form uses a $\tanh$ nonlinearity for the state update
39 +and a linear read-out:
40 +\begin{equation}
41 + \vect{h}_t
42 + = \tanh\!\left(\mat{W}_{h}\,\vect{h}_{t-1}
43 + + \mat{W}_{x}\,\vect{x}_t + \vect{b}\right),
44 + \label{eq:rnn-hidden}
45 +\end{equation}
46 +\begin{equation}
47 + \vect{y}_t = \mat{W}_{y}\,\vect{h}_t + \vect{b}_y,
48 + \label{eq:rnn-output}
49 +\end{equation}
50 +with $\mat{W}_{x} \in \R^{n \times d}$,
51 +$\mat{W}_{h} \in \R^{n \times n}$ and
52 +$\mat{W}_{y} \in \R^{m \times n}$. For classification tasks the output
53 +\eqref{eq:rnn-output} is typically passed through a softmax,
54 +$\hat{\vect{y}}_t = \softmax(\mat{W}_{y}\vect{h}_t + \vect{b}_y)$.
55 +Weight sharing across time is the defining structural property: the same
56 +pair $(\mat{W}_h, \mat{W}_x)$ is applied at every step, so the RNN is a
57 +discrete-time dynamical system whose parameters do not grow with the
58 +sequence length. Figure~\ref{fig:rnn-unrolled} shows the two equivalent
59 +views of this computation: the \emph{folded} form, a single cell with a
60 +feedback loop, and the \emph{unfolded} form, a deep network with one layer
61 +per time step and tied weights.
62 +
63 +\begin{figure}[htbp]
64 + \centering
65 + \begin{tikzpicture}
66 + % ---------- folded form ----------
67 + \node[ninput] (fx) at (0,0) {$\vect{x}_t$};
68 + \node[mem, minimum width=1.3cm] (fc) at (0,1.9) {$\vect{h}$};
69 + \node[noutput] (fy) at (0,3.8) {$\vect{y}_t$};
70 + \draw[fleche] (fx) -- (fc) node[midway,right,etiquette] {$\mat{W}_x$};
71 + \draw[fleche] (fc) -- (fy) node[midway,right,etiquette] {$\mat{W}_y$};
72 + \draw[fleche] (fc.east) .. controls +(1.5,1.0) and +(1.5,-1.0) ..
73 + (fc.east) node[pos=0.5, right=0.25cm, etiquette] {$\mat{W}_h$};
74 + % ---------- unfold symbol ----------
75 + \node at (3.35,1.9) {\Large $=$};
76 + \node[etiquette] at (3.35,2.5) {unfold};
77 + % ---------- unfolded form ----------
78 + \foreach \i/\lab in {1/{t-1}, 2/{t}, 3/{t+1}} {
79 + \node[ninput] (x\i) at (3.0+\i*2.5, 0) {$\vect{x}_{\lab}$};
80 + \node[mem, minimum width=1.3cm] (c\i) at (3.0+\i*2.5, 1.9)
81 + {$\vect{h}_{\lab}$};
82 + \node[noutput] (y\i) at (3.0+\i*2.5, 3.8) {$\vect{y}_{\lab}$};
83 + \draw[fleche] (x\i) -- (c\i);
84 + \draw[fleche] (c\i) -- (y\i);
85 + }
86 + \draw[fleche] (4.3,1.9) -- (c1.west)
87 + node[very near start, above, etiquette] {$\cdots$};
88 + \draw[fleche] (c1) -- (c2)
89 + node[midway, above, etiquette] {$\mat{W}_h$};
90 + \draw[fleche] (c2) -- (c3)
91 + node[midway, above, etiquette] {$\mat{W}_h$};
92 + \draw[fleche] (c3.east) -- (11.7,1.9)
93 + node[very near end, above, etiquette] {$\cdots$};
94 + \end{tikzpicture}
95 + \caption{The two equivalent views of a recurrent network. Left: folded
96 + form — a single cell whose hidden state $\vect{h}$ feeds back into
97 + itself through $\mat{W}_h$. Right: unfolded form — the same cell
98 + replicated over time steps $t-1$, $t$, $t+1$, with all replicas sharing
99 + the weights $(\mat{W}_x, \mat{W}_h, \mat{W}_y)$ of
100 + \eqref{eq:rnn-hidden}--\eqref{eq:rnn-output}.}
101 + \label{fig:rnn-unrolled}
102 +\end{figure}
103 +
104 +% ----------------------------------------------------------------------------
105 +\section{Backpropagation through time}
106 +\label{sec:rnn-bptt}
107 +
108 +Training proceeds by \emph{unrolling} the recurrence into the feedforward
109 +network of Figure~\ref{fig:rnn-unrolled} (right) and applying standard
110 +backpropagation to the unrolled graph — hence the name backpropagation
111 +through time (BPTT). For a sequence-level loss
112 +$\Loss = \sum_{t=1}^{T} \Loss_t$, the gradient with respect to the
113 +recurrent matrix accumulates contributions over all pairs of time steps:
114 +\begin{equation}
115 + \frac{\partial \Loss}{\partial \mat{W}_h}
116 + = \sum_{t=1}^{T} \sum_{k=1}^{t}
117 + \frac{\partial \Loss_t}{\partial \vect{h}_t}
118 + \left( \prod_{i=k+1}^{t}
119 + \frac{\partial \vect{h}_i}{\partial \vect{h}_{i-1}} \right)
120 + \frac{\partial \vect{h}_k}{\partial \mat{W}_h}.
121 + \label{eq:rnn-loss-grad}
122 +\end{equation}
123 +The critical object in \eqref{eq:rnn-loss-grad} is the product of
124 +Jacobians that transports the error signal from step $t$ back to step $k$.
125 +Differentiating \eqref{eq:rnn-hidden}, each factor is
126 +$\mat{W}_h\transp$ scaled by the local slope of the nonlinearity, so
127 +\begin{equation}
128 + \frac{\partial \vect{h}_t}{\partial \vect{h}_k}
129 + = \prod_{i=k+1}^{t}
130 + \frac{\partial \vect{h}_i}{\partial \vect{h}_{i-1}}
131 + = \prod_{i=k+1}^{t}
132 + \operatorname{diag}\!\bigl(\tanh'(\vect{a}_i)\bigr)\, \mat{W}_h\transp,
133 + \label{eq:rnn-jacobian}
134 +\end{equation}
135 +where $\vect{a}_i = \mat{W}_h \vect{h}_{i-1} + \mat{W}_x \vect{x}_i +
136 +\vect{b}$ is the pre-activation. Bounding each factor by its largest
137 +singular value $\sigma_{\max}(\mat{W}_h)$ and using
138 +$|\tanh'| \le \gamma = 1$ gives
139 +\begin{equation}
140 + \left\lVert \frac{\partial \vect{h}_t}{\partial \vect{h}_k} \right\rVert
141 + \;\le\; \bigl(\gamma\, \sigma_{\max}(\mat{W}_h)\bigr)^{\,t-k}.
142 + \label{eq:rnn-jacobian-bound}
143 +\end{equation}
144 +
145 +Equation \eqref{eq:rnn-jacobian-bound} exposes the fundamental pathology
146 +of the simple RNN. If $\sigma_{\max}(\mat{W}_h) < 1/\gamma$, the bound
147 +decays exponentially in the time lag $t-k$: gradients \emph{vanish}, and
148 +the network cannot learn dependencies spanning more than a few dozen
149 +steps. Conversely, if the spectral radius of $\mat{W}_h$ exceeds $1$ — a
150 +necessary condition — the product can grow exponentially: gradients
151 +\emph{explode}, producing loss spikes and numerical overflow. Both regimes
152 +are generic; only a narrow band around unit gain propagates error signals
153 +faithfully over long horizons.
154 +
155 +\begin{remark}[Gradient clipping]
156 +Exploding gradients admit a simple remedy: rescale the gradient
157 +$\vect{g} = \nabla_\theta \Loss$ whenever its norm exceeds a threshold
158 +$\tau$,
159 +\begin{equation}
160 + \vect{g} \;\leftarrow\;
161 + \begin{cases}
162 + \dfrac{\tau}{\lVert \vect{g} \rVert}\, \vect{g}
163 + & \text{if } \lVert \vect{g} \rVert > \tau, \\[2ex]
164 + \vect{g} & \text{otherwise.}
165 + \end{cases}
166 + \label{eq:rnn-clip}
167 +\end{equation}
168 +Vanishing gradients have no comparably simple fix; they are an
169 +\emph{architectural} problem, and it is precisely this problem that the
170 +gated cells of Sections~\ref{sec:rnn-lstm} and~\ref{sec:rnn-gru} solve.
171 +\end{remark}
172 +
173 +Algorithm~\ref{alg:rnn-bptt} assembles the complete estimation procedure:
174 +a forward sweep that stores all pre-activations and states, a backward
175 +sweep that transports the error signal $\vect{\delta}_t$ from step $T$
176 +down to step $1$ while accumulating the shared-weight gradients of
177 +\eqref{eq:rnn-loss-grad}, followed by clipping \eqref{eq:rnn-clip} and a
178 +gradient step.
179 +
180 +\begin{algorithm}[htbp]
181 + \caption{Backpropagation through time (BPTT) for the simple RNN}
182 + \label{alg:rnn-bptt}
183 + \begin{algorithmic}[1]
184 + \Require sequence $(\vect{x}_1, \dots, \vect{x}_T)$, losses $\Loss_t$,
185 + parameters $(\mat{W}_x, \mat{W}_h, \mat{W}_y, \vect{b}, \vect{b}_y)$,
186 + learning rate $\eta$, clipping threshold $\tau$
187 + \State $\vect{h}_0 \gets \vect{0}$
188 + \For{$t = 1, \dots, T$} \Comment{forward pass: store all
189 + $\vect{a}_t, \vect{h}_t$}
190 + \State $\vect{a}_t \gets \mat{W}_h \vect{h}_{t-1}
191 + + \mat{W}_x \vect{x}_t + \vect{b}$;\quad
192 + $\vect{h}_t \gets \tanh(\vect{a}_t)$
193 + \State $\vect{y}_t \gets \mat{W}_y \vect{h}_t + \vect{b}_y$
194 + \EndFor
195 + \State $\nabla_{\mat{W}_x}, \nabla_{\mat{W}_h}, \nabla_{\mat{W}_y},
196 + \nabla_{\vect{b}}, \nabla_{\vect{b}_y} \gets \vect{0}$;\quad
197 + $\vect{\delta} \gets \vect{0}$
198 + \For{$t = T, \dots, 1$} \Comment{backward pass:
199 + $\vect{\delta}$ carries $\partial\Loss/\partial\vect{h}_t$}
200 + \State $\vect{\delta} \gets
201 + \mat{W}_y\transp\, \nabla_{\vect{y}_t}\Loss_t + \vect{\delta}$
202 + \State $\vect{\delta}_a \gets \vect{\delta} \odot
203 + \bigl(\vect{1} - \tanh^2(\vect{a}_t)\bigr)$
204 + \State $\nabla_{\mat{W}_y} \gets \nabla_{\mat{W}_y}
205 + + \nabla_{\vect{y}_t}\Loss_t\, \vect{h}_t\transp$;\quad
206 + $\nabla_{\vect{b}_y} \gets \nabla_{\vect{b}_y}
207 + + \nabla_{\vect{y}_t}\Loss_t$
208 + \State $\nabla_{\mat{W}_h} \gets \nabla_{\mat{W}_h}
209 + + \vect{\delta}_a \vect{h}_{t-1}\transp$;\quad
210 + $\nabla_{\mat{W}_x} \gets \nabla_{\mat{W}_x}
211 + + \vect{\delta}_a \vect{x}_t\transp$;\quad
212 + $\nabla_{\vect{b}} \gets \nabla_{\vect{b}} + \vect{\delta}_a$
213 + \State $\vect{\delta} \gets \mat{W}_h\transp \vect{\delta}_a$
214 + \Comment{transport the error to step $t-1$}
215 + \EndFor
216 + \State clip each gradient by \eqref{eq:rnn-clip} with threshold $\tau$
217 + \State update each parameter $\theta \gets \theta - \eta\,
218 + \nabla_\theta$
219 + \end{algorithmic}
220 +\end{algorithm}
221 +
222 +% ----------------------------------------------------------------------------
223 +\section{Long short-term memory (LSTM)}
224 +\label{sec:rnn-lstm}
225 +
226 +The long short-term memory network \cite{hochreiter1997} replaces the
227 +purely multiplicative recurrence \eqref{eq:rnn-hidden} with an
228 +\emph{additive} one. It introduces a second state vector, the \emph{cell
229 +state} $\vect{c}_t$, which traverses time along a path modified only by
230 +elementwise gating — the ``constant error carousel.'' Three learned gates,
231 +each a sigmoid layer reading the current input $\vect{x}_t$ and the
232 +previous hidden state $\vect{h}_{t-1}$, control what the cell forgets,
233 +what it writes, and what it exposes:
234 +\begin{align}
235 + \vect{f}_t &= \sigma\!\left(\mat{W}_f \vect{x}_t
236 + + \mat{U}_f \vect{h}_{t-1} + \vect{b}_f\right)
237 + && \text{(forget gate)}
238 + \label{eq:rnn-lstm-f} \\
239 + \vect{i}_t &= \sigma\!\left(\mat{W}_i \vect{x}_t
240 + + \mat{U}_i \vect{h}_{t-1} + \vect{b}_i\right)
241 + && \text{(input gate)}
242 + \label{eq:rnn-lstm-i} \\
243 + \vect{o}_t &= \sigma\!\left(\mat{W}_o \vect{x}_t
244 + + \mat{U}_o \vect{h}_{t-1} + \vect{b}_o\right)
245 + && \text{(output gate)}
246 + \label{eq:rnn-lstm-o} \\
247 + \tilde{\vect{c}}_t &= \tanh\!\left(\mat{W}_c \vect{x}_t
248 + + \mat{U}_c \vect{h}_{t-1} + \vect{b}_c\right)
249 + && \text{(candidate content)}
250 + \label{eq:rnn-lstm-ctilde} \\
251 + \vect{c}_t &= \vect{f}_t \odot \vect{c}_{t-1}
252 + + \vect{i}_t \odot \tilde{\vect{c}}_t
253 + && \text{(cell state update)}
254 + \label{eq:rnn-lstm-c} \\
255 + \vect{h}_t &= \vect{o}_t \odot \tanh(\vect{c}_t)
256 + && \text{(hidden state)}
257 + \label{eq:rnn-lstm-h}
258 +\end{align}
259 +Here $\sigma$ is the logistic sigmoid, so each gate takes values in
260 +$(0,1)^n$ and acts as a soft, differentiable switch applied coordinatewise
261 +through the Hadamard product $\odot$. The forget gate $\vect{f}_t$ decides
262 +how much of the previous cell $\vect{c}_{t-1}$ to retain; the input gate
263 +$\vect{i}_t$ decides how much of the new candidate $\tilde{\vect{c}}_t$ to
264 +write; the output gate $\vect{o}_t$ decides how much of the (squashed)
265 +cell to reveal in the hidden state. Figure~\ref{fig:rnn-lstm-cell} traces
266 +these six equations through the cell.
267 +
268 +\begin{property}[Constant error carousel]
269 +\label{prop:rnn-cec}
270 +Along the cell-state path of \eqref{eq:rnn-lstm-c}, the Jacobian of the
271 +recurrence is diagonal,
272 +\begin{equation}
273 + \frac{\partial \vect{c}_t}{\partial \vect{c}_{t-1}}
274 + = \operatorname{diag}(\vect{f}_t)
275 + \quad (\text{holding the gates fixed}),
276 + \label{eq:rnn-lstm-cec}
277 +\end{equation}
278 +with entries in $(0,1)$ controlled by the forget gate rather than by
279 +repeated multiplication with $\mat{W}_h$. When the network sets
280 +$\vect{f}_t \approx \vect{1}$, error signals flow backward through
281 +\eqref{eq:rnn-lstm-cec} essentially unattenuated over hundreds of steps,
282 +in contrast with the exponential bound \eqref{eq:rnn-jacobian-bound} of
283 +the simple RNN.
284 +\end{property}
285 +
286 +\begin{remark}
287 +A useful practical consequence of Property~\ref{prop:rnn-cec} is to
288 +initialize the forget-gate bias $\vect{b}_f$ to a positive value (e.g.\
289 +$1$ or $2$), so that $\vect{f}_t \approx \vect{1}$ at the start of
290 +training and the network begins by \emph{remembering}, only later learning
291 +what to forget.
292 +\end{remark}
293 +
294 +\begin{figure}[htbp]
295 + \centering
296 + \begin{tikzpicture}
297 + % ================= cell state line (top, violet) =================
298 + \node[mem, minimum width=1.2cm] (cprev) at (0.2,5) {$\vect{c}_{t-1}$};
299 + \node[op] (multf) at (2.6,5) {$\odot$};
300 + \node[op] (plus) at (6.4,5) {$+$};
301 + \node[mem, minimum width=1.2cm] (cnew) at (12.4,5) {$\vect{c}_t$};
302 + \draw[fleche, cmem!70!black, very thick] (cprev) -- (multf);
303 + \draw[fleche, cmem!70!black, very thick] (multf) -- (plus);
304 + \draw[fleche, cmem!70!black, very thick] (plus) -- (cnew);
305 + \fill[cmem!70!black] (8.6,5) circle (1.6pt);
306 + % ================= gates (bottom row) =================
307 + \node[gate] (fgate) at (2.6,1.2) {$\sigma$};
308 + \node[gate] (igate) at (4.6,1.2) {$\sigma$};
309 + \node[gate] (cgate) at (6.4,1.2) {$\tanh$};
310 + \node[gate] (ogate) at (10.4,1.2) {$\sigma$};
311 + % ================= interior op nodes =================
312 + \node[op] (multi) at (6.4,3.3) {$\odot$}; % i_t (.) c~_t
313 + \node[op] (tanhc) at (8.6,3.3) {$\tanh$}; % tanh(c_t)
314 + \node[op] (multo) at (10.4,2.2) {$\odot$}; % o_t (.) tanh(c_t)
315 + % ================= gate outputs =================
316 + \draw[fleche] (fgate) -- (multf)
317 + node[pos=0.55, right, etiquette] {$\vect{f}_t$};
318 + \draw[fleche, rounded corners=3pt] (igate.north) |- (multi.west)
319 + node[pos=0.25, right, etiquette] {$\vect{i}_t$};
320 + \draw[fleche] (cgate) -- (multi)
321 + node[pos=0.5, right, etiquette] {$\tilde{\vect{c}}_t$};
322 + \draw[fleche] (multi) -- (plus);
323 + \draw[fleche] (ogate) -- (multo)
324 + node[pos=0.5, right, etiquette] {$\vect{o}_t$};
325 + % ================= tanh branch from the state line =================
326 + \draw[fleche] (8.6,5) -- (tanhc);
327 + \draw[fleche, rounded corners=3pt] (tanhc.south) |- (multo.west);
328 + % ================= h_t output =================
329 + \node[mem, minimum width=1.2cm] (hnew) at (12.4,2.2) {$\vect{h}_t$};
330 + \draw[fleche] (multo) -- (hnew);
331 + \draw[fleche, rounded corners=3pt] (11.6,2.2) |- (12.4,0.6)
332 + node[pos=1.0, right, etiquette] {to $\vect{y}_t$};
333 + \fill[black!70] (11.6,2.2) circle (1.4pt);
334 + % ================= input trunk (bottom) =================
335 + \node[mem, minimum width=1.2cm] (hprev) at (-0.6,0) {$\vect{h}_{t-1}$};
336 + \node[ninput] (xt) at (1.2,-1.5) {$\vect{x}_t$};
337 + \draw[thick] (hprev.east) -- (10.4,0);
338 + \draw[fleche] (xt) -- (1.2,0);
339 + \foreach \x in {1.2, 2.6, 4.6, 6.4} \fill[black!70] (\x,0) circle (1.4pt);
340 + \draw[fleche] (2.6,0) -- (fgate.south);
341 + \draw[fleche] (4.6,0) -- (igate.south);
342 + \draw[fleche] (6.4,0) -- (cgate.south);
343 + \draw[fleche] (10.4,0) -- (ogate.south);
344 + % ================= cell frame =================
345 + \begin{scope}[on background layer]
346 + \node[draw=black!35, dashed, rounded corners=6pt, fill=black!2,
347 + fit={(1.7,-0.6) (11.7,5.75)}, inner sep=2pt] (frame) {};
348 + \end{scope}
349 + \node[etiquette, anchor=south west] at (1.75,5.85) {LSTM cell};
350 + \end{tikzpicture}
351 + \caption{The LSTM cell, tracing equations
352 + \eqref{eq:rnn-lstm-f}--\eqref{eq:rnn-lstm-h}. The cell state runs
353 + horizontally along the top (violet): it is first scaled by the forget
354 + gate ($\odot$ with $\vect{f}_t$), then incremented ($+$) with the
355 + gated candidate $\vect{i}_t \odot \tilde{\vect{c}}_t$. The three
356 + $\sigma$ gates and the $\tanh$ candidate layer (bottom, green) all
357 + read the shared input trunk carrying $\vect{h}_{t-1}$ and
358 + $\vect{x}_t$. The updated cell $\vect{c}_t$ is squashed by $\tanh$ and
359 + multiplied by the output gate $\vect{o}_t$ to produce the hidden state
360 + $\vect{h}_t$, which exits right and branches toward the output.}
361 + \label{fig:rnn-lstm-cell}
362 +\end{figure}
363 +
364 +% ----------------------------------------------------------------------------
365 +\section{Gated recurrent unit (GRU)}
366 +\label{sec:rnn-gru}
367 +
368 +The gated recurrent unit \cite{cho2014} is a streamlined gated cell that
369 +merges the LSTM's cell and hidden states into a single vector
370 +$\vect{h}_t$ and uses only two gates — roughly $25\%$ fewer parameters
371 +than the LSTM:
372 +\begin{align}
373 + \vect{z}_t &= \sigma\!\left(\mat{W}_z \vect{x}_t
374 + + \mat{U}_z \vect{h}_{t-1} + \vect{b}_z\right)
375 + && \text{(update gate)}
376 + \label{eq:rnn-gru-z} \\
377 + \vect{r}_t &= \sigma\!\left(\mat{W}_r \vect{x}_t
378 + + \mat{U}_r \vect{h}_{t-1} + \vect{b}_r\right)
379 + && \text{(reset gate)}
380 + \label{eq:rnn-gru-r} \\
381 + \tilde{\vect{h}}_t &= \tanh\!\left(\mat{W}_h \vect{x}_t
382 + + \mat{U}_h (\vect{r}_t \odot \vect{h}_{t-1}) + \vect{b}_h\right)
383 + && \text{(candidate state)}
384 + \label{eq:rnn-gru-htilde} \\
385 + \vect{h}_t &= (\vect{1} - \vect{z}_t) \odot \vect{h}_{t-1}
386 + + \vect{z}_t \odot \tilde{\vect{h}}_t
387 + && \text{(interpolation)}
388 + \label{eq:rnn-gru-h}
389 +\end{align}
390 +The reset gate $\vect{r}_t$ controls how much of the past state
391 +contributes to the candidate \eqref{eq:rnn-gru-htilde}: with
392 +$\vect{r}_t \approx \vect{0}$ the unit ignores its history and behaves
393 +like a freshly initialized network. The update gate $\vect{z}_t$ then
394 +interpolates \eqref{eq:rnn-gru-h} between copying the old state and
395 +writing the new candidate — the same leaky-integration principle as the
396 +LSTM's forget/input pair, realized as an explicit convex combination.
397 +Empirically, GRU and LSTM perform comparably across language and speech
398 +benchmarks, with the LSTM slightly more robust on tasks requiring precise
399 +counting; the GRU is often preferred when parameter economy or training
400 +speed matters.
401 +
402 +% ----------------------------------------------------------------------------
403 +\section{Bidirectional networks and sequence-to-sequence learning}
404 +\label{sec:rnn-seq2seq}
405 +
406 +\paragraph{Bidirectional RNNs.}
407 +The recurrences above are causal: $\vect{h}_t$ summarizes only
408 +$\vect{x}_1, \dots, \vect{x}_t$. Many labeling tasks (tagging, speech
409 +frames, contextual encoding) benefit from future context as well. A
410 +bidirectional RNN runs two independent recurrent networks over the
411 +sequence — one forward, one backward — and concatenates their states:
412 +\begin{equation}
413 + \overrightarrow{\vect{h}}_t
414 + = f\!\left(\overrightarrow{\mat{W}} \vect{x}_t
415 + + \overrightarrow{\mat{U}}\, \overrightarrow{\vect{h}}_{t-1}\right),
416 + \qquad
417 + \overleftarrow{\vect{h}}_t
418 + = f\!\left(\overleftarrow{\mat{W}} \vect{x}_t
419 + + \overleftarrow{\mat{U}}\, \overleftarrow{\vect{h}}_{t+1}\right),
420 + \label{eq:rnn-bidir}
421 +\end{equation}
422 +\begin{equation}
423 + \vect{y}_t = g\!\left(\mat{V}\,
424 + [\,\overrightarrow{\vect{h}}_t \,;\, \overleftarrow{\vect{h}}_t\,]
425 + + \vect{b}\right),
426 + \label{eq:rnn-bidir-out}
427 +\end{equation}
428 +so each output sees both past and future. The price is that the full
429 +sequence must be available in advance: bidirectional models suit offline
430 +labeling, not streaming generation.
431 +
432 +\paragraph{Encoder--decoder (seq2seq).}
433 +To map an input sequence to an output sequence of different length —
434 +machine translation being the canonical example — the encoder--decoder
435 +architecture \cite{cho2014} uses two recurrent networks. An
436 +\emph{encoder} consumes the source $\vect{x}_1, \dots, \vect{x}_{T_x}$
437 +and compresses it into a context vector $\vect{c}$ (typically its final
438 +hidden state); a \emph{decoder} then generates the target
439 +autoregressively, each token conditioned on the context and on all
440 +previously generated tokens:
441 +\begin{equation}
442 + p(\vect{y}_1, \dots, \vect{y}_{T'} \mid
443 + \vect{x}_1, \dots, \vect{x}_{T_x})
444 + = \prod_{t=1}^{T'}
445 + p\!\left(\vect{y}_t \mid \vect{y}_{<t},\, \vect{c}\right).
446 + \label{eq:rnn-seq2seq}
447 +\end{equation}
448 +Training maximizes the log-likelihood of \eqref{eq:rnn-seq2seq} with
449 +\emph{teacher forcing} (feeding the ground-truth $\vect{y}_{t-1}$ as the
450 +decoder input at step $t$); inference replaces it with the model's own
451 +predictions, usually explored with beam search.
452 +Figure~\ref{fig:rnn-seq2seq} sketches the architecture.
453 +
454 +\begin{figure}[htbp]
455 + \centering
456 + \begin{tikzpicture}
457 + % encoder
458 + \foreach \i in {1,2,3} {
459 + \node[ninput] (ex\i) at (\i*1.9-1.9, 0) {$\vect{x}_\i$};
460 + \node[blochidden, minimum width=1.3cm] (ec\i) at (\i*1.9-1.9, 1.6) {};
461 + \draw[fleche] (ex\i) -- (ec\i);
462 + }
463 + \draw[fleche] (ec1) -- (ec2);
464 + \draw[fleche] (ec2) -- (ec3);
465 + \node[etiquette] at (1.9, 2.35) {encoder};
466 + % context
467 + \node[mem, minimum width=1.1cm] (ctx) at (5.5, 1.6) {$\vect{c}$};
468 + \draw[fleche] (ec3) -- (ctx);
469 + % decoder
470 + \foreach \i in {1,2,3} {
471 + \node[blocoutput, minimum width=1.3cm] (dc\i) at (5.6+\i*1.9, 1.6) {};
472 + \node[noutput] (dy\i) at (5.6+\i*1.9, 3.2) {$\vect{y}_\i$};
473 + \draw[fleche] (dc\i) -- (dy\i);
474 + }
475 + \draw[fleche] (ctx) -- (dc1);
476 + \draw[fleche] (dc1) -- (dc2);
477 + \draw[fleche] (dc2) -- (dc3);
478 + \node[etiquette] at (9.4, 0.6) {decoder};
479 + % autoregressive feedback (dashed)
480 + \draw[flechep] (dy1.east) to[out=-30,in=120]
481 + node[pos=0.4, above=1pt, etiquette] {$\vect{y}_{t-1}$}
482 + ([xshift=-8pt]dc2.north);
483 + \draw[flechep] (dy2.east) to[out=-30,in=120]
484 + ([xshift=-8pt]dc3.north);
485 + \end{tikzpicture}
486 + \caption{Encoder--decoder (seq2seq) architecture. The encoder compresses
487 + the source sequence into a context vector $\vect{c}$; the decoder
488 + generates the target autoregressively following
489 + \eqref{eq:rnn-seq2seq}, each step receiving the previously emitted
490 + token (dashed arrows).}
491 + \label{fig:rnn-seq2seq}
492 +\end{figure}
493 +
494 +\begin{remark}[The bottleneck that led to attention]
495 +The fixed-size context vector $\vect{c}$ in \eqref{eq:rnn-seq2seq} is an
496 +information bottleneck: translation quality degrades visibly on long
497 +sentences, because an entire source sequence must be squeezed into a
498 +single vector. Letting the decoder look back at \emph{all} encoder states
499 +— attention — removes this bottleneck and is the subject of the next
500 +chapter.
501 +\end{remark}
added chapters/06-attention.tex +590 −0
@@ -0,0 +1,590 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 6 : Attention and Transformers (chapters/06-attention.tex)
6 +% ============================================================================
7 +
8 +\chapter{Attention and Transformers}\label{chap:attention}
9 +
10 +Recurrent networks process a sequence one position at a time: the hidden
11 +state is a bottleneck through which all past information must flow, training
12 +cannot be parallelized along the time axis, and interactions between distant
13 +positions must survive a long chain of state updates. The Transformer
14 +\cite{vaswani2017} removed this bottleneck in a single stroke. It dispenses
15 +entirely with recurrence and convolution, and models every pairwise
16 +interaction between sequence positions \emph{directly}, through a mechanism
17 +called \emph{attention}. Every position can consult every other position in
18 +one step, the whole computation is expressed as a handful of dense matrix
19 +products that parallelize perfectly, and the maximum path length between any
20 +two positions drops to $O(1)$. This chapter builds the architecture from its
21 +smallest component — the scaled dot-product — to the complete
22 +encoder--decoder stack, and closes with the computational trade-offs and the
23 +three model families (BERT, GPT, T5) that grew out of it.
24 +
25 +% ----------------------------------------------------------------------------
26 +\section{From Recurrence to Attention}
27 +% ----------------------------------------------------------------------------
28 +
29 +The guiding intuition is retrieval. Suppose each position of a sequence
30 +emits a \emph{query} describing the information it is looking for, while
31 +every position also exposes a \emph{key} describing the information it
32 +holds, together with a \emph{value} carrying the actual content. Attention
33 +answers each query by comparing it against all keys, converting the
34 +similarity scores into weights, and returning the corresponding weighted
35 +average of the values. The result is a \emph{soft} dictionary lookup: fully
36 +differentiable, so it can be trained end to end by the backpropagation
37 +machinery of the earlier chapters, and content-based, so the pattern of
38 +interaction is decided by the data rather than fixed by the architecture.
39 +
40 +Formally, we work with three matrices: queries
41 +$\mat{Q} \in \R^{n \times d_k}$, keys $\mat{K} \in \R^{m \times d_k}$ and
42 +values $\mat{V} \in \R^{m \times d_v}$, where $n$ is the number of query
43 +positions, $m$ the number of key--value positions, $d_k$ the dimension in
44 +which queries and keys are compared, and $d_v$ the dimension of the
45 +returned content. In \emph{self-attention}, all three are linear
46 +projections of the same sequence ($m = n$); in \emph{cross-attention},
47 +queries come from one sequence and keys/values from another.
48 +
49 +% ----------------------------------------------------------------------------
50 +\section{Scaled Dot-Product Attention}\label{sec:att-sdpa}
51 +% ----------------------------------------------------------------------------
52 +
53 +\begin{definition}[Scaled dot-product attention]\label{def:att-sdpa}
54 +Given $\mat{Q} \in \R^{n \times d_k}$, $\mat{K} \in \R^{m \times d_k}$ and
55 +$\mat{V} \in \R^{m \times d_v}$, scaled dot-product attention is the map
56 +\begin{equation}
57 + \mathrm{Attention}(\mat{Q}, \mat{K}, \mat{V})
58 + \;=\;
59 + \softmax\!\left( \frac{\mat{Q}\mat{K}\transp}{\sqrt{d_k}} \right) \mat{V}
60 + \;\in\; \R^{n \times d_v},
61 + \label{eq:att-sdpa}
62 +\end{equation}
63 +where the softmax is applied row-wise \cite{vaswani2017}.
64 +\end{definition}
65 +
66 +Reading \eqref{eq:att-sdpa} from the inside out: the Gram matrix
67 +$\mat{Q}\mat{K}\transp \in \R^{n \times m}$ collects all $n \times m$ dot
68 +products between queries and keys; dividing by $\sqrt{d_k}$ rescales these
69 +similarity scores; the row-wise softmax turns each row into a probability
70 +distribution over the $m$ source positions; and the final product with
71 +$\mat{V}$ computes, for each query, the corresponding convex combination of
72 +value vectors. Writing $\vect{q}_i$ for the $i$-th query and
73 +$\vect{k}_j, \vect{v}_j$ for the $j$-th key and value, the attention weight
74 +that position $i$ places on position $j$ is
75 +\begin{equation}
76 + \alpha_{ij}
77 + \;=\;
78 + \frac{\exp\!\big( \vect{q}_i\transp \vect{k}_j / \sqrt{d_k} \big)}
79 + {\sum_{j'=1}^{m} \exp\!\big( \vect{q}_i\transp \vect{k}_{j'} / \sqrt{d_k} \big)},
80 + \qquad
81 + \sum_{j=1}^{m} \alpha_{ij} = 1,
82 + \label{eq:att-alpha}
83 +\end{equation}
84 +and the $i$-th output row is $\sum_j \alpha_{ij}\vect{v}_j$. The whole
85 +computation is displayed in Figure~\ref{fig:att-sdpa-flow}.
86 +
87 +\begin{figure}[htbp]
88 + \centering
89 + \begin{tikzpicture}[font=\small]
90 + % --- input blocks ---
91 + \node[blocinput, minimum width=1.2cm] (Q) at (-1.4, 0) {$\mat{Q}$};
92 + \node[blocinput, minimum width=1.2cm] (K) at ( 1.4, 0) {$\mat{K}$};
93 + \node[blocinput, minimum width=1.2cm] (V) at ( 4.6, 0) {$\mat{V}$};
94 + % --- pipeline ---
95 + \node[op] (mm1) at (0, 1.5) {$\times$};
96 + \node[bloc, minimum width=2.7cm] (scale) at (0, 2.7) {Scale by $1/\sqrt{d_k}$};
97 + \node[bloc, minimum width=2.7cm, dashed] (mask) at (0, 3.9) {Mask (optional)};
98 + \node[gate, minimum width=2.7cm] (sm) at (0, 5.1) {$\softmax$ (row-wise)};
99 + \node[op] (mm2) at (0, 6.4) {$\times$};
100 + \node[blocoutput, minimum width=3.4cm] (out) at (0, 7.6)
101 + {$\mathrm{Attention}(\mat{Q},\mat{K},\mat{V})$};
102 + % --- arrows ---
103 + \draw[fleche] (Q) -- (mm1);
104 + \draw[fleche] (K) -- (mm1);
105 + \draw[fleche] (mm1) -- (scale);
106 + \draw[fleche] (scale)-- (mask);
107 + \draw[fleche] (mask) -- (sm);
108 + \draw[fleche] (sm) -- (mm2);
109 + \draw[fleche, rounded corners=3pt] (V.north) -- (4.6, 6.4) -- (mm2);
110 + \draw[fleche] (mm2) -- (out);
111 + % --- annotations ---
112 + \node[etiquette, anchor=west] at (0.55, 1.5)
113 + {$\mat{Q}\mat{K}\transp \in \R^{n \times m}$};
114 + \node[etiquette, anchor=west, align=left] at (1.75, 5.1)
115 + {weights $\alpha_{ij}$,\\rows sum to $1$};
116 + \node[etiquette, anchor=west] at (0.55, 6.9)
117 + {$\in \R^{n \times d_v}$};
118 + \node[etiquette, anchor=north] at (4.6, -0.55)
119 + {$\R^{m \times d_v}$};
120 + \node[etiquette, anchor=north] at (-1.4, -0.55) {$\R^{n \times d_k}$};
121 + \node[etiquette, anchor=north] at ( 1.4, -0.55) {$\R^{m \times d_k}$};
122 + \end{tikzpicture}
123 + \caption{Computational flow of scaled dot-product attention,
124 + equation~\eqref{eq:att-sdpa}. Queries and keys meet in a matrix product,
125 + the scores are rescaled by $1/\sqrt{d_k}$, optionally masked
126 + (Section~\ref{sec:att-block}), normalized row-wise by a softmax, and the
127 + resulting weights average the values.}
128 + \label{fig:att-sdpa-flow}
129 +\end{figure}
130 +
131 +\subsection{Why the factor \texorpdfstring{$1/\sqrt{d_k}$}{1/sqrt(dk)}?}
132 +
133 +The scaling factor is not cosmetic; without it, training destabilizes as
134 +$d_k$ grows. The reason is a simple variance computation.
135 +
136 +\begin{property}[Variance of a dot product]\label{prop:att-variance}
137 +Let $\vect{q}, \vect{k} \in \R^{d_k}$ have independent components with
138 +$\E[q_i] = \E[k_i] = 0$ and $\E[q_i^2] = \E[k_i^2] = 1$. Then the dot
139 +product $s = \vect{q}\transp\vect{k}$ satisfies
140 +\begin{equation}
141 + \E[s] = 0,
142 + \qquad
143 + \operatorname{Var}(s)
144 + = \sum_{i=1}^{d_k} \E[q_i^2]\,\E[k_i^2]
145 + = d_k .
146 + \label{eq:att-variance}
147 +\end{equation}
148 +Hence $s/\sqrt{d_k}$ has unit variance, independently of $d_k$.
149 +\end{property}
150 +
151 +\begin{proof}
152 +By independence, $\E[s] = \sum_i \E[q_i]\E[k_i] = 0$. The terms $q_i k_i$
153 +are uncorrelated with mean zero, so
154 +$\operatorname{Var}(s) = \sum_i \E[q_i^2 k_i^2]
155 + = \sum_i \E[q_i^2]\,\E[k_i^2] = d_k$.
156 +\end{proof}
157 +
158 +The consequence for \eqref{eq:att-alpha} is direct. With the typical head
159 +dimension $d_k = 64$, unscaled logits would have standard deviation
160 +$\sqrt{64} = 8$: the softmax would saturate, placing almost all mass on the
161 +single largest score. In the saturated regime the Jacobian of the softmax is
162 +nearly zero, so the gradients flowing back into $\mat{Q}$ and $\mat{K}$
163 +vanish and the attention pattern stops learning. Dividing by $\sqrt{d_k}$
164 +keeps the logits at unit variance regardless of the head width, holding the
165 +softmax in its responsive regime throughout training \cite{vaswani2017}.
166 +
167 +\begin{remark}
168 +Attention imposes no notion of distance: position $1$ reaches position
169 +$10\,000$ exactly as easily as its neighbour. This is the source of the
170 +Transformer's $O(1)$ maximum path length — and also the reason position
171 +information must be injected explicitly
172 +(Section~\ref{sec:att-positional}).
173 +\end{remark}
174 +
175 +% ----------------------------------------------------------------------------
176 +\section{Multi-Head Attention}\label{sec:att-multihead}
177 +% ----------------------------------------------------------------------------
178 +
179 +A single attention map forces one pattern of interaction per layer: each
180 +query produces one distribution over source positions, and all $d_v$
181 +channels of the output are averaged with those same weights. The
182 +Transformer instead projects queries, keys and values into $h$
183 +lower-dimensional subspaces and runs attention in each subspace in
184 +parallel. With learned projection matrices
185 +$\mat{W}_i^Q \in \R^{d_{\text{model}} \times d_k}$,
186 +$\mat{W}_i^K \in \R^{d_{\text{model}} \times d_k}$ and
187 +$\mat{W}_i^V \in \R^{d_{\text{model}} \times d_v}$, head $i$ computes
188 +\begin{equation}
189 + \mathrm{head}_i
190 + \;=\;
191 + \mathrm{Attention}\big( \mat{Q}\mat{W}_i^Q,\;
192 + \mat{K}\mat{W}_i^K,\;
193 + \mat{V}\mat{W}_i^V \big),
194 + \qquad i = 1, \dots, h,
195 + \label{eq:att-head}
196 +\end{equation}
197 +and the heads are concatenated and mixed by an output matrix
198 +$\mat{W}^O \in \R^{h d_v \times d_{\text{model}}}$:
199 +\begin{equation}
200 + \mathrm{MultiHead}(\mat{Q}, \mat{K}, \mat{V})
201 + \;=\;
202 + \mathrm{Concat}\big( \mathrm{head}_1, \dots, \mathrm{head}_h \big)\,
203 + \mat{W}^O .
204 + \label{eq:att-multihead}
205 +\end{equation}
206 +
207 +In the base model of \cite{vaswani2017}, $h = 8$ and
208 +$d_k = d_v = d_{\text{model}}/h = 512/8 = 64$, so the total computational
209 +cost matches that of single-head attention at full width. What multi-head
210 +buys is expressive diversity: because each head owns its own projections
211 +\eqref{eq:att-head}, different heads can attend according to different
212 +criteria at the same time — one tracking short-range syntax, another
213 +long-range agreement, another positional neighbourhoods. Empirically, heads
214 +do specialize in this way, and the concatenation
215 +\eqref{eq:att-multihead} lets the next layer combine their findings.
216 +
217 +% ----------------------------------------------------------------------------
218 +\section{Positional Encoding}\label{sec:att-positional}
219 +% ----------------------------------------------------------------------------
220 +
221 +Equations \eqref{eq:att-sdpa}--\eqref{eq:att-multihead} are permutation
222 +\emph{equivariant}: shuffling the rows of the inputs merely shuffles the
223 +rows of the output. Word order — the backbone of syntax — is invisible to
224 +the mechanism. The Transformer therefore adds a deterministic
225 +\emph{positional encoding} to the token embeddings before the first layer.
226 +For position $pos$ and dimension pair index $i$, the sinusoidal encoding
227 +of \cite{vaswani2017} is
228 +\begin{align}
229 + PE_{(pos,\, 2i)} &= \sin\!\left( \frac{pos}{10000^{2i/d_{\text{model}}}} \right),
230 + \label{eq:att-pe-sin} \\
231 + PE_{(pos,\, 2i+1)} &= \cos\!\left( \frac{pos}{10000^{2i/d_{\text{model}}}} \right).
232 + \label{eq:att-pe-cos}
233 +\end{align}
234 +Each coordinate pair traces a sinusoid whose wavelength grows geometrically
235 +from $2\pi$ to $10000 \cdot 2\pi$ as $i$ increases: low dimensions
236 +oscillate rapidly and resolve fine position differences, high dimensions
237 +vary slowly and encode coarse position.
238 +
239 +\begin{property}[Relative positions are linear]\label{prop:att-pe-linear}
240 +For any fixed offset $k$, the pair of components $PE_{(pos+k,\,2i)}$ and
241 +$PE_{(pos+k,\,2i+1)}$ is obtained from the pair $PE_{(pos,\,2i)}$ and
242 +$PE_{(pos,\,2i+1)}$ by a rotation whose angle
243 +depends only on $k$ and $i$ — a consequence of the angle-addition formulas
244 +for sine and cosine. Relative displacement is therefore a fixed
245 +\emph{linear} map of the encoding, which a learned projection can pick up
246 +easily.
247 +\end{property}
248 +
249 +\begin{remark}
250 +The sinusoidal scheme requires no learned parameters and extends, in
251 +principle, to sequence lengths never seen in training. Learned absolute
252 +position embeddings are an equally common alternative; they trade
253 +extrapolation for flexibility.
254 +\end{remark}
255 +
256 +% ----------------------------------------------------------------------------
257 +\section{The Transformer Block}\label{sec:att-block}
258 +% ----------------------------------------------------------------------------
259 +
260 +Attention alone is a weighted average — a linear operation for fixed
261 +weights. The Transformer interleaves it with three further ingredients:
262 +residual connections, layer normalization, and a position-wise
263 +feed-forward network. Each sub-layer (attention or feed-forward) is wrapped
264 +as
265 +\begin{equation}
266 + \vect{x} \;\longmapsto\;
267 + \mathrm{LayerNorm}\big( \vect{x} + \mathrm{Sublayer}(\vect{x}) \big),
268 + \label{eq:att-addnorm}
269 +\end{equation}
270 +the ``Add \& Norm'' of the diagrams: the residual sum preserves a direct
271 +gradient path through the depth of the stack, and the normalization,
272 +\begin{equation}
273 + \mathrm{LayerNorm}(\vect{x})
274 + \;=\;
275 + \vect{\gamma} \odot \frac{\vect{x} - \mu}{\sigma} + \vect{\beta},
276 + \qquad
277 + \mu = \frac{1}{d}\sum_{i=1}^{d} x_i,
278 + \quad
279 + \sigma^2 = \frac{1}{d}\sum_{i=1}^{d} (x_i - \mu)^2,
280 + \label{eq:att-layernorm}
281 +\end{equation}
282 +computed over the feature dimension of each position independently, keeps
283 +activations at a stable scale no matter the sequence length or batch
284 +composition. Between the attention sub-layers sits a two-layer network
285 +applied identically — and independently — at every position:
286 +\begin{equation}
287 + \mathrm{FFN}(\vect{x})
288 + \;=\;
289 + \max\big( \vect{0},\; \vect{x}\mat{W}_1 + \vect{b}_1 \big)\, \mat{W}_2
290 + + \vect{b}_2,
291 + \label{eq:att-ffn}
292 +\end{equation}
293 +with an inner dimension $d_{f\!f} = 2048$, a fourfold expansion over
294 +$d_{\text{model}} = 512$ \cite{vaswani2017}. If attention is where
295 +positions \emph{communicate}, the FFN \eqref{eq:att-ffn} is where each
296 +position \emph{computes} on what it has gathered.
297 +
298 +\begin{remark}[Pre-LN versus post-LN]
299 +Equation~\eqref{eq:att-addnorm} is the original ``post-LN'' arrangement.
300 +Most modern implementations normalize \emph{before} the sub-layer
301 +($\vect{x} + \mathrm{Sublayer}(\mathrm{LayerNorm}(\vect{x}))$), which
302 +keeps the residual path entirely free of normalization and trains more
303 +stably at large depth.
304 +\end{remark}
305 +
306 +\subsection{The causal mask}
307 +
308 +A language model generating text left to right must not let position $i$
309 +peek at positions $j > i$: prediction of the next token would otherwise be
310 +trivial in training and impossible at inference. Autoregressive decoding is
311 +enforced \emph{inside} the attention by adding a mask matrix
312 +$\mat{M} \in \R^{n \times n}$ to the scaled scores,
313 +\begin{equation}
314 + M_{ij}
315 + \;=\;
316 + \begin{cases}
317 + 0 & \text{if } j \le i, \\[2pt]
318 + -\infty & \text{if } j > i,
319 + \end{cases}
320 + \label{eq:att-mask}
321 +\end{equation}
322 +so that masked attention reads
323 +\begin{equation}
324 + \mathrm{MaskedAttention}(\mat{Q}, \mat{K}, \mat{V})
325 + \;=\;
326 + \softmax\!\left( \frac{\mat{Q}\mat{K}\transp}{\sqrt{d_k}} + \mat{M}
327 + \right) \mat{V}.
328 + \label{eq:att-masked}
329 +\end{equation}
330 +The $-\infty$ entries become exact zeros after the softmax
331 +\eqref{eq:att-alpha}, so each position attends only to itself and its past.
332 +This is the ``Mask (optional)'' stage of Figure~\ref{fig:att-sdpa-flow}.
333 +
334 +% ----------------------------------------------------------------------------
335 +\section{The Complete Encoder--Decoder Architecture}\label{sec:att-arch}
336 +% ----------------------------------------------------------------------------
337 +
338 +The full Transformer of \cite{vaswani2017}, designed for
339 +sequence-to-sequence tasks such as translation, assembles these pieces into
340 +the two-column architecture of Figure~\ref{fig:att-transformer}.
341 +
342 +\begin{itemize}
343 + \item \textbf{Encoder} (left column): the input tokens are embedded,
344 + positional encodings \eqref{eq:att-pe-sin}--\eqref{eq:att-pe-cos} are
345 + added, and the result passes through a stack of $N = 6$ identical
346 + layers. Each layer applies multi-head \emph{self}-attention
347 + \eqref{eq:att-multihead} — every input position attends to every other
348 + — followed by the feed-forward network \eqref{eq:att-ffn}, each wrapped
349 + in Add \& Norm \eqref{eq:att-addnorm}.
350 + \item \textbf{Decoder} (right column): the output tokens, shifted right
351 + by one position, are embedded and encoded likewise, then pass through
352 + $N = 6$ layers of \emph{three} sub-layers each: masked self-attention
353 + \eqref{eq:att-masked}, which respects causality; \emph{cross}-attention,
354 + whose queries come from the decoder while keys and values come from
355 + the encoder output (the $\mat{K}, \mat{V}$ arrow in
356 + Figure~\ref{fig:att-transformer}); and the feed-forward network.
357 + \item \textbf{Head}: a final linear projection to vocabulary size and a
358 + softmax produce the next-token distribution.
359 +\end{itemize}
360 +
361 +Cross-attention is where the two columns meet: each partially generated
362 +target position formulates a query, and retrieves from the source sequence
363 +the content most relevant to producing the next token — a learned,
364 +differentiable alignment between input and output.
365 +
366 +At inference time the decoder is run autoregressively: the source is
367 +encoded once, and tokens are emitted one at a time, each new token being
368 +appended to the prefix that conditions the next step.
369 +Algorithm~\ref{alg:att-decode} states the greedy variant, which commits at
370 +every step to the most probable token; beam search generalizes it by
371 +carrying the $B$ most probable prefixes instead of one.
372 +
373 +\begin{algorithm}[htbp]
374 + \caption{Greedy autoregressive decoding with a Transformer}
375 + \label{alg:att-decode}
376 + \begin{algorithmic}[1]
377 + \Require source tokens $(x_1, \dots, x_{T_x})$, trained encoder--decoder,
378 + maximum length $T_{\max}$
379 + \State $\mat{H}_{\mathrm{enc}} \gets
380 + \mathrm{Encoder}(x_1, \dots, x_{T_x})$
381 + \Comment{encode the source once; reused at every step}
382 + \State $\vect{y} \gets (\langle\mathrm{BOS}\rangle)$
383 + \Comment{generated prefix}
384 + \For{$t = 1, \dots, T_{\max}$}
385 + \State $\mat{Z} \gets$ embed $\vect{y}$ and add positional
386 + encodings \eqref{eq:att-pe-sin}--\eqref{eq:att-pe-cos}
387 + \State $\mat{Z} \gets$ decoder stack applied to $\mat{Z}$:
388 + masked self-attention \eqref{eq:att-masked},
389 + cross-attention on $\mat{H}_{\mathrm{enc}}$, FFN
390 + \eqref{eq:att-ffn}, each with Add \& Norm \eqref{eq:att-addnorm}
391 + \State $\vect{p} \gets \softmax\!\left(\mat{W}_{\mathrm{vocab}}\,
392 + \vect{z}_t + \vect{b}\right)$
393 + \Comment{$\vect{z}_t$: last position of $\mat{Z}$}
394 + \State $y_{t} \gets \argmax_{v}\; p_v$
395 + \State append $y_t$ to $\vect{y}$
396 + \If{$y_t = \langle\mathrm{EOS}\rangle$} \State \textbf{break}
397 + \EndIf
398 + \EndFor
399 + \State \Return $\vect{y}$
400 + \end{algorithmic}
401 +\end{algorithm}
402 +
403 +\begin{figure}[p]
404 + \centering
405 + \begin{tikzpicture}[font=\small]
406 + % ================= ENCODER column =================
407 + \node[etiquette] (enc-in) at (0, -0.15) {Inputs};
408 + \node[blocinput, minimum width=3.2cm] (enc-emb) at (0, 0.9) {Input Embedding};
409 + \node[op] (enc-pe) at (0, 2.0) {$+$};
410 + \node[etiquette, align=center] at (-1.75, 2.0) {Positional\\Encoding};
411 + \node[blochidden, minimum width=3.4cm, align=center] (enc-attn) at (0, 3.5)
412 + {Multi-Head\\Self-Attention};
413 + \node[bloc, minimum width=3.4cm] (enc-an1) at (0, 4.9) {Add \& Norm};
414 + \node[blochidden, minimum width=3.4cm] (enc-ffn) at (0, 6.1) {Feed-Forward};
415 + \node[bloc, minimum width=3.4cm] (enc-an2) at (0, 7.3) {Add \& Norm};
416 + % main arrows (encoder)
417 + \draw[fleche] (enc-in) -- (enc-emb);
418 + \draw[fleche] (enc-emb) -- (enc-pe);
419 + \draw[fleche] (enc-pe) -- (enc-attn);
420 + \draw[fleche] (enc-attn)-- (enc-an1);
421 + \draw[fleche] (enc-an1) -- (enc-ffn);
422 + \draw[fleche] (enc-ffn) -- (enc-an2);
423 + % residual arcs (outer/left side)
424 + \draw[fleche, semithick, rounded corners=2pt]
425 + (0, 2.72) -- (-2.35, 2.72) -- (-2.35, 4.9) -- (enc-an1.west);
426 + \draw[fleche, semithick, rounded corners=2pt]
427 + (0, 5.5) -- (-2.35, 5.5) -- (-2.35, 7.3) -- (enc-an2.west);
428 + % encoder frame (xN)
429 + \coordinate (enc-fit-w) at (-2.6, 4.9);
430 + \begin{scope}[on background layer]
431 + \node[draw=black!55, rounded corners=4pt, fill=chidden!6, inner sep=9pt,
432 + fit=(enc-attn)(enc-an1)(enc-ffn)(enc-an2)(enc-fit-w)]
433 + (encframe) {};
434 + \end{scope}
435 + \node[font=\small\bfseries, anchor=east] at ($(encframe.west)+(-0.12,0)$)
436 + {$\times N$};
437 + % ================= DECODER column =================
438 + \node[etiquette, align=center] (dec-in) at (7.4, -0.15)
439 + {Outputs (shifted right)};
440 + \node[blocinput, minimum width=3.2cm] (dec-emb) at (7.4, 0.9) {Output Embedding};
441 + \node[op] (dec-pe) at (7.4, 2.0) {$+$};
442 + \node[etiquette, align=center] at (9.15, 2.0) {Positional\\Encoding};
443 + \node[blochidden, minimum width=3.4cm, align=center] (dec-attn) at (7.4, 3.5)
444 + {Masked Multi-Head\\Self-Attention};
445 + \node[bloc, minimum width=3.4cm] (dec-an1) at (7.4, 4.9) {Add \& Norm};
446 + \node[blochidden, minimum width=3.4cm, align=center] (dec-cross) at (7.4, 6.3)
447 + {Multi-Head\\Cross-Attention};
448 + \node[bloc, minimum width=3.4cm] (dec-an2) at (7.4, 7.7) {Add \& Norm};
449 + \node[blochidden, minimum width=3.4cm] (dec-ffn) at (7.4, 8.9) {Feed-Forward};
450 + \node[bloc, minimum width=3.4cm] (dec-an3) at (7.4, 10.1) {Add \& Norm};
451 + \node[blocoutput, minimum width=3.4cm] (dec-lin) at (7.4, 11.4) {Linear};
452 + \node[blocoutput, minimum width=3.4cm] (dec-sm) at (7.4, 12.5) {$\softmax$};
453 + \node[etiquette] (dec-out) at (7.4, 13.45) {Output probabilities};
454 + % main arrows (decoder)
455 + \draw[fleche] (dec-in) -- (dec-emb);
456 + \draw[fleche] (dec-emb) -- (dec-pe);
457 + \draw[fleche] (dec-pe) -- (dec-attn);
458 + \draw[fleche] (dec-attn) -- (dec-an1);
459 + \draw[fleche] (dec-an1) -- (dec-cross);
460 + \draw[fleche] (dec-cross)-- (dec-an2);
461 + \draw[fleche] (dec-an2) -- (dec-ffn);
462 + \draw[fleche] (dec-ffn) -- (dec-an3);
463 + \draw[fleche] (dec-an3) -- (dec-lin);
464 + \draw[fleche] (dec-lin) -- (dec-sm);
465 + \draw[fleche] (dec-sm) -- (dec-out);
466 + % residual arcs (outer/right side)
467 + \draw[fleche, semithick, rounded corners=2pt]
468 + (7.4, 2.72) -- (9.75, 2.72) -- (9.75, 4.9) -- (dec-an1.east);
469 + \draw[fleche, semithick, rounded corners=2pt]
470 + (7.4, 5.5) -- (9.75, 5.5) -- (9.75, 7.7) -- (dec-an2.east);
471 + \draw[fleche, semithick, rounded corners=2pt]
472 + (7.4, 8.3) -- (9.75, 8.3) -- (9.75, 10.1) -- (dec-an3.east);
473 + % decoder frame (xN)
474 + \coordinate (dec-fit-e) at (10.0, 6.3);
475 + \begin{scope}[on background layer]
476 + \node[draw=black!55, rounded corners=4pt, fill=chidden!6, inner sep=9pt,
477 + fit=(dec-attn)(dec-an1)(dec-cross)(dec-an2)(dec-ffn)(dec-an3)(dec-fit-e)]
478 + (decframe) {};
479 + \end{scope}
480 + \node[font=\small\bfseries, anchor=west] at ($(decframe.east)+(0.12,0)$)
481 + {$\times N$};
482 + % ================= K,V bridge encoder -> decoder =================
483 + \draw[fleche, rounded corners=3pt]
484 + (enc-an2.north) -- (0, 8.6) -- (3.6, 8.6) -- (3.6, 6.3)
485 + -- (dec-cross.west);
486 + \node[etiquette, anchor=south] at (1.8, 8.63) {$\mat{K},\ \mat{V}$ (encoder output)};
487 + \end{tikzpicture}
488 + \caption{The complete Transformer encoder--decoder architecture
489 + \cite{vaswani2017}. Left: the encoder — embedding, positional encoding
490 + \eqref{eq:att-pe-sin}--\eqref{eq:att-pe-cos}, then $N$ identical layers
491 + of self-attention and feed-forward, each wrapped in Add \& Norm
492 + \eqref{eq:att-addnorm} with the residual paths drawn as outer arcs.
493 + Right: the decoder — masked self-attention \eqref{eq:att-masked},
494 + cross-attention receiving keys and values $\mat{K}, \mat{V}$ from the
495 + encoder output, feed-forward, then a linear layer and softmax producing
496 + the next-token distribution.}
497 + \label{fig:att-transformer}
498 +\end{figure}
499 +
500 +% ----------------------------------------------------------------------------
501 +\section{Computational Complexity}\label{sec:att-complexity}
502 +% ----------------------------------------------------------------------------
503 +
504 +For sequence length $n$ and model dimension $d$, the score matrix
505 +$\mat{Q}\mat{K}\transp$ of \eqref{eq:att-sdpa} costs $O(n^2 d)$ time and
506 +$O(n^2)$ memory per layer: self-attention is \emph{quadratic in sequence
507 +length}. The feed-forward network \eqref{eq:att-ffn}, by contrast, costs
508 +$O(n d^2)$ — linear in $n$ but quadratic in width. Which term dominates
509 +depends on the regime: for $n < d$ (short sequences, wide models) the FFN
510 +dominates; for long contexts the $n^2$ term takes over and becomes the
511 +principal obstacle, motivating an entire literature of sparse,
512 +low-rank and IO-aware attention variants. Table~\ref{tab:att-complexity}
513 +compares the layer types on the three axes emphasized in
514 +\cite{vaswani2017}: total computation, sequential operations (the obstacle
515 +to parallelism), and maximum path length between two positions (the
516 +obstacle to learning long-range dependencies).
517 +
518 +\begin{table}[htbp]
519 + \centering
520 + \caption{Per-layer complexity for sequence length $n$, representation
521 + dimension $d$ and convolution kernel size $k$ \cite{vaswani2017}.}
522 + \label{tab:att-complexity}
523 + \begin{tabular}{lccc}
524 + \toprule
525 + Layer type & Complexity per layer & Sequential ops & Max path length \\
526 + \midrule
527 + Self-attention & $O(n^2 \, d)$ & $O(1)$ & $O(1)$ \\
528 + Recurrent & $O(n \, d^2)$ & $O(n)$ & $O(n)$ \\
529 + Convolutional & $O(k \, n \, d^2)$ & $O(1)$ & $O(\log_k n)$ \\
530 + \bottomrule
531 + \end{tabular}
532 +\end{table}
533 +
534 +The trade the Transformer makes is explicit in the first row: it pays a
535 +quadratic compute bill in exchange for constant-depth parallelism and
536 +constant-length interaction paths. For the sequence lengths of machine
537 +translation this trade was decisively favourable, and hardware trends —
538 +matrix units that reward dense, regular computation — have only widened
539 +the advantage since.
540 +
541 +% ----------------------------------------------------------------------------
542 +\section{Model Families: BERT, GPT, T5}\label{sec:att-families}
543 +% ----------------------------------------------------------------------------
544 +
545 +The encoder--decoder of Figure~\ref{fig:att-transformer} contains two
546 +self-sufficient halves, and the field promptly split it apart. Three
547 +canonical families resulted, distinguished by which half they keep and by
548 +their pre-training objective (Table~\ref{tab:att-families}).
549 +
550 +\textbf{BERT} keeps only the \emph{encoder}: attention is bidirectional,
551 +every position sees the whole sequence. It is pre-trained by masked
552 +language modelling — a fraction of input tokens is hidden and must be
553 +reconstructed from both sides of context — which makes it a powerful text
554 +\emph{understanding} machine (classification, retrieval, extraction) but
555 +not a generator.
556 +
557 +\textbf{GPT} keeps only the \emph{decoder}: every layer uses the causal
558 +mask \eqref{eq:att-mask}, and pre-training maximizes the autoregressive
559 +log-likelihood $\sum_t \log p_\theta(x_t \mid x_{<t})$. Generation is
560 +native — sampling one token at a time — and scaling this single recipe to
561 +ever larger models produced the modern lineage of large language models.
562 +
563 +\textbf{T5} keeps \emph{both} halves and casts every task — translation,
564 +summarization, classification, question answering — as text-to-text, with
565 +a span-corruption pre-training objective in which contiguous spans are
566 +masked and regenerated by the decoder.
567 +
568 +\begin{table}[htbp]
569 + \centering
570 + \caption{The three canonical Transformer families.}
571 + \label{tab:att-families}
572 + \begin{tabular}{llll}
573 + \toprule
574 + Family & Architecture & Pre-training objective & Typical use \\
575 + \midrule
576 + BERT & Encoder only & Masked language modelling & Understanding \\
577 + GPT & Decoder only & Next-token prediction & Generation \\
578 + T5 & Encoder--decoder & Span corruption & Text-to-text \\
579 + \bottomrule
580 + \end{tabular}
581 +\end{table}
582 +
583 +Whatever the family, the interior is the same handful of equations: scaled
584 +dot-product attention \eqref{eq:att-sdpa} with its variance-controlled
585 +logits \eqref{eq:att-variance}, multiple heads
586 +\eqref{eq:att-head}--\eqref{eq:att-multihead}, positional information
587 +\eqref{eq:att-pe-sin}--\eqref{eq:att-pe-cos}, and the residual--normalized
588 +block \eqref{eq:att-addnorm}--\eqref{eq:att-ffn}. Few architectures in the
589 +history of the field have combined such simplicity of definition with such
590 +range of consequence.
added chapters/07-modern-transformers.tex +678 −0
@@ -0,0 +1,678 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 7 : Modern Transformer Variants (chapters/07-modern-transformers.tex)
6 +% ============================================================================
7 +\chapter{Modern Transformer Variants: ViT, Mixture of Experts and State-Space Models}
8 +\label{chap:modern}
9 +
10 +The Transformer of the previous chapter is less a single architecture than a
11 +\emph{substrate}: a stack of residually connected token-mixing and
12 +channel-mixing operations that can be re-instantiated for new modalities,
13 +scaled to extreme parameter counts, or re-engineered to escape its quadratic
14 +cost. This chapter develops the three directions along which that substrate
15 +has evolved most consequentially. First, the \emph{Vision Transformer}
16 +transplants the architecture, essentially unchanged, from token sequences to
17 +images by an embedding trick \cite{dosovitskiy2021}. Second, the
18 +\emph{mixture of experts} decouples parameter count from per-token
19 +computation through sparse conditional routing \cite{shazeer2017}. Third,
20 +\emph{efficient attention} and \emph{selective state-space models} attack
21 +the $O(n^{2}d)$ bottleneck of self-attention itself, the latter replacing
22 +attention with a linear dynamical system whose inference cost is constant
23 +in sequence length \cite{gu2023}. We close with the empirical scaling laws
24 +that govern how all of these models improve with size, data and compute.
25 +
26 +% ============================================================================
27 +\section{The Vision Transformer}
28 +\label{sec:mt-vit}
29 +
30 +% ----------------------------------------------------------------------------
31 +\subsection{From pixels to tokens}
32 +
33 +Self-attention consumes a \emph{sequence} of vectors; an image
34 +$\mathsf{x} \in \R^{H \times W \times C}$ is not one. The Vision
35 +Transformer (ViT) of Dosovitskiy et al.\ \cite{dosovitskiy2021} resolves
36 +the mismatch with a deliberately simple embedding: partition the image into
37 +$N$ non-overlapping square patches of side $P$ (typically $P = 16$),
38 +flatten each patch into a vector, and treat the result as a sequence of
39 +tokens,
40 +\begin{equation}
41 + \vect{x}_p^{i} \in \R^{P^{2}C},
42 + \qquad
43 + i = 1, \dots, N,
44 + \qquad
45 + N = \frac{HW}{P^{2}} .
46 + \label{eq:mt-patches}
47 +\end{equation}
48 +Each flattened patch is mapped to the model dimension $D$ by a single
49 +learned linear projection $\mat{E} \in \R^{(P^{2}C) \times D}$ — the visual
50 +analogue of a word-embedding table. A learnable classification token
51 +$\vect{x}_{\mathrm{class}} \in \R^{D}$ is prepended, and learned positional
52 +embeddings $\mat{E}_{\mathrm{pos}} \in \R^{(N+1) \times D}$ are added to
53 +restore the spatial arrangement that flattening destroyed:
54 +\begin{equation}
55 + \mat{Z}_0
56 + =
57 + \bigl[\,\vect{x}_{\mathrm{class}};\;
58 + \vect{x}_p^{1}\mat{E};\;
59 + \vect{x}_p^{2}\mat{E};\;
60 + \dots;\;
61 + \vect{x}_p^{N}\mat{E}\,\bigr]
62 + + \mat{E}_{\mathrm{pos}} .
63 + \label{eq:mt-embed}
64 +\end{equation}
65 +
66 +The sequence $\mat{Z}_0$ then passes through $L$ standard pre-norm
67 +Transformer encoder blocks — multi-head self-attention (MSA) and a
68 +position-wise MLP, each wrapped in layer normalization and a residual
69 +connection \cite{vaswani2017}:
70 +\begin{align}
71 + \mat{Z}'_{\ell} &= \operatorname{MSA}\bigl(\operatorname{LN}(\mat{Z}_{\ell-1})\bigr)
72 + + \mat{Z}_{\ell-1},
73 + \label{eq:mt-msa}\\
74 + \mat{Z}_{\ell} &= \operatorname{MLP}\bigl(\operatorname{LN}(\mat{Z}'_{\ell})\bigr)
75 + + \mat{Z}'_{\ell},
76 + \qquad \ell = 1, \dots, L .
77 + \label{eq:mt-mlp}
78 +\end{align}
79 +Classification reads out only the final state of the class token,
80 +$\vect{z}_{L}^{0}$:
81 +\begin{equation}
82 + \hat{\vect{y}}
83 + =
84 + \softmax\!\bigl(\operatorname{LN}(\vect{z}_{L}^{0})\,\mat{W}_{\mathrm{head}}\bigr),
85 + \qquad
86 + \mat{W}_{\mathrm{head}} \in \R^{D \times K},
87 + \label{eq:mt-head}
88 +\end{equation}
89 +for $K$ classes. Figure~\ref{fig:mt-vit} traces the full pipeline, and
90 +Algorithm~\ref{alg:mt-vit} states the forward pass together with one
91 +supervised training step.
92 +
93 +\begin{figure}[t]
94 + \centering
95 + \begin{tikzpicture}[font=\small]
96 + % ---- input image as 2x2 patch grid ----
97 + \fill[cinput!15] (0,0) rectangle (0.8,0.8);
98 + \fill[cinput!30] (0.8,0) rectangle (1.6,0.8);
99 + \fill[cinput!45] (0,0.8) rectangle (0.8,1.6);
100 + \fill[cinput!60] (0.8,0.8) rectangle (1.6,1.6);
101 + \draw[black!70] (0,0) rectangle (1.6,1.6);
102 + \draw[black!70] (0.8,0) -- (0.8,1.6);
103 + \draw[black!70] (0,0.8) -- (1.6,0.8);
104 + \node[etiquette, anchor=north] at (0.8,-0.15) {input image, $N$ patches};
105 +
106 + % ---- flatten arrow ----
107 + \draw[fleche] (1.8,0.8) -- (2.9,0.8)
108 + node[midway, above, etiquette, align=center] {flatten +\\ project $\mat{E}$};
109 +
110 + % ---- token row ----
111 + \node[mem, minimum width=8mm, minimum height=8mm] (cls) at (3.65,0.8) {$\vect{x}_{\mathrm{cls}}$};
112 + \node[blochidden, minimum width=8mm, minimum height=8mm] (t1) at (4.72,0.8) {$\vect{x}_p^{1}\mat{E}$};
113 + \node[blochidden, minimum width=8mm, minimum height=8mm] (t2) at (5.79,0.8) {$\vect{x}_p^{2}\mat{E}$};
114 + \node[blochidden, minimum width=8mm, minimum height=8mm] (t3) at (6.86,0.8) {$\vect{x}_p^{3}\mat{E}$};
115 + \node[blochidden, minimum width=8mm, minimum height=8mm] (t4) at (7.93,0.8) {$\vect{x}_p^{4}\mat{E}$};
116 + \node[etiquette, anchor=north] at (5.79,0.25)
117 + {$+\;\mat{E}_{\mathrm{pos}}$ (learned positional embeddings)};
118 + \node[etiquette, anchor=south] at (3.65,1.35) {\texttt{[CLS]}};
119 +
120 + % ---- encoder ----
121 + \draw[fleche] (8.5,0.8) -- (9.2,0.8);
122 + \node[blochidden, minimum width=1.9cm, minimum height=1.7cm, align=center]
123 + (enc) at (10.35,0.8) {Transformer\\ encoder $\times L$};
124 +
125 + % ---- head ----
126 + \draw[fleche] (enc.east) -- ++(0.65,0)
127 + node[midway, above, etiquette] {$\vect{z}_L^0$};
128 + \node[blocoutput, minimum width=1.3cm, minimum height=8mm]
129 + (head) at (12.7,0.8) {MLP head};
130 + \draw[fleche] (head.east) -- ++(0.6,0) node[right] {$\hat{\vect{y}}$};
131 + \end{tikzpicture}
132 + \caption{The Vision Transformer \cite{dosovitskiy2021}. The image is cut
133 + into $N$ non-overlapping patches (here $N = 4$ for legibility), each
134 + flattened and linearly projected to $\R^{D}$ as in
135 + \eqref{eq:mt-embed}. A learnable \texttt{[CLS]} token (violet) is
136 + prepended, learned positional embeddings are added, and the resulting
137 + sequence flows through $L$ standard encoder blocks,
138 + eqs.~\eqref{eq:mt-msa}--\eqref{eq:mt-mlp}; the classification
139 + head \eqref{eq:mt-head} reads only the final \texttt{[CLS]} state.}
140 + \label{fig:mt-vit}
141 +\end{figure}
142 +
143 +\begin{algorithm}[t]
144 +\caption{Vision Transformer: forward pass and one training step}
145 +\label{alg:mt-vit}
146 +\begin{algorithmic}[1]
147 +\Require image $\mathsf{x}$, label $y$, patch size $P$, depth $L$,
148 + parameters $\theta = \{\mat{E}, \mat{E}_{\mathrm{pos}},
149 + \vect{x}_{\mathrm{class}}, \text{encoder blocks},
150 + \mat{W}_{\mathrm{head}}\}$
151 +\Function{ViTForward}{$\mathsf{x}$}
152 + \State split $\mathsf{x}$ into $N = HW/P^{2}$ patches; flatten each to
153 + $\vect{x}_p^{i} \in \R^{P^{2}C}$
154 + \State $\mat{Z}_0 \gets [\vect{x}_{\mathrm{class}};\,
155 + \vect{x}_p^{1}\mat{E};\, \dots;\, \vect{x}_p^{N}\mat{E}]
156 + + \mat{E}_{\mathrm{pos}}$
157 + \For{$\ell = 1, \dots, L$}
158 + \State $\mat{Z}'_{\ell} \gets
159 + \operatorname{MSA}(\operatorname{LN}(\mat{Z}_{\ell-1}))
160 + + \mat{Z}_{\ell-1}$
161 + \State $\mat{Z}_{\ell} \gets
162 + \operatorname{MLP}(\operatorname{LN}(\mat{Z}'_{\ell}))
163 + + \mat{Z}'_{\ell}$
164 + \EndFor
165 + \State \Return $\hat{\vect{y}} =
166 + \softmax(\operatorname{LN}(\vect{z}_{L}^{0})\,\mat{W}_{\mathrm{head}})$
167 +\EndFunction
168 +\State $\hat{\vect{y}} \gets \Call{ViTForward}{\mathsf{x}}$
169 +\State $\Loss \gets -\log \hat{y}_{y}$
170 + \Comment{cross-entropy on the true class}
171 +\State compute $\nabla_{\theta} \Loss$ by backpropagation
172 +\State update $\theta$ with AdamW (weight decay, warmup, cosine decay)
173 +\end{algorithmic}
174 +\end{algorithm}
175 +
176 +% ----------------------------------------------------------------------------
177 +\subsection{What is lost and what is gained}
178 +
179 +\begin{remark}[Inductive bias versus data]
180 +\label{rem:mt-inductive}
181 +A convolutional layer hard-wires locality and translation equivariance;
182 +ViT's attention layers assume neither — any patch may interact with any
183 +other from the first layer onward. The architecture therefore
184 +\emph{underperforms} CNNs of similar size when trained on modest datasets,
185 +but \emph{overtakes} them once pre-training data reaches tens or hundreds
186 +of millions of images \cite{dosovitskiy2021}: given enough data, the model
187 +learns locality where it is useful instead of being confined to it, and the
188 +global receptive field is immediate — no stacking of layers is needed to
189 +relate distant patches.
190 +\end{remark}
191 +
192 +Two costs follow from the embedding. The attention cost is quadratic in
193 +the number of patches, so halving $P$ quadruples $N$ and multiplies the
194 +attention cost by sixteen; hierarchical variants confine attention to
195 +local windows to recover linear complexity in image size. And discarding
196 +all tokens but the class token in \eqref{eq:mt-head} is a design choice,
197 +not a necessity: dense prediction tasks (detection, segmentation) instead
198 +read out the full sequence $\mat{Z}_L$.
199 +
200 +% ============================================================================
201 +\section{Mixture of Experts: Scaling by Sparsity}
202 +\label{sec:mt-moe}
203 +
204 +% ----------------------------------------------------------------------------
205 +\subsection{Conditional computation}
206 +
207 +In a dense Transformer every parameter participates in every token's
208 +forward pass: doubling the parameters doubles the per-token compute. The
209 +sparsely-gated mixture of experts (MoE) of Shazeer et al.\
210 +\cite{shazeer2017} severs that link.
211 +
212 +\begin{definition}[Sparse mixture of experts]
213 +\label{def:mt-moe}
214 +Let $E_1, \dots, E_{E}$ be $E$ \emph{expert} networks (in Transformers,
215 +independent FFN blocks) with identical input and output dimensions, and let
216 +$G : \R^{d} \to \R^{E}$ be a \emph{router} producing a sparse weight vector
217 +with at most $k \ll E$ non-zero entries. The layer computes
218 +$\vect{y} = \sum_{i=1}^{E} G(\vect{x})_i \, E_i(\vect{x})$, evaluating only
219 +the experts for which $G(\vect{x})_i \neq 0$. Parameter count grows with
220 +$E$; per-token compute grows only with $k$.
221 +\end{definition}
222 +
223 +The simplest router is a linear map followed by a softmax,
224 +\begin{equation}
225 + G_{\mathrm{dense}}(\vect{x}) = \softmax\bigl(\mat{W}_g\transp \vect{x}\bigr),
226 + \qquad \mat{W}_g \in \R^{d \times E},
227 + \label{eq:mt-gate}
228 +\end{equation}
229 +but \eqref{eq:mt-gate} is dense — every expert receives every token.
230 +Sparsity is obtained by keeping only the $k$ largest logits, after adding
231 +tunable Gaussian noise that encourages exploration and load dispersion
232 +during training:
233 +\begin{equation}
234 + H(\vect{x})_i
235 + =
236 + \bigl(\mat{W}_g\transp \vect{x}\bigr)_i
237 + + \varepsilon_i \cdot \operatorname{softplus}\!\bigl(
238 + (\mat{W}_{\mathrm{noise}}\transp \vect{x})_i\bigr),
239 + \qquad \varepsilon_i \sim \mathcal{N}(0, 1),
240 + \label{eq:mt-noisy}
241 +\end{equation}
242 +\begin{equation}
243 + \operatorname{TopK}(\vect{h}, k)_i
244 + =
245 + \begin{cases}
246 + h_i & \text{if } h_i \text{ is among the } k \text{ largest entries of } \vect{h},\\
247 + -\infty & \text{otherwise},
248 + \end{cases}
249 + \label{eq:mt-topk}
250 +\end{equation}
251 +so that the router output and the layer output are
252 +\begin{equation}
253 + G(\vect{x}) = \softmax\bigl(\operatorname{TopK}(H(\vect{x}), k)\bigr),
254 + \qquad
255 + \vect{y} = \sum_{i \,\in\, \mathcal{S}(\vect{x})} G(\vect{x})_i \, E_i(\vect{x}),
256 + \label{eq:mt-combine}
257 +\end{equation}
258 +where $\mathcal{S}(\vect{x})$ is the selected index set. The $-\infty$
259 +entries vanish under the softmax, so gradients flow only to the selected
260 +experts and to the router itself. Figure~\ref{fig:mt-moe} shows one token's
261 +route through the layer.
262 +
263 +\begin{figure}[t]
264 + \centering
265 + \begin{tikzpicture}[font=\small]
266 + \node[blocinput, minimum width=1.4cm] (tok) at (0,0) {token $\vect{x}$};
267 + \node[gate, minimum width=1.6cm, minimum height=1.0cm, align=center]
268 + (router) at (2.7,0) {router\\ $G(\vect{x})$};
269 + \draw[fleche] (tok) -- (router);
270 +
271 + % experts
272 + \node[blochidden, minimum width=1.7cm] (e1) at (6.3, 2.25) {$E_1$ (FFN)};
273 + \node[blochidden, minimum width=1.7cm] (e2) at (6.3, 0.75) {$E_2$ (FFN)};
274 + \node[blochidden, minimum width=1.7cm] (e3) at (6.3,-0.75) {$E_3$ (FFN)};
275 + \node[blochidden, minimum width=1.7cm] (e4) at (6.3,-2.25) {$E_4$ (FFN)};
276 +
277 + % dispatch: top-2 = experts 2 and 3 solid, others dashed faded
278 + \draw[flechep, black!30] (router.east) -- (e1.west);
279 + \draw[fleche, cgate!80!black] (router.east) -- (e2.west)
280 + node[midway, above, sloped, etiquette] {$g_2$};
281 + \draw[fleche, cgate!80!black] (router.east) -- (e3.west)
282 + node[midway, below, sloped, etiquette] {$g_3$};
283 + \draw[flechep, black!30] (router.east) -- (e4.west);
284 +
285 + % combine
286 + \node[op] (sum) at (9.6,0) {$+$};
287 + \draw[flechep, black!30] (e1.east) -- (sum);
288 + \draw[fleche] (e2.east) -- (sum)
289 + node[pos=0.32, above, sloped, etiquette] {$g_2 E_2(\vect{x})$};
290 + \draw[fleche] (e3.east) -- (sum)
291 + node[pos=0.32, below, sloped, etiquette] {$g_3 E_3(\vect{x})$};
292 + \draw[flechep, black!30] (e4.east) -- (sum);
293 + \draw[fleche] (sum.east) -- ++(1.0,0) node[right] {$\vect{y}$};
294 + \end{tikzpicture}
295 + \caption{A sparsely-gated mixture-of-experts layer with $E = 4$ experts
296 + and top-$k$ routing, $k = 2$ \cite{shazeer2017}. For this token the
297 + router \eqref{eq:mt-combine} selects experts $E_2$ and $E_3$ (solid
298 + green arrows); the non-selected experts (dashed grey) receive neither
299 + the token nor any gradient. The output is the gate-weighted sum of the
300 + two active experts.}
301 + \label{fig:mt-moe}
302 +\end{figure}
303 +
304 +% ----------------------------------------------------------------------------
305 +\subsection{Load balancing and capacity}
306 +
307 +Left to itself, the router collapses: a few experts win early, receive
308 +more gradient, and win forever. Training therefore adds an auxiliary loss
309 +that pushes the dispatch distribution toward uniformity. With $f_i$ the
310 +fraction of tokens in a batch whose first choice is expert $i$, and $P_i$
311 +the mean router probability assigned to expert $i$ over the batch,
312 +\begin{equation}
313 + \Loss_{\mathrm{aux}}
314 + =
315 + \alpha \, E \sum_{i=1}^{E} f_i \, P_i ,
316 + \label{eq:mt-aux}
317 +\end{equation}
318 +which is minimized when both distributions are uniform
319 +($f_i = P_i = 1/E$ gives $\Loss_{\mathrm{aux}} = \alpha$); the
320 +coefficient $\alpha \approx 10^{-2}$ trades balance against task loss.
321 +The product form makes \eqref{eq:mt-aux} differentiable through $P_i$
322 +even though the counts $f_i$ are not.
323 +
324 +\begin{remark}[Capacity factor and overflow]
325 +\label{rem:mt-capacity}
326 +On parallel hardware every expert is allocated a fixed buffer of
327 +$\lceil C \cdot kT/E \rceil$ token slots per batch of $T$ tokens, where
328 +$C \geq 1$ is the \emph{capacity factor}. Tokens routed to a full expert
329 +\emph{overflow}: they skip the expert and pass through the residual
330 +connection unchanged. Algorithm~\ref{alg:mt-moe} makes this explicit.
331 +Top-1 routing (the Switch simplification) and top-2 routing are the
332 +dominant regimes; with $E = 8$ and $k = 2$, a model can hold $47$ billion
333 +parameters while activating only ${\sim}13$ billion per token.
334 +\end{remark}
335 +
336 +\begin{algorithm}[t]
337 +\caption{MoE layer: noisy top-$k$ routing with capacity factor}
338 +\label{alg:mt-moe}
339 +\begin{algorithmic}[1]
340 +\Require batch of $T$ token vectors $\{\vect{x}_t\}$, experts
341 + $E_1, \dots, E_E$, router weights $\mat{W}_g,
342 + \mat{W}_{\mathrm{noise}}$, top-$k$, capacity factor $C$
343 +\State $\mathrm{cap} \gets \lceil C \cdot kT/E \rceil$;\quad
344 + $\mathrm{load}_i \gets 0$ for $i = 1, \dots, E$
345 +\For{$t = 1, \dots, T$}
346 + \State $\vect{h}_t \gets$ noisy logits by \eqref{eq:mt-noisy}
347 + \State $\mathcal{S}_t \gets$ indices of the $k$ largest entries of
348 + $\vect{h}_t$
349 + \State $\vect{g}_t \gets \softmax\bigl(\operatorname{TopK}(\vect{h}_t,
350 + k)\bigr)$ \Comment{eq.~\eqref{eq:mt-topk}}
351 + \State $\vect{y}_t \gets \vect{0}$
352 + \For{$i \in \mathcal{S}_t$}
353 + \If{$\mathrm{load}_i < \mathrm{cap}$}
354 + \State $\vect{y}_t \gets \vect{y}_t + g_{t,i} \, E_i(\vect{x}_t)$;
355 + \quad $\mathrm{load}_i \gets \mathrm{load}_i + 1$
356 + \EndIf \Comment{overflowed tokens rely on the residual path}
357 + \EndFor
358 +\EndFor
359 +\State add $\Loss_{\mathrm{aux}}$ of \eqref{eq:mt-aux} to the task loss
360 +\end{algorithmic}
361 +\end{algorithm}
362 +
363 +% ============================================================================
364 +\section{Efficient Attention}
365 +\label{sec:mt-efficient}
366 +
367 +Self-attention over $n$ tokens of width $d$ costs
368 +\begin{equation}
369 + \underbrace{O(n^{2} d)}_{\text{time}}
370 + \qquad \text{and} \qquad
371 + \underbrace{O(n^{2})}_{\text{memory for } \mat{Q}\mat{K}\transp},
372 + \label{eq:mt-complexity}
373 +\end{equation}
374 +which at $n = 10^{5}$ tokens makes the attention matrix alone prohibitive.
375 +Three families of remedies exist: \emph{restrict} which pairs may
376 +interact, \emph{approximate} the softmax kernel, or \emph{reorganize} the
377 +exact computation around the memory hierarchy.
378 +
379 +\paragraph{Sliding-window (local) attention.} Each token attends only to
380 +the $W$ preceding tokens. The additive mask
381 +\begin{equation}
382 + M_{ij} =
383 + \begin{cases}
384 + 0 & \text{if } 0 \leq i - j < W,\\
385 + -\infty & \text{otherwise},
386 + \end{cases}
387 + \label{eq:mt-window}
388 +\end{equation}
389 +reduces the cost to $O(nWd)$, and stacking $L$ such layers still yields an
390 +effective receptive field of $L \cdot W$ positions — exactly as stacked
391 +small convolutions enlarge a CNN's receptive field.
392 +
393 +\paragraph{Linear (kernelized) attention.} If the exponential kernel of
394 +the softmax is replaced — or approximated — by an inner product of feature
395 +maps, $\exp(\vect{q}\transp\vect{k}) \approx
396 +\phi(\vect{q})\transp\phi(\vect{k})$ with
397 +$\phi : \R^{d} \to \R^{r}$, the attention output factorizes, and the
398 +multiplication order can be changed:
399 +\begin{equation}
400 + \operatorname{Attn}(\mat{Q}, \mat{K}, \mat{V})
401 + \approx
402 + \phi(\mat{Q}) \,
403 + \bigl(\phi(\mat{K})\transp \mat{V}\bigr),
404 + \label{eq:mt-linear}
405 +\end{equation}
406 +where the bracketed product is $r \times d$ — independent of $n$ — so the
407 +total cost is $O(nrd)$, linear in sequence length. The price is an
408 +approximation, made unbiased by suitable random-feature constructions for
409 +$\phi$.
410 +
411 +\paragraph{Exact IO-aware attention.} A complementary line accelerates
412 +\emph{exact} attention by observing that the bottleneck on modern
413 +accelerators is memory traffic, not arithmetic: tiling
414 +$\mat{Q}, \mat{K}, \mat{V}$ into on-chip blocks, computing the softmax
415 +incrementally with a running maximum and normalizer, and never
416 +materializing the $n \times n$ matrix reduces memory from $O(n^{2})$ to
417 +$O(n)$ with the output unchanged. Table~\ref{tab:mt-complexity} compares
418 +the regimes.
419 +
420 +\begin{table}[t]
421 + \centering
422 + \caption{Cost of one attention (or mixing) layer over $n$ tokens of
423 + width $d$; $W$ is the window size of \eqref{eq:mt-window}, $r$ the
424 + feature dimension of the kernel map in \eqref{eq:mt-linear}, and $N$
425 + the state size of the SSM in \eqref{eq:mt-selective}.}
426 + \label{tab:mt-complexity}
427 + \begin{tabular}{lccc}
428 + \toprule
429 + Mechanism & Time & Memory & Exact?\\
430 + \midrule
431 + Full softmax attention \cite{vaswani2017} & $O(n^{2}d)$ & $O(n^{2})$ & yes\\
432 + Sliding window, eq.~\eqref{eq:mt-window} & $O(nWd)$ & $O(nW)$ & restricted\\
433 + Linear / kernelized, eq.~\eqref{eq:mt-linear} & $O(nrd)$ & $O(nr)$ & approximate\\
434 + Tiled exact (IO-aware) & $O(n^{2}d)$ & $O(n)$ & yes\\
435 + Selective SSM, eq.~\eqref{eq:mt-selective} & $O(nNd)$ & $O(Nd)$ & different model\\
436 + \bottomrule
437 + \end{tabular}
438 +\end{table}
439 +
440 +% ============================================================================
441 +\section{State-Space Models and Mamba}
442 +\label{sec:mt-ssm}
443 +
444 +% ----------------------------------------------------------------------------
445 +\subsection{The linear dynamical view of sequence modeling}
446 +
447 +\begin{definition}[State-space model]
448 +\label{def:mt-ssm}
449 +A (continuous-time, linear) state-space model maps an input signal
450 +$u(t) \in \R$ to an output $y(t) \in \R$ through a hidden state
451 +$\vect{h}(t) \in \R^{N}$ obeying
452 +\begin{equation}
453 + \vect{h}'(t) = \mat{A}\,\vect{h}(t) + \vect{b}\,u(t),
454 + \qquad
455 + y(t) = \vect{c}\transp \vect{h}(t),
456 + \label{eq:mt-ssm-cont}
457 +\end{equation}
458 +with parameters $\mat{A} \in \R^{N \times N}$ and
459 +$\vect{b}, \vect{c} \in \R^{N}$. In deep SSMs each channel of a
460 +$d$-dimensional sequence carries its own scalar system, and the layer is
461 +wrapped in the usual residual and normalization scaffolding.
462 +\end{definition}
463 +
464 +To operate on sampled sequences $u_1, u_2, \dots$, the continuous system
465 +\eqref{eq:mt-ssm-cont} is discretized with step size $\Delta$ by the
466 +zero-order hold, exact when $u(t)$ is piecewise constant between samples:
467 +\begin{equation}
468 + \bar{\mat{A}} = \exp(\Delta \mat{A}),
469 + \qquad
470 + \bar{\vect{b}}
471 + = (\Delta \mat{A})^{-1}\bigl(\exp(\Delta \mat{A}) - \mat{I}\bigr)\,
472 + \Delta \vect{b},
473 + \label{eq:mt-zoh}
474 +\end{equation}
475 +yielding the linear recurrence
476 +\begin{equation}
477 + \vect{h}_t = \bar{\mat{A}}\,\vect{h}_{t-1} + \bar{\vect{b}}\,u_t,
478 + \qquad
479 + y_t = \vect{c}\transp \vect{h}_t .
480 + \label{eq:mt-ssm-disc}
481 +\end{equation}
482 +
483 +\begin{property}[An LTI recurrence unrolls into a convolution]
484 +\label{prop:mt-conv}
485 +If $\bar{\mat{A}}, \bar{\vect{b}}, \vect{c}$ do not depend on $t$ (a
486 +linear \emph{time-invariant} system), unrolling \eqref{eq:mt-ssm-disc}
487 +from $\vect{h}_0 = \vect{0}$ gives
488 +$y_t = \sum_{j=0}^{t-1} \vect{c}\transp \bar{\mat{A}}^{\,j}
489 +\bar{\vect{b}}\, u_{t-j}$, i.e.\ a causal convolution
490 +\begin{equation}
491 + \vect{y} = \vect{u} * \bar{\vect{K}},
492 + \qquad
493 + \bar{\vect{K}}
494 + = \bigl(\vect{c}\transp\bar{\vect{b}},\;
495 + \vect{c}\transp\bar{\mat{A}}\bar{\vect{b}},\;
496 + \vect{c}\transp\bar{\mat{A}}^{2}\bar{\vect{b}},\; \dots\bigr),
497 + \label{eq:mt-kernel}
498 +\end{equation}
499 +computable for a length-$n$ sequence in $O(n \log n)$ by the FFT. The
500 +same model therefore trains \emph{in parallel} as a convolution and runs
501 +inference \emph{recurrently} with $O(1)$ memory per step — a duality that
502 +attention does not possess.
503 +\end{property}
504 +
505 +% ----------------------------------------------------------------------------
506 +\subsection{Selectivity: making the dynamics depend on the input}
507 +
508 +An LTI system applies the same dynamics to every token: it cannot decide,
509 +based on \emph{content}, what to store and what to forget. The selective
510 +SSM of Mamba \cite{gu2023} breaks time invariance by making the step
511 +size and the input/output projections functions of the current input
512 +$\vect{u}_t \in \R^{d}$:
513 +\begin{equation}
514 + \Delta_t = \operatorname{softplus}\bigl(\mat{W}_{\Delta}\vect{u}_t\bigr),
515 + \qquad
516 + \vect{b}_t = \mat{W}_B \vect{u}_t,
517 + \qquad
518 + \vect{c}_t = \mat{W}_C \vect{u}_t,
519 + \qquad
520 + \vect{h}_t = \bar{\mat{A}}_t\,\vect{h}_{t-1}
521 + + \bar{\vect{b}}_t\,u_t,
522 + \label{eq:mt-selective}
523 +\end{equation}
524 +with $\bar{\mat{A}}_t, \bar{\vect{b}}_t$ obtained from \eqref{eq:mt-zoh}
525 +using $\Delta_t$ and $\vect{b}_t$. A large $\Delta_t$ resets the state
526 +toward the current input (\emph{attend}); $\Delta_t \to 0$ leaves the
527 +state untouched (\emph{ignore}) — a content-dependent gate reminiscent of
528 +the LSTM's, embedded in a principled continuous-time model. Input
529 +dependence destroys the convolutional form \eqref{eq:mt-kernel}, so
530 +training uses a \emph{hardware-aware parallel scan}: the recurrence
531 +\eqref{eq:mt-selective} is associative in the pairs
532 +$(\bar{\mat{A}}_t, \bar{\vect{b}}_t u_t)$, so $n$ steps reduce in
533 +$O(\log n)$ parallel depth with states kept in on-chip memory.
534 +Algorithm~\ref{alg:mt-scan} gives the sequential form, which is also the
535 +constant-memory inference procedure; Figure~\ref{fig:mt-ssm} contrasts
536 +the two mixing mechanisms.
537 +
538 +\begin{algorithm}[t]
539 +\caption{Selective scan (sequential form; inference-time recurrence)}
540 +\label{alg:mt-scan}
541 +\begin{algorithmic}[1]
542 +\Require sequence $\vect{u}_1, \dots, \vect{u}_n$, parameters $\mat{A},
543 + \mat{W}_{\Delta}, \mat{W}_B, \mat{W}_C$
544 +\State $\vect{h}_0 \gets \vect{0}$
545 +\For{$t = 1, \dots, n$}
546 + \State $\Delta_t \gets \operatorname{softplus}(\mat{W}_{\Delta}\vect{u}_t)$;
547 + \quad $\vect{b}_t \gets \mat{W}_B \vect{u}_t$;
548 + \quad $\vect{c}_t \gets \mat{W}_C \vect{u}_t$
549 + \State $\bar{\mat{A}}_t \gets \exp(\Delta_t \mat{A})$;\quad
550 + $\bar{\vect{b}}_t \gets (\Delta_t\mat{A})^{-1}
551 + (\exp(\Delta_t \mat{A}) - \mat{I})\,\Delta_t \vect{b}_t$
552 + \Comment{ZOH, eq.~\eqref{eq:mt-zoh}}
553 + \State $\vect{h}_t \gets \bar{\mat{A}}_t \vect{h}_{t-1}
554 + + \bar{\vect{b}}_t u_t$;\qquad
555 + $y_t \gets \vect{c}_t\transp \vect{h}_t$
556 +\EndFor
557 +\State \Return $y_1, \dots, y_n$
558 + \Comment{training uses an associative parallel scan instead}
559 +\end{algorithmic}
560 +\end{algorithm}
561 +
562 +\begin{figure}[t]
563 + \centering
564 + \begin{tikzpicture}[font=\small]
565 + % ================= panel (a): SSM recurrence =================
566 + \begin{scope}
567 + \node[mem, minimum width=1.0cm] (h1) at (0,0) {$\vect{h}_1$};
568 + \node[mem, minimum width=1.0cm] (h2) at (1.8,0) {$\vect{h}_2$};
569 + \node[mem, minimum width=1.0cm] (h3) at (3.6,0) {$\vect{h}_3$};
570 + \node[mem, minimum width=1.0cm] (h4) at (5.4,0) {$\vect{h}_4$};
571 + \draw[fleche, cmem!80!black] (h1) -- (h2)
572 + node[midway, above, etiquette] {$\bar{\mat{A}}_2$};
573 + \draw[fleche, cmem!80!black] (h2) -- (h3)
574 + node[midway, above, etiquette] {$\bar{\mat{A}}_3$};
575 + \draw[fleche, cmem!80!black] (h3) -- (h4)
576 + node[midway, above, etiquette] {$\bar{\mat{A}}_4$};
577 + \foreach \i in {1,...,4} {
578 + \node[ninput] (u\i) at ({(\i-1)*1.8}, -1.6) {$u_{\i}$};
579 + \node[noutput] (y\i) at ({(\i-1)*1.6*1.125}, 1.6) {$y_{\i}$};
580 + \draw[fleche] (u\i) -- (h\i);
581 + \draw[fleche] (h\i) -- (y\i);
582 + }
583 + \node[etiquette, anchor=east] at (-0.15,-0.95) {$\bar{\vect{b}}_t$};
584 + \node[etiquette, anchor=east] at (-0.15,0.95) {$\vect{c}_t\transp$};
585 + \node[align=center, font=\small] at (2.7,-2.75)
586 + {(a) selective SSM: $O(n)$ time,\\ state of fixed size $N$};
587 + \end{scope}
588 + % ================= panel (b): attention all-pairs =================
589 + \begin{scope}[xshift=8.6cm]
590 + \foreach \i in {1,...,4}
591 + \node[nhidden] (t\i) at ({(\i-1)*1.8}, 0) {$\vect{x}_{\i}$};
592 + % causal all-pairs arcs above
593 + \draw[fleche, chidden!80!black] (t1) to[bend left=45] (t2);
594 + \draw[fleche, chidden!80!black] (t2) to[bend left=45] (t3);
595 + \draw[fleche, chidden!80!black] (t3) to[bend left=45] (t4);
596 + \draw[fleche, chidden!80!black] (t1) to[bend left=55] (t3);
597 + \draw[fleche, chidden!80!black] (t2) to[bend left=55] (t4);
598 + \draw[fleche, chidden!80!black] (t1) to[bend left=65] (t4);
599 + \node[align=center, font=\small] at (2.7,-2.75)
600 + {(b) causal self-attention:\\ $O(n^{2})$ pairwise interactions};
601 + \end{scope}
602 + \end{tikzpicture}
603 + \caption{Two mechanisms for mixing information along a sequence.
604 + (a)~The selective state-space recurrence \eqref{eq:mt-selective}:
605 + inputs $u_t$ enter a fixed-size state $\vect{h}_t$ (violet) that is
606 + carried forward by input-dependent transitions $\bar{\mat{A}}_t$; the
607 + cost is linear in length and the inference memory constant.
608 + (b)~Causal self-attention: every token interacts directly with every
609 + earlier token, an immediate global view at quadratic cost
610 + \cite{vaswani2017,gu2023}.}
611 + \label{fig:mt-ssm}
612 +\end{figure}
613 +
614 +\begin{remark}[Where each mechanism wins]
615 +\label{rem:mt-duality}
616 +Attention retrieves \emph{exactly}: any past token can be recalled
617 +verbatim, which underlies in-context learning and copying. An SSM
618 +compresses the past into a fixed-size state — retrieval is lossy, but the
619 +cost is $O(n)$ in time and $O(1)$ in inference memory
620 +(Table~\ref{tab:mt-complexity}). Selective SSMs match Transformers of
621 +roughly twice their size on language modeling \cite{gu2023}, and hybrid
622 +stacks interleaving the two layer types are increasingly common, spending
623 +quadratic attention only where exact retrieval pays for itself.
624 +\end{remark}
625 +
626 +% ============================================================================
627 +\section{Scaling Laws}
628 +\label{sec:mt-scaling}
629 +
630 +All the architectures of this chapter obey strikingly regular
631 +\emph{scaling laws}: over many orders of magnitude, the test cross-entropy
632 +of an autoregressive Transformer falls as a power law in the parameter
633 +count $N_{\mathrm{par}}$ and the dataset size $D$ (in tokens),
634 +\begin{equation}
635 + \Loss(N_{\mathrm{par}}) =
636 + \Bigl(\frac{N_c}{N_{\mathrm{par}}}\Bigr)^{\alpha_N},
637 + \qquad
638 + \Loss(D) = \Bigl(\frac{D_c}{D}\Bigr)^{\alpha_D},
639 + \qquad
640 + \alpha_N \approx 0.076,\;
641 + \alpha_D \approx 0.095,
642 + \label{eq:mt-kaplan}
643 +\end{equation}
644 +when the other factor is not binding. A refined joint parametrization
645 +separates an irreducible entropy of text $E_0$ from two reducible terms,
646 +\begin{equation}
647 + \Loss(N_{\mathrm{par}}, D)
648 + =
649 + E_0 + \frac{A}{N_{\mathrm{par}}^{\alpha}} + \frac{B}{D^{\beta}},
650 + \qquad
651 + \alpha \approx 0.34,\;
652 + \beta \approx 0.28 .
653 + \label{eq:mt-chinchilla}
654 +\end{equation}
655 +
656 +\begin{remark}[Compute-optimal training]
657 +\label{rem:mt-chinchilla}
658 +Training cost is approximately $C \approx 6 N_{\mathrm{par}} D$ FLOPs.
659 +Minimizing \eqref{eq:mt-chinchilla} subject to fixed $C$ gives
660 +$N_{\mathrm{par}}^{\mathrm{opt}} \propto C^{a}$ and
661 +$D^{\mathrm{opt}} \propto C^{b}$ with $a \approx b \approx 0.5$:
662 +parameters and tokens should be scaled \emph{in equal proportion},
663 +roughly twenty tokens per parameter — a sharp correction to earlier
664 +practice, which grew models far faster than their training sets.
665 +Production systems now deliberately train \emph{past} this optimum on
666 +smaller models, accepting extra training compute to reduce the inference
667 +cost that dominates a deployed model's lifetime. The MoE construction of
668 +Section~\ref{sec:mt-moe} bends these laws favourably by growing
669 +$N_{\mathrm{par}}$ without growing per-token compute, and selective SSMs
670 +(Section~\ref{sec:mt-ssm}) do so by removing the $O(n^{2})$ cost of
671 +context length itself.
672 +\end{remark}
673 +
674 +Together, the three levers of this chapter — new modalities through
675 +embeddings, more parameters through sparsity, longer contexts through
676 +sub-quadratic mixing — define the current design space of large-scale
677 +neural networks, all resting on the same residual substrate introduced in
678 +the previous chapter.
added chapters/08-gnn.tex +685 −0
@@ -0,0 +1,685 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 8 : Graph Neural Networks (chapters/08-gnn.tex)
6 +% ============================================================================
7 +\chapter{Graph Neural Networks}\label{chap:gnn}
8 +
9 +The architectures of the preceding chapters assume that data live on a
10 +regular lattice: images are grids, sequences are chains. A vast portion of
11 +real-world data, however, is naturally \emph{relational} — molecules,
12 +social networks, citation graphs, road networks, protein--protein
13 +interaction maps. Graph neural networks (GNNs) extend deep learning to
14 +this irregular domain. The central design problem is invariance: the
15 +prediction for a node must not depend on the arbitrary order in which its
16 +neighbours are listed. This chapter develops the message-passing framework
17 +that solves this problem, then derives the four canonical instantiations —
18 +the graph convolutional network (GCN)~\cite{kipf2017}, the inductive
19 +sample-and-aggregate network GraphSAGE~\cite{hamilton2017}, the graph
20 +attention network (GAT)~\cite{velickovic2018}, and the graph isomorphism
21 +network (GIN)~\cite{xu2019} — each with its propagation equations, its
22 +estimation algorithm and its graphical representation.
23 +
24 +% ----------------------------------------------------------------------------
25 +\section{Graphs, Notation and the Message-Passing Framework}
26 +\label{sec:gnn-mp}
27 +% ----------------------------------------------------------------------------
28 +
29 +Let $G = (V, E)$ be a graph with $n = |V|$ nodes. Each node $v \in V$
30 +carries a feature vector $\vect{x}_v \in \R^{d}$, collected row-wise in
31 +the matrix $\mat{X} \in \R^{n \times d}$; an edge $(u,v) \in E$ may carry
32 +a feature vector $\vect{e}_{uv}$. The connectivity is encoded by the
33 +adjacency matrix and the diagonal degree matrix,
34 +\begin{equation}
35 + \mat{A} \in \{0,1\}^{n \times n}, \quad
36 + A_{uv} = \begin{cases} 1 & \text{if } (u,v) \in E,\\
37 + 0 & \text{otherwise,} \end{cases}
38 + \qquad
39 + \mat{D} = \operatorname{diag}(d_1, \dots, d_n), \quad
40 + d_v = \sum_{u} A_{vu},
41 + \label{eq:gnn-adj}
42 +\end{equation}
43 +and the neighbourhood of $v$ is $\mathcal{N}(v) = \{u : (u,v) \in E\}$.
44 +Because a graph has no canonical node ordering, any layer that maps node
45 +states to node states must be \emph{permutation-equivariant}: relabelling
46 +the nodes must merely relabel the outputs. The message-passing framework
47 +achieves this by construction.
48 +
49 +\begin{definition}[Message-passing layer]\label{def:gnn-mpnn}
50 +A message-passing layer updates the state $\vect{h}_v^{(k-1)}$ of every
51 +node $v$ in three steps: a \emph{message} computed on each incoming edge
52 +and summed,
53 +\begin{equation}
54 + \vect{m}_v^{(k)}
55 + = \sum_{u \in \mathcal{N}(v)}
56 + M_k\bigl(\vect{h}_v^{(k-1)}, \vect{h}_u^{(k-1)}, \vect{e}_{uv}\bigr),
57 + \label{eq:gnn-message}
58 +\end{equation}
59 +an \emph{update} that combines the aggregated message with the node's own
60 +state,
61 +\begin{equation}
62 + \vect{h}_v^{(k)} = U_k\bigl(\vect{h}_v^{(k-1)}, \vect{m}_v^{(k)}\bigr),
63 + \label{eq:gnn-update}
64 +\end{equation}
65 +where $M_k$ and $U_k$ are learned functions (typically small MLPs or
66 +gated units), and, for graph-level tasks after $K$ layers, a
67 +permutation-\emph{invariant} \emph{readout}
68 +\begin{equation}
69 + \hat{\vect{y}}_G = R\bigl(\{\vect{h}_v^{(K)} : v \in V\}\bigr).
70 + \label{eq:gnn-readout}
71 +\end{equation}
72 +The initial state is $\vect{h}_v^{(0)} = \vect{x}_v$.
73 +\end{definition}
74 +
75 +Modern practice separates the neighbourhood \emph{aggregation} from the
76 +self-\emph{combination}, a form to which every architecture in this
77 +chapter reduces:
78 +\begin{equation}
79 + \vect{a}_v^{(k)}
80 + = \operatorname{AGG}^{(k)}
81 + \bigl(\{\!\!\{\vect{h}_u^{(k-1)} : u \in \mathcal{N}(v)\}\!\!\}\bigr),
82 + \qquad
83 + \vect{h}_v^{(k)}
84 + = \operatorname{COMB}^{(k)}
85 + \bigl(\vect{h}_v^{(k-1)}, \vect{a}_v^{(k)}\bigr),
86 + \label{eq:gnn-aggcomb}
87 +\end{equation}
88 +where $\{\!\!\{\cdot\}\!\!\}$ denotes a \emph{multiset} — neighbours may
89 +carry identical states, and their multiplicity matters. The aggregator
90 +must be invariant under permutations of its multiset argument; the
91 +standard choices are the sum, the mean, the element-wise maximum, and the
92 +attention-weighted sum. The choice among them is not cosmetic: it
93 +determines the discriminative power of the whole network, as
94 +Section~\ref{sec:gnn-gin} makes precise. After $k$ layers, the state
95 +$\vect{h}_v^{(k)}$ is a function of the subgraph within $k$ hops of $v$ —
96 +the graph analogue of the receptive field of a convolutional network.
97 +Figure~\ref{fig:gnn-mp} depicts one application of
98 +\eqref{eq:gnn-message}--\eqref{eq:gnn-update}, and
99 +Algorithm~\ref{alg:gnn-forward} summarizes the full forward pass.
100 +
101 +\begin{figure}[htbp]
102 + \centering
103 + \begin{tikzpicture}[scale=1.0]
104 + % ---- left panel: the graph ----
105 + \node[noutput] (v) at (0,0) {$v$};
106 + \node[ninput] (u1) at (-1.9,1.4) {$u_1$};
107 + \node[ninput] (u2) at (-2.3,-0.4){$u_2$};
108 + \node[ninput] (u3) at (-0.6,-1.9){$u_3$};
109 + \node[ninput] (u4) at (1.4,1.6) {$u_4$};
110 + \node[neuron] (w1) at (-3.6,1.9) {$w_1$};
111 + \node[neuron] (w2) at (-3.9,-1.2){$w_2$};
112 + \node[neuron] (w3) at (0.9,-2.6) {$w_3$};
113 + % plain graph edges (2-hop)
114 + \draw[black!45, semithick] (u1) -- (w1);
115 + \draw[black!45, semithick] (u2) -- (w1);
116 + \draw[black!45, semithick] (u2) -- (w2);
117 + \draw[black!45, semithick] (u3) -- (w3);
118 + % message arrows into v
119 + \draw[fleche, cinput!80!black] (u1) to[bend left=12]
120 + node[etiquette, above right=-1pt] {$\vect{m}_{u_1 \to v}$} (v);
121 + \draw[fleche, cinput!80!black] (u2) to[bend right=12] (v);
122 + \draw[fleche, cinput!80!black] (u3) to[bend right=12] (v);
123 + \draw[fleche, cinput!80!black] (u4) to[bend left=12] (v);
124 + % ---- right panel: aggregate / combine pipeline ----
125 + \node[etiquette, align=center] (ms) at (5.4,1.9)
126 + {$\{\!\!\{\vect{h}_{u}^{(k-1)} : u \in \mathcal{N}(v)\}\!\!\}$};
127 + \node[gate, minimum width=2.6cm] (agg) at (5.4,0.9)
128 + {$\operatorname{AGG}^{(k)}$};
129 + \node[blochidden, minimum width=2.6cm] (comb) at (5.4,-0.7)
130 + {$\operatorname{COMB}^{(k)}$};
131 + \node[etiquette] (hout) at (5.4,-1.9) {$\vect{h}_v^{(k)}$};
132 + \draw[fleche] (ms) -- (agg);
133 + \draw[fleche] (agg) -- node[etiquette, right]
134 + {$\vect{a}_v^{(k)}$} (comb);
135 + \draw[fleche] (comb) -- (hout);
136 + % self state feeding the combine step
137 + \node[etiquette] (hself) at (8.1,-0.7) {$\vect{h}_v^{(k-1)}$};
138 + \draw[fleche] (hself) -- (comb);
139 + % dashed link from the graph to the pipeline
140 + \draw[flechep, black!55] (v.east) to[bend left=10] (agg.west);
141 + \end{tikzpicture}
142 + \caption{One message-passing layer at node $v$
143 + (Definition~\ref{def:gnn-mpnn}). Left: the neighbours
144 + $u_1, \dots, u_4$ (blue) send messages along the edges into the
145 + target node $v$ (red); the nodes $w_i$ are two hops away and will
146 + only influence $v$ at the next layer. Right: the received multiset is
147 + reduced by a permutation-invariant aggregator, then combined with the
148 + node's previous state to produce $\vect{h}_v^{(k)}$, following
149 + \eqref{eq:gnn-aggcomb}.}
150 + \label{fig:gnn-mp}
151 +\end{figure}
152 +
153 +\begin{algorithm}[htbp]
154 + \caption{Generic message-passing forward pass (MPNN)}
155 + \label{alg:gnn-forward}
156 + \begin{algorithmic}[1]
157 + \Require graph $G=(V,E)$, features $\{\vect{x}_v\}$, depth $K$,
158 + layers $\{(M_k, U_k)\}_{k=1}^{K}$, readout $R$
159 + \For{$v \in V$}
160 + \State $\vect{h}_v^{(0)} \gets \vect{x}_v$
161 + \EndFor
162 + \For{$k = 1, \dots, K$}
163 + \For{$v \in V$}
164 + \State $\vect{m}_v^{(k)} \gets
165 + \sum_{u \in \mathcal{N}(v)}
166 + M_k\bigl(\vect{h}_v^{(k-1)}, \vect{h}_u^{(k-1)},
167 + \vect{e}_{uv}\bigr)$
168 + \Comment{messages, Eq.~\eqref{eq:gnn-message}}
169 + \EndFor
170 + \For{$v \in V$}
171 + \State $\vect{h}_v^{(k)} \gets
172 + U_k\bigl(\vect{h}_v^{(k-1)}, \vect{m}_v^{(k)}\bigr)$
173 + \Comment{update, Eq.~\eqref{eq:gnn-update}}
174 + \EndFor
175 + \EndFor
176 + \State \Return node states $\{\vect{h}_v^{(K)}\}$, or
177 + $\hat{\vect{y}}_G = R\bigl(\{\vect{h}_v^{(K)}\}\bigr)$ for
178 + graph-level tasks
179 + \end{algorithmic}
180 +\end{algorithm}
181 +
182 +\begin{remark}
183 +The two inner loops of Algorithm~\ref{alg:gnn-forward} are never executed
184 +node by node in practice: with sum or mean aggregation the whole layer
185 +collapses into a pair of sparse--dense matrix products (cf.\
186 +\eqref{eq:gnn-gcnlayer}), so a full propagation step costs
187 +$\mathcal{O}(|E| \cdot d)$ — linear in the number of edges.
188 +\end{remark}
189 +
190 +% ----------------------------------------------------------------------------
191 +\section{Graph Convolutional Networks}\label{sec:gnn-gcn}
192 +% ----------------------------------------------------------------------------
193 +
194 +The GCN of Kipf and Welling~\cite{kipf2017} descends from spectral graph
195 +theory. On a graph, the Fourier basis is provided by the eigenvectors of
196 +the normalized graph Laplacian
197 +$\mat{L} = \mat{I} - \mat{D}^{-1/2}\mat{A}\mat{D}^{-1/2}
198 += \mat{U}\mat{\Lambda}\mat{U}\transp$, and a spectral convolution with
199 +filter $g_\theta$ acts on a signal $\vect{x} \in \R^n$ as
200 +\begin{equation}
201 + g_\theta \star \vect{x}
202 + = \mat{U}\, g_\theta(\mat{\Lambda})\, \mat{U}\transp \vect{x}.
203 + \label{eq:gnn-spectral}
204 +\end{equation}
205 +Evaluating \eqref{eq:gnn-spectral} exactly requires the full
206 +eigendecomposition — $\mathcal{O}(n^3)$, prohibitive beyond small graphs.
207 +Truncating a Chebyshev expansion of $g_\theta$ at first order, and tying
208 +its two remaining coefficients, collapses the filter to a strictly local
209 +operation:
210 +\begin{equation}
211 + g_\theta \star \vect{x}
212 + \;\approx\;
213 + \theta \bigl(\mat{I} + \mat{D}^{-1/2}\mat{A}\mat{D}^{-1/2}\bigr)\,
214 + \vect{x}.
215 + \label{eq:gnn-cheby}
216 +\end{equation}
217 +The operator in \eqref{eq:gnn-cheby} has eigenvalues in $[0,2]$; stacking
218 +many such layers can therefore amplify or shrink signals. The
219 +\emph{renormalization trick} restores stability by adding self-loops
220 +\emph{before} normalizing:
221 +\begin{equation}
222 + \tilde{\mat{A}} = \mat{A} + \mat{I},
223 + \qquad
224 + \tilde{D}_{ii} = \sum_j \tilde{A}_{ij},
225 + \qquad
226 + \hat{\mat{A}}
227 + = \tilde{\mat{D}}^{-1/2}\, \tilde{\mat{A}}\, \tilde{\mat{D}}^{-1/2}.
228 + \label{eq:gnn-ahat}
229 +\end{equation}
230 +With $\mat{H}^{(0)} = \mat{X}$, the celebrated layer-wise propagation
231 +rule reads
232 +\begin{equation}
233 + \mat{H}^{(\ell+1)}
234 + = \varphi\bigl(\hat{\mat{A}}\, \mat{H}^{(\ell)}\, \mat{W}^{(\ell)}\bigr),
235 + \label{eq:gnn-gcnlayer}
236 +\end{equation}
237 +with $\mat{W}^{(\ell)}$ the trainable weights and $\varphi$ typically the
238 +ReLU. Equation~\eqref{eq:gnn-gcnlayer} is an instance of
239 +\eqref{eq:gnn-aggcomb}: written for a single node it becomes a
240 +degree-weighted mean over the closed neighbourhood,
241 +\begin{equation}
242 + \vect{h}_v^{(\ell+1)}
243 + = \varphi\Biggl(
244 + \sum_{u \in \mathcal{N}(v) \cup \{v\}}
245 + \frac{1}{\sqrt{\tilde{d}_v\, \tilde{d}_u}}\,
246 + \mat{W}^{(\ell)} \vect{h}_u^{(\ell)}
247 + \Biggr).
248 + \label{eq:gnn-gcnnode}
249 +\end{equation}
250 +The symmetric normalization $1/\sqrt{\tilde d_v \tilde d_u}$ downweights
251 +messages that either \emph{leave} or \emph{enter} a high-degree hub, and
252 +keeps the propagation operator symmetric, hence with a real spectrum.
253 +
254 +For semi-supervised node classification — the task that made the GCN
255 +famous — a two-layer network suffices:
256 +\begin{equation}
257 + \mat{Z} = \softmax\Bigl(
258 + \hat{\mat{A}}\,
259 + \operatorname{ReLU}\bigl(\hat{\mat{A}} \mat{X} \mat{W}^{(0)}\bigr)\,
260 + \mat{W}^{(1)}
261 + \Bigr),
262 + \label{eq:gnn-gcntwo}
263 +\end{equation}
264 +trained by minimizing the cross-entropy over the (small) labelled subset
265 +$V_L \subset V$ only,
266 +\begin{equation}
267 + \Loss(\theta)
268 + = -\sum_{v \in V_L} \sum_{c=1}^{C} Y_{vc} \ln Z_{vc},
269 + \label{eq:gnn-gcnloss}
270 +\end{equation}
271 +while the propagation through $\hat{\mat{A}}$ spreads label information
272 +to the unlabelled nodes — the graph structure itself acts as the
273 +regularizer.
274 +
275 +\begin{remark}[Over-smoothing and transductivity]\label{rem:gnn-oversmooth}
276 +Repeated multiplication by $\hat{\mat{A}}$ is a low-pass filter on the
277 +graph: as depth grows, all node states converge towards a
278 +degree-dependent stationary vector and become indistinguishable. Deep
279 +GCNs therefore \emph{lose} discriminative power; in practice two or three
280 +layers are optimal, and deeper stacks require residual connections or
281 +normalization to remain trainable. A second limitation is that
282 +\eqref{eq:gnn-gcntwo} needs the full matrix $\hat{\mat{A}}$ at training
283 +time: the vanilla GCN is \emph{transductive} and cannot embed nodes
284 +unseen during training.
285 +\end{remark}
286 +
287 +% ----------------------------------------------------------------------------
288 +\section{GraphSAGE: Inductive Learning by Sampled Aggregation}
289 +\label{sec:gnn-sage}
290 +% ----------------------------------------------------------------------------
291 +
292 +GraphSAGE~\cite{hamilton2017} removes both limitations of
293 +Remark~\ref{rem:gnn-oversmooth} at once: it learns \emph{aggregator
294 +functions} rather than per-node embeddings, and it evaluates them on
295 +\emph{sampled} fixed-size neighbourhoods $\mathcal{S}(v) \subseteq
296 +\mathcal{N}(v)$ with $|\mathcal{S}(v)| = s$, so that the cost per node is
297 +bounded regardless of the degree distribution. One layer performs
298 +\begin{align}
299 + \vect{h}_{\mathcal{N}(v)}^{(k)}
300 + &= \operatorname{AGG}_k\bigl(
301 + \{\!\!\{\vect{h}_u^{(k-1)} : u \in \mathcal{S}(v)\}\!\!\}\bigr),
302 + \label{eq:gnn-sageagg}\\
303 + \vect{h}_v^{(k)}
304 + &= \varphi\Bigl(\mat{W}^{(k)}
305 + \bigl[\vect{h}_v^{(k-1)} \,\Vert\, \vect{h}_{\mathcal{N}(v)}^{(k)}
306 + \bigr]\Bigr),
307 + \qquad
308 + \vect{h}_v^{(k)} \leftarrow
309 + \frac{\vect{h}_v^{(k)}}{\bigl\lVert \vect{h}_v^{(k)} \bigr\rVert_2},
310 + \label{eq:gnn-sageupd}
311 +\end{align}
312 +where $\Vert$ denotes concatenation. Concatenating — rather than summing
313 +— the self-state with the neighbourhood summary acts as a skip
314 +connection that preserves the node's own identity through depth. Two of
315 +the proposed aggregators are the mean and the max-pooling aggregator,
316 +\begin{equation}
317 + \operatorname{AGG}^{\text{mean}}
318 + = \frac{1}{|\mathcal{S}(v)|} \sum_{u \in \mathcal{S}(v)}
319 + \vect{h}_u^{(k-1)},
320 + \label{eq:gnn-sagemean}
321 +\end{equation}
322 +\begin{equation}
323 + \operatorname{AGG}^{\text{pool}}
324 + = \max_{u \in \mathcal{S}(v)}
325 + \varphi\bigl(\mat{W}_{\text{pool}}\, \vect{h}_u^{(k-1)}
326 + + \vect{b}\bigr),
327 + \label{eq:gnn-sagepool}
328 +\end{equation}
329 +the max taken element-wise (a third variant applies an LSTM to a random
330 +permutation of the neighbours — expressive, but not
331 +permutation-invariant). Because the aggregators are shared functions of
332 +local structure, a trained GraphSAGE model embeds \emph{previously unseen
333 +nodes} — and even entirely new graphs — by simply running
334 +\eqref{eq:gnn-sageagg}--\eqref{eq:gnn-sageupd} on their neighbourhoods:
335 +this is what \emph{inductive} means here.
336 +
337 +When no labels are available, GraphSAGE is trained with a random-walk
338 +co-occurrence loss with negative sampling,
339 +\begin{equation}
340 + J(\vect{z}_u)
341 + = -\ln \sigma\bigl(\vect{z}_u\transp \vect{z}_v\bigr)
342 + - Q \cdot \E_{v_n \sim P_n}
343 + \bigl[\ln \sigma\bigl(-\vect{z}_u\transp \vect{z}_{v_n}\bigr)\bigr],
344 + \label{eq:gnn-sageloss}
345 +\end{equation}
346 +which pulls together the embeddings of nodes $u, v$ that co-occur on
347 +short random walks and pushes $\vect{z}_u$ away from $Q$ negative
348 +samples $v_n$ drawn from a noise distribution $P_n$; here $\sigma$ is the
349 +logistic sigmoid. The supervised variant simply replaces
350 +\eqref{eq:gnn-sageloss} with the cross-entropy \eqref{eq:gnn-gcnloss} on
351 +the batch. Algorithm~\ref{alg:gnn-sage} gives the complete minibatch
352 +estimation procedure; the unrolled sampling it induces is visualized in
353 +Figure~\ref{fig:gnn-tree}.
354 +
355 +\begin{algorithm}[htbp]
356 + \caption{GraphSAGE minibatch training with neighbour sampling
357 + (supervised)}
358 + \label{alg:gnn-sage}
359 + \begin{algorithmic}[1]
360 + \Require graph $G$, features $\{\vect{x}_v\}$, labels on $V_L$,
361 + depth $K$, sample sizes $s_1, \dots, s_K$,
362 + learning rate $\eta$
363 + \While{not converged}
364 + \State sample a batch $B \subseteq V_L$;\quad $B^{(K)} \gets B$
365 + \For{$k = K, \dots, 1$}
366 + \Comment{backward neighbourhood expansion}
367 + \State $B^{(k-1)} \gets B^{(k)} \cup
368 + \bigcup_{v \in B^{(k)}} \mathcal{S}_k(v)$,
369 + \quad $|\mathcal{S}_k(v)| = s_k$
370 + \EndFor
371 + \State $\vect{h}_v^{(0)} \gets \vect{x}_v$
372 + for all $v \in B^{(0)}$
373 + \For{$k = 1, \dots, K$}
374 + \For{$v \in B^{(k)}$}
375 + \State $\vect{h}_{\mathcal{N}(v)}^{(k)} \gets$
376 + aggregate over $\mathcal{S}_k(v)$ by \eqref{eq:gnn-sageagg}
377 + \State $\vect{h}_v^{(k)} \gets$ combine and normalize by
378 + \eqref{eq:gnn-sageupd}
379 + \EndFor
380 + \EndFor
381 + \State $\Loss \gets -\frac{1}{|B|} \sum_{v \in B}
382 + \sum_{c} Y_{vc} \ln
383 + \softmax\bigl(\mat{W}_{\text{out}}\vect{h}_v^{(K)}\bigr)_c$
384 + \State $\theta \gets \theta - \eta\, \nabla_\theta \Loss$
385 + \Comment{SGD or Adam step}
386 + \EndWhile
387 + \end{algorithmic}
388 +\end{algorithm}
389 +
390 +\begin{figure}[htbp]
391 + \centering
392 + \begin{tikzpicture}[scale=1.0]
393 + % root
394 + \node[noutput] (r) at (0,0) {$v$};
395 + % level 1
396 + \node[nhidden] (a1) at (-3.0,-1.9) {$u_1$};
397 + \node[nhidden] (a2) at (0,-1.9) {$u_2$};
398 + \node[nhidden] (a3) at (3.0,-1.9) {$u_3$};
399 + % level 2
400 + \node[ninput] (b1) at (-4.4,-3.8) {$w_1$};
401 + \node[ninput] (b2) at (-3.0,-3.8) {$w_2$};
402 + \node[ninput] (b3) at (-1.6,-3.8) {$w_3$};
403 + \node[ninput] (b4) at (0.0,-3.8) {$w_4$};
404 + \node[ninput] (b5) at (1.6,-3.8) {$w_5$};
405 + \node[ninput] (b6) at (3.0,-3.8) {$w_6$};
406 + \node[ninput] (b7) at (4.4,-3.8) {$w_7$};
407 + % arrows upward (aggregation direction)
408 + \draw[fleche, chidden!85!black] (a1) -- (r);
409 + \draw[fleche, chidden!85!black] (a2) -- (r);
410 + \draw[fleche, chidden!85!black] (a3) -- (r);
411 + \draw[fleche, cinput!80!black] (b1) -- (a1);
412 + \draw[fleche, cinput!80!black] (b2) -- (a1);
413 + \draw[fleche, cinput!80!black] (b3) -- (a2);
414 + \draw[fleche, cinput!80!black] (b4) -- (a2);
415 + \draw[fleche, cinput!80!black] (b5) -- (a3);
416 + \draw[fleche, cinput!80!black] (b6) -- (a3);
417 + \draw[fleche, cinput!80!black] (b7) -- (a3);
418 + % depth annotations on the right
419 + \node[etiquette, anchor=west] at (5.4,0)
420 + {layer $k=2$: $\vect{h}_v^{(2)}$};
421 + \node[etiquette, anchor=west] at (5.4,-1.9)
422 + {layer $k=1$: $\vect{h}_{u_i}^{(1)}$};
423 + \node[etiquette, anchor=west] at (5.4,-3.8)
424 + {layer $k=0$: $\vect{h}_{w_j}^{(0)} = \vect{x}_{w_j}$};
425 + % brace for sampled neighbourhoods
426 + \draw[decorate, decoration={brace, mirror, amplitude=5pt},
427 + black!60]
428 + (-4.9,-4.5) -- (4.9,-4.5)
429 + node[etiquette, midway, below=7pt]
430 + {sampled two-hop neighbourhood:
431 + $w_j \in \mathcal{S}_1(u_i)$, $u_i \in \mathcal{S}_2(v)$};
432 + \end{tikzpicture}
433 + \caption{The computation tree unrolled by a depth-2 GraphSAGE forward
434 + pass at node $v$ (Algorithm~\ref{alg:gnn-sage}). Layer-0 states of
435 + the sampled two-hop nodes (blue) are aggregated into layer-1 states
436 + of the sampled one-hop neighbours (orange), which are in turn
437 + aggregated into the final state of $v$ (red). Sampling fixes the
438 + branching factor of the tree, bounding the cost independently of the
439 + node degrees.}
440 + \label{fig:gnn-tree}
441 +\end{figure}
442 +
443 +% ----------------------------------------------------------------------------
444 +\section{Graph Attention Networks}\label{sec:gnn-gat}
445 +% ----------------------------------------------------------------------------
446 +
447 +The GCN weighs the message from $u$ to $v$ by the purely structural
448 +coefficient $1/\sqrt{\tilde d_v \tilde d_u}$ of
449 +\eqref{eq:gnn-gcnnode}: two neighbours with equal degrees are equally
450 +important, whatever their features. Graph attention
451 +networks~\cite{velickovic2018} replace this fixed coefficient with a
452 +\emph{learned}, feature-dependent one, importing the attention mechanism
453 +into message passing. With a shared projection
454 +$\mat{W} \in \R^{F' \times F}$ and an attention vector
455 +$\vect{a} \in \R^{2F'}$, the unnormalized score of edge $(j \to i)$ is
456 +\begin{equation}
457 + e_{ij}
458 + = \operatorname{LeakyReLU}\Bigl(
459 + \vect{a}\transp
460 + \bigl[\mat{W}\vect{h}_i \,\Vert\, \mat{W}\vect{h}_j\bigr]
461 + \Bigr),
462 + \label{eq:gnn-gatlogit}
463 +\end{equation}
464 +normalized by a softmax masked to the neighbourhood (including $i$
465 +itself),
466 +\begin{equation}
467 + \alpha_{ij}
468 + = \frac{\exp(e_{ij})}
469 + {\sum_{k \in \mathcal{N}(i) \cup \{i\}} \exp(e_{ik})},
470 + \label{eq:gnn-gatalpha}
471 +\end{equation}
472 +and the node update is the attention-weighted aggregation
473 +\begin{equation}
474 + \vect{h}_i'
475 + = \varphi\Biggl(
476 + \sum_{j \in \mathcal{N}(i) \cup \{i\}}
477 + \alpha_{ij}\, \mat{W} \vect{h}_j
478 + \Biggr).
479 + \label{eq:gnn-gatupd}
480 +\end{equation}
481 +As in the Transformer, several attention heads stabilize learning and
482 +attend to different relational patterns; hidden layers concatenate the
483 +heads while the final prediction layer averages them:
484 +\begin{equation}
485 + \vect{h}_i'
486 + = \bigl\Vert_{k=1}^{K} \varphi\Bigl(
487 + \textstyle\sum_{j} \alpha_{ij}^{k}\, \mat{W}^{k} \vect{h}_j
488 + \Bigr),
489 + \qquad
490 + \vect{h}_i'
491 + = \varphi\Bigl(
492 + \tfrac{1}{K} \textstyle\sum_{k=1}^{K} \sum_{j}
493 + \alpha_{ij}^{k}\, \mat{W}^{k} \vect{h}_j
494 + \Bigr)
495 + \quad \text{(final layer)}.
496 + \label{eq:gnn-gatmulti}
497 +\end{equation}
498 +Figure~\ref{fig:gnn-gat} shows the resulting anisotropic aggregation:
499 +unlike in the GCN, the incoming edges of a node carry \emph{different}
500 +weights, and those weights change with the node features rather than
501 +being frozen by the topology. GAT is inductive for the same reason
502 +GraphSAGE is — all parameters ($\mat{W}$, $\vect{a}$) are shared
503 +functions, none is tied to a node identity — and the attention
504 +coefficients offer a degree of built-in interpretability: inspecting
505 +$\alpha_{ij}$ reveals which neighbours drove a prediction.
506 +
507 +\begin{remark}
508 +The scoring function \eqref{eq:gnn-gatlogit} applies its nonlinearity
509 +\emph{after} the inner product with $\vect{a}$; the neighbour ranking it
510 +induces is therefore shared by all query nodes (\emph{static}
511 +attention). Moving the nonlinearity inside,
512 +$e_{ij} = \vect{a}\transp \operatorname{LeakyReLU}
513 +(\mat{W}[\vect{h}_i \Vert \vect{h}_j])$, yields the strictly more
514 +expressive dynamic variant known as GATv2.
515 +\end{remark}
516 +
517 +\begin{figure}[htbp]
518 + \centering
519 + \begin{tikzpicture}[scale=1.0]
520 + \node[noutput] (v) at (0,0) {$i$};
521 + \node[ninput] (u1) at (-2.3,1.6) {$j_1$};
522 + \node[ninput] (u2) at (-2.7,-0.6){$j_2$};
523 + \node[ninput] (u3) at (-0.4,-2.3){$j_3$};
524 + \node[ninput] (u4) at (2.1,1.6) {$j_4$};
525 + % head 1 (blue, thickness ~ alpha), arcs on one side of each chord
526 + \draw[fleche, cinput!80!black, line width=1.8pt]
527 + (u1) to[bend left=16] (v);
528 + \draw[fleche, cinput!80!black, line width=0.6pt]
529 + (u2) to[bend left=16] (v);
530 + \draw[fleche, cinput!80!black, line width=1.1pt]
531 + (u3) to[bend left=16] (v);
532 + \draw[fleche, cinput!80!black, line width=1.1pt]
533 + (u4) to[bend right=16] (v);
534 + % head 2 (orange), arcs on the other side
535 + \draw[fleche, chidden!85!black, line width=0.7pt]
536 + (u1) to[bend right=16] (v);
537 + \draw[fleche, chidden!85!black, line width=1.7pt]
538 + (u2) to[bend right=16] (v);
539 + \draw[fleche, chidden!85!black, line width=0.8pt]
540 + (u3) to[bend right=16] (v);
541 + \draw[fleche, chidden!85!black, line width=1.2pt]
542 + (u4) to[bend left=16] (v);
543 + % coefficient labels, colour-coded by head, pinned off the arcs
544 + \node[etiquette, text=cinput!80!black] at (-0.85, 1.30) {$0.42$};
545 + \node[etiquette, text=chidden!85!black] at (-2.05, 0.55) {$0.15$};
546 + \node[etiquette, text=cinput!80!black] at (-1.55, 0.10) {$0.11$};
547 + \node[etiquette, text=chidden!85!black] at (-1.75,-1.10) {$0.39$};
548 + \node[etiquette, text=cinput!80!black] at (-0.95,-1.35) {$0.23$};
549 + \node[etiquette, text=chidden!85!black] at ( 0.55,-1.45) {$0.18$};
550 + \node[etiquette, text=cinput!80!black] at ( 1.05, 1.30) {$0.24$};
551 + \node[etiquette, text=chidden!85!black] at ( 1.90, 0.55) {$0.28$};
552 + % legend
553 + \node[etiquette, anchor=west, text=cinput!80!black]
554 + at (4.0,0.7) {head $k=1$: coefficients $\alpha_{ij}^{1}$};
555 + \node[etiquette, anchor=west, text=chidden!85!black]
556 + at (4.0,0.1) {head $k=2$: coefficients $\alpha_{ij}^{2}$};
557 + \node[etiquette, anchor=west, align=left] at (4.0,-0.9)
558 + {line width $\propto \alpha_{ij}^{k}$;\\
559 + each head sums to $1$ over\\
560 + $\mathcal{N}(i)\cup\{i\}$ (self-loop\\
561 + omitted for clarity)};
562 + \end{tikzpicture}
563 + \caption{Graph attention at node $i$,
564 + equations~\eqref{eq:gnn-gatlogit}--\eqref{eq:gnn-gatmulti}. Each of
565 + the two heads (blue, orange) computes its own normalized
566 + coefficients $\alpha_{ij}^{k}$ over the same neighbourhood; the
567 + thickness of each arrow is proportional to the learned coefficient.
568 + Structurally identical neighbours may thus receive very different
569 + weights, in contrast with the degree-based coefficients of the GCN
570 + in \eqref{eq:gnn-gcnnode}.}
571 + \label{fig:gnn-gat}
572 +\end{figure}
573 +
574 +% ----------------------------------------------------------------------------
575 +\section{How Powerful Are GNNs? The Graph Isomorphism Network}
576 +\label{sec:gnn-gin}
577 +% ----------------------------------------------------------------------------
578 +
579 +The freedom in choosing $\operatorname{AGG}$ in \eqref{eq:gnn-aggcomb}
580 +raises a theoretical question: which graphs can a message-passing network
581 +tell apart at all? Xu et al.~\cite{xu2019} answered it by relating GNNs
582 +to the classical one-dimensional Weisfeiler--Lehman (1-WL) colour
583 +refinement test, which iteratively re-hashes each node's colour together
584 +with the multiset of its neighbours' colours.
585 +
586 +\begin{theorem}[Expressive power of message passing]
587 +\label{thm:gnn-wl}
588 +Any GNN of the form \eqref{eq:gnn-aggcomb} maps two non-isomorphic graphs
589 +to different embeddings only if the 1-WL test also distinguishes them:
590 +message passing is \emph{at most} as discriminative as 1-WL. This upper
591 +bound is attained if the aggregation, combination and readout functions
592 +are all \emph{injective} on multisets~\cite{xu2019}.
593 +\end{theorem}
594 +
595 +Injectivity is where the common aggregators part ways. The mean loses
596 +multiplicities — it cannot distinguish
597 +$\{\!\!\{\vect{a}, \vect{a}, \vect{b}, \vect{b}\}\!\!\}$ from
598 +$\{\!\!\{\vect{a}, \vect{b}\}\!\!\}$ — and the max loses everything but
599 +the support; the \emph{sum} preserves both, and over countable feature
600 +spaces sum-based aggregation composed with an MLP can represent any
601 +multiset function.
602 +
603 +\begin{property}[Aggregator ranking]\label{prop:gnn-agg}
604 +In discriminative power over multisets,
605 +$\mathrm{sum} \succ \mathrm{mean} \succ \mathrm{max}$: the mean captures
606 +the distribution of neighbour features but not their multiplicities; the
607 +max captures only the underlying set.
608 +\end{property}
609 +
610 +The graph isomorphism network makes the injective choice concrete, with
611 +a learnable scalar $\epsilon^{(k)}$ that disambiguates the node's own
612 +state from the neighbour sum, and an MLP as a universal approximator on
613 +top:
614 +\begin{equation}
615 + \vect{h}_v^{(k)}
616 + = \operatorname{MLP}^{(k)}\Bigl(
617 + \bigl(1 + \epsilon^{(k)}\bigr)\, \vect{h}_v^{(k-1)}
618 + + \sum_{u \in \mathcal{N}(v)} \vect{h}_u^{(k-1)}
619 + \Bigr).
620 + \label{eq:gnn-gin}
621 +\end{equation}
622 +For graph-level prediction, GIN concatenates a summed readout of
623 +\emph{every} depth, retaining both local and global structure:
624 +\begin{equation}
625 + \vect{h}_G
626 + = \bigl\Vert_{k=0}^{K}\,
627 + \sum_{v \in V} \vect{h}_v^{(k)}.
628 + \label{eq:gnn-ginread}
629 +\end{equation}
630 +By Theorem~\ref{thm:gnn-wl}, the network
631 +\eqref{eq:gnn-gin}--\eqref{eq:gnn-ginread} is a \emph{maximally
632 +powerful} message-passing GNN: whatever 1-WL can distinguish, GIN can
633 +learn to distinguish.
634 +
635 +\begin{remark}
636 +The 1-WL ceiling is a genuine ceiling: no network of the form
637 +\eqref{eq:gnn-aggcomb} can, for instance, count triangles or separate
638 +certain pairs of regular graphs. Escaping it requires strictly more
639 +machinery — higher-order message passing over node tuples, random node
640 +identifiers, or positional and structural encodings appended to the
641 +input features $\vect{x}_v$.
642 +\end{remark}
643 +
644 +% ----------------------------------------------------------------------------
645 +\section{Prediction Heads and Training Objectives}\label{sec:gnn-heads}
646 +% ----------------------------------------------------------------------------
647 +
648 +The message-passing trunk of
649 +Sections~\ref{sec:gnn-mp}--\ref{sec:gnn-gin} is shared by three families
650 +of tasks, which differ only in the head applied to the final states
651 +$\vect{z}_v = \vect{h}_v^{(K)}$:
652 +\begin{align}
653 + \hat{\vect{y}}_v
654 + &= \softmax\bigl(\mat{W}_{\text{out}}\, \vect{z}_v\bigr)
655 + && \text{(node classification)},
656 + \label{eq:gnn-nodehead}\\
657 + s_{uv}
658 + &= \sigma\bigl(\vect{z}_u\transp \vect{z}_v\bigr)
659 + && \text{(link prediction)},
660 + \label{eq:gnn-linkhead}\\
661 + \hat{\vect{y}}_G
662 + &= \softmax\bigl(\mat{W}_G\, \vect{h}_G\bigr),
663 + \qquad \vect{h}_G \text{ from } \eqref{eq:gnn-ginread}
664 + && \text{(graph classification)}.
665 + \label{eq:gnn-graphhead}
666 +\end{align}
667 +Node and graph heads are trained with the cross-entropy
668 +\eqref{eq:gnn-gcnloss}; the link-prediction head with the logistic loss
669 +of \eqref{eq:gnn-sageloss}, treating observed edges as positives and
670 +sampled non-edges as negatives. All four architectures of this chapter
671 +slot into this scheme unchanged — the choice among GCN, GraphSAGE, GAT
672 +and GIN is a choice of aggregation rule
673 +(\eqref{eq:gnn-gcnnode}, \eqref{eq:gnn-sageagg},
674 +\eqref{eq:gnn-gatupd}, \eqref{eq:gnn-gin}), not a choice of task.
675 +
676 +These models power a remarkable range of applications: semi-supervised
677 +classification of citation networks, molecular property prediction and
678 +drug discovery (message passing over atoms and bonds), billion-node
679 +recommender systems built on sampled aggregation, learned physics
680 +simulation, traffic forecasting and protein structure prediction, where
681 +attention over residue-pair graphs is a central ingredient. In every
682 +case the inductive bias is the same: what a node \emph{is} should be
683 +computable from what its neighbourhood \emph{looks like} — the
684 +graph-structured analogue of the translation equivariance that motivated
685 +convolutional networks.
added chapters/09-energy.tex +596 −0
@@ -0,0 +1,596 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 9 : Energy-Based and Associative-Memory Networks
6 +% (chapters/09-energy.tex)
7 +% ============================================================================
8 +\chapter{Energy-Based and Associative-Memory Networks}\label{chap:energy}
9 +
10 +The architectures of the preceding chapters compute by \emph{propagating}
11 +signals through a directed graph. This chapter studies a different
12 +computational metaphor, imported from statistical physics: a network is
13 +assigned a scalar \emph{energy} for every joint configuration of its units,
14 +and computation proceeds by \emph{relaxation} — the state slides downhill
15 +on the energy surface until it settles in a minimum. Memories are not
16 +stored at addresses; they are sculpted into the landscape as attractors,
17 +and retrieval is descent. The deterministic version of this idea is the
18 +Hopfield network \cite{hopfield1982}; its stochastic generalization, the
19 +Boltzmann machine \cite{ackley1985}, turns the energy into a probability
20 +distribution and thereby becomes a trainable generative model. The
21 +restricted Boltzmann machine and its stack, the deep belief network
22 +\cite{hinton2006}, converted these physical ideas into the first practical
23 +recipe for training deep architectures — the spark of the modern deep
24 +learning era.
25 +
26 +\section{The Classical Hopfield Network}
27 +
28 +\begin{definition}[Hopfield network]\label{def:en-hopfield}
29 +A Hopfield network is a fully connected recurrent network of $N$ binary
30 +units with states $s_i \in \{-1,+1\}$, symmetric coupling weights
31 +$w_{ij} = w_{ji}$, no self-connections ($w_{ii}=0$), and thresholds
32 +$\theta_i$. The network state is the vector
33 +$\vect{s} = (s_1,\dots,s_N)\transp \in \{-1,+1\}^N$.
34 +\end{definition}
35 +
36 +Every configuration $\vect{s}$ is scored by the energy function
37 +\begin{equation}
38 + E(\vect{s})
39 + \;=\;
40 + -\frac{1}{2}\sum_{i,j} w_{ij}\, s_i s_j \;+\; \sum_i \theta_i s_i .
41 + \label{eq:en-hopfield-energy}
42 +\end{equation}
43 +The dynamics are \emph{asynchronous}: at each step a single unit $i$ is
44 +selected (in random or cyclic order) and set to the sign of its local
45 +field,
46 +\begin{equation}
47 + s_i \;\leftarrow\;
48 + \operatorname{sign}\!\Bigl(\sum_j w_{ij}\, s_j - \theta_i\Bigr).
49 + \label{eq:en-hopfield-update}
50 +\end{equation}
51 +The central fact — and the reason the network computes anything at all —
52 +is that this update can only lower the energy.
53 +
54 +\begin{theorem}[Energy descent]\label{thm:en-descent}
55 +Under the asynchronous update \eqref{eq:en-hopfield-update} with symmetric
56 +weights and $w_{ii}=0$, the energy \eqref{eq:en-hopfield-energy} never
57 +increases. Since $E$ is bounded below on the finite state space, the
58 +dynamics converge in finitely many steps to a fixed point — a local
59 +minimum of $E$.
60 +\end{theorem}
61 +
62 +\begin{proof}
63 +Suppose unit $i$ flips from $s_i$ to $s_i' = -s_i$; all other units are
64 +frozen. Writing $h_i = \sum_j w_{ij}s_j - \theta_i$ for the local field,
65 +the energy change collects exactly the terms involving $s_i$ (symmetry
66 +counts each pair once, and $w_{ii}=0$ removes the diagonal):
67 +$\Delta E = E(\vect{s}') - E(\vect{s}) = -(s_i' - s_i)\,h_i$.
68 +The rule \eqref{eq:en-hopfield-update} flips $s_i$ only when $s_i$ and
69 +$h_i$ disagree, i.e.\ when $s_i' = \operatorname{sign}(h_i)$, so
70 +$(s_i' - s_i)\,h_i = 2\,|h_i| \ge 0$ and $\Delta E \le 0$. A strict flip
71 +with $h_i \neq 0$ strictly decreases $E$; as the state space is finite,
72 +only finitely many strict decreases can occur.
73 +\end{proof}
74 +
75 +\subsection{Hebbian storage and capacity}
76 +
77 +To turn descent into \emph{recall}, one shapes the landscape so that the
78 +desired memories $\{\vect{\xi}^\mu\}_{\mu=1}^{P}$,
79 +$\vect{\xi}^\mu \in \{-1,+1\}^N$, sit at the bottoms of basins. The
80 +Hebbian (outer-product) prescription is
81 +\begin{equation}
82 + w_{ij} \;=\; \frac{1}{N}\sum_{\mu=1}^{P} \xi_i^\mu\, \xi_j^\mu ,
83 + \qquad w_{ii}=0,
84 + \label{eq:en-hebb}
85 +\end{equation}
86 +which makes each $\vect{\xi}^\mu$ (approximately) a fixed point of
87 +\eqref{eq:en-hopfield-update}: units that fire together are wired
88 +together, so a corrupted probe within a basin is pulled back to the
89 +stored pattern. This is \emph{content-addressable} memory — the query is
90 +a partial or noisy version of the content itself.
91 +
92 +\begin{remark}[Capacity and spurious states]\label{rem:en-capacity}
93 +Statistical-mechanics analysis of \eqref{eq:en-hebb} shows that reliable
94 +retrieval survives only up to
95 +\begin{equation}
96 + P_{\max} \;\approx\; 0.138\, N
97 + \label{eq:en-capacity}
98 +\end{equation}
99 +stored patterns; for essentially error-free recall the bound tightens to
100 +$N/(2\ln N)$. Beyond capacity, crosstalk between patterns proliferates
101 +\emph{spurious attractors} — mixture states and inverted patterns that
102 +were never stored — and retrieval degrades abruptly (catastrophic
103 +interference).
104 +\end{remark}
105 +
106 +\begin{algorithm}[htbp]
107 +\caption{Hopfield network: Hebbian storage and asynchronous recall}
108 +\label{alg:en-hopfield}
109 +\begin{algorithmic}[1]
110 +\Require patterns $\vect{\xi}^1,\dots,\vect{\xi}^P \in \{-1,+1\}^N$;
111 + probe $\vect{x} \in \{-1,+1\}^N$
112 +\State $w_{ij} \gets \frac{1}{N}\sum_{\mu=1}^{P}\xi_i^\mu \xi_j^\mu$
113 + for all $i \neq j$; \quad $w_{ii} \gets 0$
114 + \Comment{storage \eqref{eq:en-hebb}}
115 +\State $\vect{s} \gets \vect{x}$ \Comment{initialize at the probe}
116 +\Repeat
117 + \State draw a unit index $i$ (random or cyclic sweep)
118 + \State $h_i \gets \sum_j w_{ij} s_j - \theta_i$
119 + \State $s_i \gets \operatorname{sign}(h_i)$
120 + \Comment{energy never increases (Theorem~\ref{thm:en-descent})}
121 +\Until{no unit changed during a full sweep}
122 +\State \Return $\vect{s}$ \Comment{an attractor: ideally the stored
123 + pattern nearest $\vect{x}$}
124 +\end{algorithmic}
125 +\end{algorithm}
126 +
127 +Figure~\ref{fig:en-hopfield} shows the two complementary views of the same
128 +object: the fully connected graph of units, and the energy landscape over
129 +the state space in which the stored patterns are the basins.
130 +
131 +\begin{figure}[htbp]
132 + \centering
133 + \begin{tikzpicture}
134 + % ---- left: fully connected 6-unit Hopfield graph (hexagon) ----
135 + \begin{scope}[local bounding box=graph]
136 + \foreach \i in {1,...,6}
137 + \node[nhidden] (s\i) at ({90-(\i-1)*60}:1.9) {$s_{\i}$};
138 + \foreach \i [evaluate=\i as \nexti using int(\i+1)] in {1,...,5}
139 + \foreach \j in {\nexti,...,6}
140 + \draw[black!45, semithick] (s\i) -- (s\j);
141 + \node[etiquette, sloped, above] at ($(s1)!0.5!(s2)$) {$w_{12}$};
142 + \node[etiquette, sloped, above] at ($(s2)!0.5!(s3)$) {$w_{23}$};
143 + \node[etiquette, sloped, below] at ($(s4)!0.5!(s5)$) {$w_{45}$};
144 + \end{scope}
145 + \node[etiquette, align=center] at ($(graph.south)+(0,-0.5)$)
146 + {fully connected, symmetric $w_{ij}=w_{ji}$,\\ no self-loops};
147 +
148 + % ---- right: 1D energy landscape with two basins ----
149 + \begin{scope}[xshift=6.4cm, yshift=-2.2cm]
150 + \begin{axis}[
151 + width=7.2cm, height=5.6cm,
152 + axis lines=left,
153 + xlabel={state space (schematic)},
154 + ylabel={$E(\vect{s})$},
155 + xtick=\empty, ytick=\empty,
156 + domain=-2.35:2.45, samples=120,
157 + ymin=-1.9, ymax=1.4,
158 + clip=false,
159 + ]
160 + \addplot[cmem, very thick] {0.25*x^4 - x^2 + 0.15*x};
161 + % basin minima near x=-1.49 and x=+1.34
162 + \node[etiquette, align=center] at (axis cs:-1.49,-1.55)
163 + {stored pattern $\vect{\xi}^1$};
164 + \node[etiquette, align=center] at (axis cs:1.34,-1.35)
165 + {stored pattern $\vect{\xi}^2$};
166 + % descending ball
167 + \fill[coutput] (axis cs:-2.18,0.57) circle (2.2pt);
168 + \node[etiquette, anchor=west] at (axis cs:-1.98,0.85)
169 + {probe $\vect{x}$};
170 + \draw[fleche, coutput]
171 + (axis cs:-2.1,0.32) to[bend left=25] (axis cs:-1.62,-0.85);
172 + \end{axis}
173 + \end{scope}
174 + \end{tikzpicture}
175 + \caption{The Hopfield network \cite{hopfield1982}. Left: six binary
176 + units with symmetric all-to-all couplings (a few weights labeled).
177 + Right: the energy \eqref{eq:en-hopfield-energy} as a landscape over
178 + the state space; Hebbian storage \eqref{eq:en-hebb} carves one basin
179 + per memory, and asynchronous updates
180 + \eqref{eq:en-hopfield-update} roll a noisy probe downhill into the
181 + nearest attractor.}
182 + \label{fig:en-hopfield}
183 +\end{figure}
184 +
185 +\section{Modern Hopfield Networks}
186 +
187 +The classical construction is limited by its $O(N)$ capacity
188 +\eqref{eq:en-capacity}. Modern Hopfield networks \cite{ramsauer2021}
189 +replace binary states by continuous queries $\vect{q} \in \R^{d}$ and the
190 +quadratic energy by a \emph{log-sum-exp} energy over stored patterns
191 +$\mat{X} = [\vect{x}_1,\dots,\vect{x}_P] \in \R^{d\times P}$:
192 +\begin{equation}
193 + E(\vect{q})
194 + \;=\;
195 + -\frac{1}{\beta}\,
196 + \log \sum_{i=1}^{P} \exp\bigl(\beta\, \vect{x}_i\transp \vect{q}\bigr)
197 + \;+\; \frac{1}{2}\,\lVert \vect{q}\rVert^2 \;+\; \text{const},
198 + \label{eq:en-mhn-energy}
199 +\end{equation}
200 +where $\beta > 0$ is an inverse temperature. Minimizing
201 +\eqref{eq:en-mhn-energy} by a concave--convex procedure yields the
202 +strikingly simple fixed-point iteration
203 +\begin{equation}
204 + \vect{q}^{\mathrm{new}}
205 + \;=\;
206 + \mat{X}\,\softmax\bigl(\beta\, \mat{X}\transp \vect{q}\bigr),
207 + \label{eq:en-mhn-update}
208 +\end{equation}
209 +a convex combination of the stored patterns weighted by their similarity
210 +to the query.
211 +
212 +\begin{property}[Exponential capacity, one-step retrieval]
213 +\label{prop:en-mhn}
214 +For patterns placed on a sphere in $\R^d$, the energy
215 +\eqref{eq:en-mhn-energy} admits on the order of $\exp(c\,d)$
216 +well-separated attractors (for a constant $c$ depending on the required
217 +separation), and for well-separated patterns the update
218 +\eqref{eq:en-mhn-update} retrieves the correct pattern to exponentially
219 +small error in a \emph{single} step \cite{ramsauer2021}.
220 +\end{property}
221 +
222 +\begin{remark}[Associative memory is attention]\label{rem:en-attention}
223 +Read \eqref{eq:en-mhn-update} with $\vect{q}$ as a query and the columns
224 +of $\mat{X}$ as keys and values: it is exactly the scaled dot-product
225 +attention update of the Transformer, with $\beta = 1/\sqrt{d_k}$.
226 +Retrieval from an associative memory and an attention head are the same
227 +computation — a result that unifies one of the oldest ideas in the field
228 +with one of the newest.
229 +\end{remark}
230 +
231 +\section{Boltzmann Machines}
232 +
233 +The Hopfield network is deterministic: it can only descend, so it settles
234 +in whichever basin it starts in, and it has no principled notion of
235 +\emph{learning} a data distribution. The Boltzmann machine
236 +\cite{ackley1985} makes two moves. First, units become stochastic, and
237 +the network is augmented with \emph{hidden} units $\vect{h}$ alongside
238 +the \emph{visible} units $\vect{v}$ that carry the data; the energy
239 +$E(\vect{v},\vect{h})$ keeps the quadratic form of
240 +\eqref{eq:en-hopfield-energy} over the concatenated state. Second, the
241 +energy is exponentiated into a Gibbs (Boltzmann) distribution at
242 +temperature $T$:
243 +\begin{equation}
244 + p(\vect{v},\vect{h})
245 + \;=\;
246 + \frac{e^{-E(\vect{v},\vect{h})/T}}{Z},
247 + \qquad
248 + Z \;=\; \sum_{\vect{v}',\vect{h}'} e^{-E(\vect{v}',\vect{h}')/T},
249 + \label{eq:en-bm-gibbs}
250 +\end{equation}
251 +so that low-energy configurations are exponentially more probable. The
252 +network is now a latent-variable generative model, and maximum-likelihood
253 +learning has a beautifully symmetric gradient: for a coupling $w_{ij}$
254 +between any two units $s_i$ and $s_j$,
255 +\begin{equation}
256 + \frac{\partial \log p(\vect{v})}{\partial w_{ij}}
257 + \;=\;
258 + \E\bigl[s_i s_j\bigr]_{\text{data}}
259 + \;-\;
260 + \E\bigl[s_i s_j\bigr]_{\text{model}},
261 + \label{eq:en-bm-grad}
262 +\end{equation}
263 +a \emph{positive phase} that measures correlations with the visibles
264 +clamped to data, minus a \emph{negative phase} that measures the same
265 +correlations under the model's own equilibrium distribution
266 +\eqref{eq:en-bm-gibbs}. Learning stops when the model dreams what it
267 +sees. The catch is the negative phase: equilibrium expectations require
268 +Markov chains run to convergence under an intractable $Z$, which made
269 +general Boltzmann machines impractical for two decades.
270 +
271 +\section{Restricted Boltzmann Machines and Contrastive Divergence}
272 +
273 +\begin{definition}[Restricted Boltzmann machine]\label{def:en-rbm}
274 +An RBM is a Boltzmann machine whose connectivity graph is
275 +\emph{bipartite}: $n_v$ visible units $\vect{v} \in \{0,1\}^{n_v}$ and
276 +$n_h$ hidden units $\vect{h} \in \{0,1\}^{n_h}$ interact through
277 +$\mat{W} \in \R^{n_v \times n_h}$, with biases $\vect{a}, \vect{b}$ and
278 +\emph{no} intra-layer connections. Its energy is
279 +\begin{equation}
280 + E(\vect{v},\vect{h})
281 + \;=\;
282 + -\vect{a}\transp \vect{v}
283 + \;-\; \vect{b}\transp \vect{h}
284 + \;-\; \vect{v}\transp \mat{W}\, \vect{h},
285 + \label{eq:en-rbm-energy}
286 +\end{equation}
287 +with joint distribution
288 +$p(\vect{v},\vect{h}) = e^{-E(\vect{v},\vect{h})}/Z$ as in
289 +\eqref{eq:en-bm-gibbs} (absorbing $T$ into the parameters).
290 +\end{definition}
291 +
292 +Bipartiteness (Figure~\ref{fig:en-rbm}) is precisely what makes the model
293 +usable: conditioned on one layer, the units of the other layer are
294 +independent, and the conditionals factorize into sigmoids,
295 +\begin{align}
296 + p(h_j = 1 \mid \vect{v})
297 + &\;=\;
298 + \sigma\Bigl(b_j + \sum_i v_i\, w_{ij}\Bigr),
299 + \label{eq:en-rbm-condh}\\
300 + p(v_i = 1 \mid \vect{h})
301 + &\;=\;
302 + \sigma\Bigl(a_i + \sum_j w_{ij}\, h_j\Bigr),
303 + \label{eq:en-rbm-condv}
304 +\end{align}
305 +with $\sigma(x) = 1/(1+e^{-x})$. An entire layer can therefore be sampled
306 +in one parallel \emph{block Gibbs} step. Marginalizing the hiddens also
307 +gives a closed-form \emph{free energy} for a visible vector,
308 +\begin{equation}
309 + F(\vect{v})
310 + \;=\;
311 + -\vect{a}\transp\vect{v}
312 + \;-\;
313 + \sum_{j=1}^{n_h}
314 + \log\Bigl(1 + e^{\,b_j + \vect{v}\transp \mat{W}_{:j}}\Bigr),
315 + \qquad
316 + p(\vect{v}) = \frac{e^{-F(\vect{v})}}{Z},
317 + \label{eq:en-rbm-free}
318 +\end{equation}
319 +useful for monitoring training. The exact likelihood gradient specializes
320 +\eqref{eq:en-bm-grad} to
321 +\begin{equation}
322 + \frac{\partial \log p(\vect{v})}{\partial w_{ij}}
323 + \;=\;
324 + \E\bigl[v_i h_j\bigr]_{\text{data}}
325 + \;-\;
326 + \E\bigl[v_i h_j\bigr]_{\text{model}} .
327 + \label{eq:en-rbm-grad}
328 +\end{equation}
329 +
330 +\begin{figure}[htbp]
331 + \centering
332 + \begin{tikzpicture}
333 + \foreach \j in {1,...,3}
334 + \node[nhidden] (h\j) at (1.1+\j*1.7, 2.1) {$h_{\j}$};
335 + \foreach \i in {1,...,4}
336 + \node[ninput] (v\i) at (0.25+\i*1.7, 0) {$v_{\i}$};
337 + \foreach \i in {1,...,4}
338 + \foreach \j in {1,...,3}
339 + \draw[black!45, semithick] (v\i) -- (h\j);
340 + \node[etiquette, left=3mm of h1] {hidden layer $\vect{h}$};
341 + \node[etiquette, left=3mm of v1] {visible layer $\vect{v}$};
342 + \node[etiquette, right=2mm of h3, yshift=-9mm] {$\mat{W}$};
343 + \node[etiquette, align=center] at (4.2,-0.95)
344 + {bipartite: no intra-layer connections};
345 + \end{tikzpicture}
346 + \caption{The restricted Boltzmann machine
347 + (Definition~\ref{def:en-rbm}). Visible and hidden units interact
348 + through $\mat{W}$ only across layers; the missing intra-layer edges
349 + are what make the conditionals
350 + \eqref{eq:en-rbm-condh}--\eqref{eq:en-rbm-condv} factorize and block
351 + Gibbs sampling efficient.}
352 + \label{fig:en-rbm}
353 +\end{figure}
354 +
355 +\subsection{Contrastive divergence}
356 +
357 +The negative phase of \eqref{eq:en-rbm-grad} still asks for equilibrium
358 +samples. Contrastive divergence \cite{hinton2002} replaces equilibrium
359 +with \emph{proximity}: run the block Gibbs chain
360 +\begin{equation}
361 + \vect{v}^{(0)} \xrightarrow{\;p(\vect{h}\mid\vect{v})\;} \vect{h}^{(0)}
362 + \xrightarrow{\;p(\vect{v}\mid\vect{h})\;} \vect{v}^{(1)}
363 + \xrightarrow{\;\cdots\;} \vect{v}^{(k)}
364 + \xrightarrow{\;p(\vect{h}\mid\vect{v})\;} \vect{h}^{(k)}
365 + \label{eq:en-gibbs-chain}
366 +\end{equation}
367 +for only $k$ steps (typically $k=1$), \emph{started at the data}
368 +$\vect{v}^{(0)}$, and contrast the correlations at both ends:
369 +\begin{equation}
370 + \Delta w_{ij}
371 + \;\propto\;
372 + \bigl\langle v_i h_j \bigr\rangle_{0}
373 + \;-\;
374 + \bigl\langle v_i h_j \bigr\rangle_{k}
375 + \qquad \text{(CD-}k\text{)}.
376 + \label{eq:en-cdk}
377 +\end{equation}
378 +The estimator is biased — the chain has not mixed — but the bias is small
379 +near the data manifold and the signal is strong enough to train excellent
380 +features; persistent CD further improves the negative samples by never
381 +restarting the chain. Algorithm~\ref{alg:en-cdk} is the chapter's
382 +estimation centerpiece, and Figure~\ref{fig:en-cdk} traces the chain
383 +\eqref{eq:en-gibbs-chain}.
384 +
385 +\begin{algorithm}[htbp]
386 +\caption{Contrastive divergence CD-$k$ for one mini-batch (RBM)}
387 +\label{alg:en-cdk}
388 +\begin{algorithmic}[1]
389 +\Require mini-batch $\{\vect{v}_1,\dots,\vect{v}_m\}$, parameters
390 + $(\mat{W},\vect{a},\vect{b})$, learning rate $\eta$,
391 + Gibbs steps $k$
392 +\For{$n = 1,\dots,m$}
393 + \State $\vect{v}^{(0)} \gets \vect{v}_n$
394 + \State $\hat{\vect{h}}^{(0)} \gets
395 + \sigma\bigl(\vect{b} + \mat{W}\transp \vect{v}^{(0)}\bigr)$
396 + \Comment{positive phase, \eqref{eq:en-rbm-condh}}
397 + \For{$t = 0,\dots,k-1$}
398 + \State sample $\vect{h}^{(t)} \sim
399 + p(\vect{h}\mid \vect{v}^{(t)})$
400 + \Comment{block Gibbs step, \eqref{eq:en-rbm-condh}}
401 + \State sample $\vect{v}^{(t+1)} \sim
402 + p(\vect{v}\mid \vect{h}^{(t)})$
403 + \Comment{block Gibbs step, \eqref{eq:en-rbm-condv}}
404 + \EndFor
405 + \State $\hat{\vect{h}}^{(k)} \gets
406 + \sigma\bigl(\vect{b} + \mat{W}\transp \vect{v}^{(k)}\bigr)$
407 + \State $\Delta\mat{W}_n \gets
408 + \vect{v}^{(0)}\hat{\vect{h}}^{(0)\top}
409 + - \vect{v}^{(k)}\hat{\vect{h}}^{(k)\top}$
410 + \Comment{contrast \eqref{eq:en-cdk}}
411 + \State $\Delta\vect{a}_n \gets \vect{v}^{(0)} - \vect{v}^{(k)}$;
412 + \quad
413 + $\Delta\vect{b}_n \gets
414 + \hat{\vect{h}}^{(0)} - \hat{\vect{h}}^{(k)}$
415 +\EndFor
416 +\State $\mat{W} \gets \mat{W} + \frac{\eta}{m}\sum_n \Delta\mat{W}_n$;
417 + \quad
418 + $\vect{a} \gets \vect{a} + \frac{\eta}{m}\sum_n \Delta\vect{a}_n$;
419 + \quad
420 + $\vect{b} \gets \vect{b} + \frac{\eta}{m}\sum_n \Delta\vect{b}_n$
421 +\end{algorithmic}
422 +\end{algorithm}
423 +
424 +\begin{figure}[htbp]
425 + \centering
426 + \begin{tikzpicture}[node distance=6mm and 13mm]
427 + \node[blocinput, minimum width=1.35cm] (v0) {$\vect{v}^{(0)}$};
428 + \node[blochidden, minimum width=1.35cm, right=of v0] (h0)
429 + {$\vect{h}^{(0)}$};
430 + \node[blocinput, minimum width=1.35cm, right=of h0] (v1)
431 + {$\vect{v}^{(1)}$};
432 + \node[blochidden, minimum width=1.35cm, right=9mm of v1] (h1)
433 + {$\vect{h}^{(1)}$};
434 + \node[right=5mm of h1] (dots) {$\cdots$};
435 + \node[blocinput, minimum width=1.35cm, right=5mm of dots] (vk)
436 + {$\vect{v}^{(k)}$};
437 + \node[blochidden, minimum width=1.35cm, right=of vk] (hk)
438 + {$\vect{h}^{(k)}$};
439 + \draw[fleche] (v0) -- node[etiquette, above]
440 + {$p(\vect{h}\!\mid\!\vect{v})$} (h0);
441 + \draw[fleche] (h0) -- node[etiquette, above]
442 + {$p(\vect{v}\!\mid\!\vect{h})$} (v1);
443 + \draw[fleche] (v1) -- (h1);
444 + \draw[fleche] (h1) -- (dots);
445 + \draw[fleche] (dots) -- (vk);
446 + \draw[fleche] (vk) -- node[etiquette, above]
447 + {$p(\vect{h}\!\mid\!\vect{v})$} (hk);
448 + \node[etiquette, below=2.5mm of v0] {data (clamped)};
449 + \node[etiquette, below=2.5mm of hk] {negative phase};
450 + \draw[flechep, coutput]
451 + ($(v0.south)+(0,-9mm)$) -- node[etiquette, below]
452 + {contrast
453 + $\langle v_i h_j\rangle_0 - \langle v_i h_j\rangle_k$
454 + \ \eqref{eq:en-cdk}}
455 + ($(hk.south)+(0,-9mm)$);
456 + \end{tikzpicture}
457 + \caption{The CD-$k$ Gibbs chain \eqref{eq:en-gibbs-chain}
458 + \cite{hinton2002}. The chain starts at the data, alternates the
459 + factorized conditionals
460 + \eqref{eq:en-rbm-condh}--\eqref{eq:en-rbm-condv} for $k$ block
461 + steps, and the weight update contrasts correlations at the two ends
462 + of the chain.}
463 + \label{fig:en-cdk}
464 +\end{figure}
465 +
466 +\section{Deep Belief Networks}
467 +
468 +A single RBM learns one layer of features. Deep belief networks
469 +\cite{hinton2006} stack RBMs: train an RBM on the data, freeze it, feed
470 +its hidden activations as ``data'' to a second RBM, and repeat. The
471 +resulting generative model is a hybrid — the top two layers keep an
472 +undirected RBM joint, while the lower layers become a directed belief
473 +network:
474 +\begin{equation}
475 + p\bigl(\vect{v}, \vect{h}^{(1)}, \dots, \vect{h}^{(L)}\bigr)
476 + \;=\;
477 + p\bigl(\vect{h}^{(L-1)}, \vect{h}^{(L)}\bigr)
478 + \prod_{\ell=L-2}^{0}
479 + p\bigl(\vect{h}^{(\ell)} \mid \vect{h}^{(\ell+1)}\bigr),
480 + \label{eq:en-dbn-joint}
481 +\end{equation}
482 +with $\vect{h}^{(0)} \equiv \vect{v}$. Greediness is not a mere
483 +heuristic: each newly stacked RBM, initialized as the transpose of the
484 +one below, starts from an equivalent model and can only improve a
485 +variational lower bound on the data log-likelihood,
486 +\begin{equation}
487 + \log p(\vect{v})
488 + \;\ge\;
489 + \E_{q(\vect{h}^{(1)}\mid\vect{v})}
490 + \Bigl[\log p\bigl(\vect{v}\mid\vect{h}^{(1)}\bigr)
491 + + \log p\bigl(\vect{h}^{(1)}\bigr)\Bigr]
492 + \;+\;
493 + \mathcal{H}\bigl[q(\vect{h}^{(1)}\mid\vect{v})\bigr],
494 + \label{eq:en-dbn-bound}
495 +\end{equation}
496 +where $q$ is the first RBM's posterior, $\mathcal{H}[\cdot]$ denotes the
497 +entropy, and training the upper stack improves the prior term
498 +$\log p(\vect{h}^{(1)})$. After pretraining, the stack is optionally
499 +\emph{fine-tuned} — generatively by the wake--sleep procedure, or
500 +discriminatively by appending a classifier head and running
501 +backpropagation through the unrolled weights
502 +(Algorithm~\ref{alg:en-dbn}, Figure~\ref{fig:en-dbn}).
503 +
504 +\begin{algorithm}[htbp]
505 +\caption{Greedy layer-wise pretraining of a deep belief network}
506 +\label{alg:en-dbn}
507 +\begin{algorithmic}[1]
508 +\Require dataset $\mathcal{D} = \{\vect{v}_n\}$, layer sizes
509 + $n_1,\dots,n_L$, Gibbs steps $k$
510 +\State $\mathcal{D}^{(0)} \gets \mathcal{D}$
511 +\For{$\ell = 1,\dots,L$}
512 + \State train RBM$_\ell$ with parameters $(\mat{W}^{(\ell)},
513 + \vect{a}^{(\ell)}, \vect{b}^{(\ell)})$ on
514 + $\mathcal{D}^{(\ell-1)}$ by CD-$k$
515 + (Algorithm~\ref{alg:en-cdk})
516 + \State $\mathcal{D}^{(\ell)} \gets
517 + \bigl\{\, \sigma\bigl(\vect{b}^{(\ell)} +
518 + \mat{W}^{(\ell)\top}\vect{x}\bigr)
519 + \;:\; \vect{x} \in \mathcal{D}^{(\ell-1)} \bigr\}$
520 + \Comment{propagate features upward}
521 +\EndFor
522 +\State \Return stack $\{\mat{W}^{(\ell)}\}_{\ell=1}^{L}$; optionally
523 + fine-tune (wake--sleep, or backpropagation with a supervised
524 + head)
525 +\end{algorithmic}
526 +\end{algorithm}
527 +
528 +\begin{figure}[htbp]
529 + \centering
530 + \begin{tikzpicture}[node distance=9mm]
531 + \node[blocinput, minimum width=3.4cm] (v) {$\vect{v}$};
532 + \node[blochidden, minimum width=3.4cm, above=of v] (h1)
533 + {$\vect{h}^{(1)}$};
534 + \node[blochidden, minimum width=3.4cm, above=of h1] (h2)
535 + {$\vect{h}^{(2)}$};
536 + \node[mem, minimum width=3.4cm, above=of h2] (h3)
537 + {$\vect{h}^{(3)}$};
538 + % upward recognition / pretraining arrows
539 + \draw[fleche, cgate!70!black]
540 + ([xshift=-9mm]v.north) -- ([xshift=-9mm]h1.south)
541 + node[etiquette, midway, left=1.5mm] {$\mat{W}^{(1)\top}$};
542 + \draw[fleche, cgate!70!black]
543 + ([xshift=-9mm]h1.north) -- ([xshift=-9mm]h2.south)
544 + node[etiquette, midway, left=1.5mm] {$\mat{W}^{(2)\top}$};
545 + \draw[fleche, cgate!70!black]
546 + ([xshift=-9mm]h2.north) -- ([xshift=-9mm]h3.south)
547 + node[etiquette, midway, left=1.5mm] {$\mat{W}^{(3)\top}$};
548 + % downward generative arrows
549 + \draw[flechep, cmem!80!black]
550 + ([xshift=9mm]h3.south) -- ([xshift=9mm]h2.north)
551 + node[etiquette, midway, right=1.5mm] {$\mat{W}^{(3)}$};
552 + \draw[flechep, cmem!80!black]
553 + ([xshift=9mm]h2.south) -- ([xshift=9mm]h1.north)
554 + node[etiquette, midway, right=1.5mm] {$\mat{W}^{(2)}$};
555 + \draw[flechep, cmem!80!black]
556 + ([xshift=9mm]h1.south) -- ([xshift=9mm]v.north)
557 + node[etiquette, midway, right=1.5mm] {$\mat{W}^{(1)}$};
558 + % RBM braces on the left
559 + \draw[decorate, decoration={brace, amplitude=5pt}, black!60]
560 + ($(h1.west)+(-1.65,0.12)$) -- ($(v.west)+(-1.65,-0.12)$)
561 + node[etiquette, midway, left=7pt] {RBM 1};
562 + \draw[decorate, decoration={brace, amplitude=5pt}, black!60]
563 + ($(h2.west)+(-2.5,0.12)$) -- ($(h1.west)+(-2.5,-0.12)$)
564 + node[etiquette, midway, left=7pt] {RBM 2};
565 + \draw[decorate, decoration={brace, amplitude=5pt}, black!60]
566 + ($(h3.west)+(-1.65,0.12)$) -- ($(h2.west)+(-1.65,-0.12)$)
567 + node[etiquette, midway, left=7pt] {RBM 3};
568 + % legend
569 + \node[etiquette, align=left, anchor=west]
570 + at ($(v.east)+(0.75,0.7)$)
571 + {\textcolor{cgate!70!black}{$\longrightarrow$}\ pretraining
572 + (recognition)\\[1pt]
573 + \textcolor{cmem!80!black}{$\dashrightarrow$}\ generation};
574 + % top RBM annotation
575 + \node[etiquette, above=2.5mm of h3]
576 + {top pair: undirected joint
577 + $p(\vect{h}^{(2)},\vect{h}^{(3)})$};
578 + \end{tikzpicture}
579 + \caption{A three-layer deep belief network \cite{hinton2006}. Each
580 + RBM (braces) is trained greedily on the features of the layer below
581 + (Algorithm~\ref{alg:en-dbn}); the trained stack generates by
582 + sampling the top RBM and propagating down the dashed directed
583 + connections, as in \eqref{eq:en-dbn-joint}.}
584 + \label{fig:en-dbn}
585 +\end{figure}
586 +
587 +\begin{remark}[Historical impact]\label{rem:en-history}
588 +Greedy DBN pretraining was the first broadly reliable method for
589 +initializing deep networks, at a time when random initialization plus
590 +backpropagation stalled. Modern practice — rectified activations,
591 +normalization, residual connections and large datasets — later made
592 +unsupervised pretraining unnecessary for most supervised tasks, but the
593 +energy-based view survives: contrastive objectives, score matching, and
594 +the attention--Hopfield correspondence of
595 +Remark~\ref{rem:en-attention} are all its descendants.
596 +\end{remark}
added chapters/10-generative.tex +552 −0
@@ -0,0 +1,552 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 7 : Generative Models (chapters/07-generative.tex)
6 +% ============================================================================
7 +\chapter{Generative Models}\label{chap:generative}
8 +
9 +The architectures of the preceding chapters were, for the most part,
10 +\emph{discriminative}: they learn a mapping from an input $\vect{x}$ to a
11 +target $\vect{y}$. Generative models pursue a more ambitious goal — to learn
12 +a representation of the data distribution $p_{\mathrm{data}}(\vect{x})$
13 +itself, so that new samples can be drawn from it. The major families differ
14 +in \emph{how} they represent this density: approximately, through a
15 +variational lower bound (variational autoencoders); implicitly, through a
16 +sampling procedure trained adversarially (generative adversarial networks);
17 +or exactly, through the chain-rule factorization (autoregressive models).
18 +This chapter presents each family with its objective function, its
19 +gradients, and its computational structure, and closes with the iterative
20 +denoising paradigm — diffusion models — that dominates image synthesis
21 +today.
22 +
23 +\begin{definition}[Generative model]\label{def:gen-model}
24 +A generative model is a parametric family $p_\theta(\vect{x})$ together
25 +with a sampling procedure, trained so that $p_\theta \approx
26 +p_{\mathrm{data}}$, where $p_{\mathrm{data}}$ is the (unknown) distribution
27 +from which the training set $\{\vect{x}_1,\dots,\vect{x}_N\}$ was drawn.
28 +\end{definition}
29 +
30 +% ============================================================================
31 +\section{Autoencoders}\label{sec:gen-ae}
32 +
33 +The autoencoder is the conceptual ancestor of the modern deep generative
34 +model. It consists of an \emph{encoder} $f_{\mathrm{enc}} \colon \R^{d}
35 +\to \R^{m}$, which maps an input to a low-dimensional \emph{latent code},
36 +and a \emph{decoder} $f_{\mathrm{dec}} \colon \R^{m} \to \R^{d}$, which
37 +attempts to reconstruct the input from that code:
38 +\begin{equation}
39 + \vect{z} = f_{\mathrm{enc}}(\vect{x}),
40 + \qquad
41 + \hat{\vect{x}} = f_{\mathrm{dec}}(\vect{z}),
42 + \qquad m \ll d .
43 + \label{eq:gen-ae}
44 +\end{equation}
45 +Both maps are neural networks — typically mirror-image MLPs or
46 +convolutional networks — trained jointly to minimize the
47 +\emph{reconstruction loss}
48 +\begin{equation}
49 + \Loss_{\mathrm{AE}}
50 + = \frac{1}{N}\sum_{i=1}^{N}
51 + \bigl\lVert \vect{x}_i -
52 + f_{\mathrm{dec}}\bigl(f_{\mathrm{enc}}(\vect{x}_i)\bigr)
53 + \bigr\rVert_2^2 ,
54 + \label{eq:gen-ae-loss}
55 +\end{equation}
56 +or a cross-entropy loss when the inputs are modeled as Bernoulli variables
57 +(e.g.\ binarized pixels). Because the bottleneck dimension $m$ in
58 +\eqref{eq:gen-ae} is much smaller than the input dimension $d$, the network
59 +cannot learn the identity map; it is forced to discover a compressed
60 +representation that preserves exactly the information needed to reconstruct
61 +the data. Figure~\ref{fig:gen-ae} shows the characteristic hourglass shape.
62 +
63 +\begin{figure}[ht]
64 + \centering
65 + \begin{tikzpicture}[scale=0.92, transform shape]
66 + % input layer: 6 neurons
67 + \foreach \i in {1,...,6}
68 + \node[ninput] (x\i) at (0, 3.0-\i) {};
69 + % encoder hidden: 4 neurons
70 + \foreach \i in {1,...,4}
71 + \node[nhidden] (e\i) at (2.0, 2.0-\i) {};
72 + % latent: 2 neurons (memory color)
73 + \foreach \i in {1,2}
74 + \node[neuron, fill=cmem!25, draw=cmem!70!black]
75 + (z\i) at (4.0, 1.0-\i) {};
76 + % decoder hidden: 4 neurons
77 + \foreach \i in {1,...,4}
78 + \node[nhidden] (d\i) at (6.0, 2.0-\i) {};
79 + % output layer: 6 neurons
80 + \foreach \i in {1,...,6}
81 + \node[noutput] (y\i) at (8.0, 3.0-\i) {};
82 + % connections
83 + \foreach \i in {1,...,6} \foreach \j in {1,...,4}
84 + \draw[black!30, semithick] (x\i) -- (e\j);
85 + \foreach \i in {1,...,4} \foreach \j in {1,2}
86 + \draw[black!30, semithick] (e\i) -- (z\j);
87 + \foreach \i in {1,2} \foreach \j in {1,...,4}
88 + \draw[black!30, semithick] (z\i) -- (d\j);
89 + \foreach \i in {1,...,4} \foreach \j in {1,...,6}
90 + \draw[black!30, semithick] (d\i) -- (y\j);
91 + % column labels
92 + \node[etiquette] at (0, 2.6) {$\vect{x}$};
93 + \node[etiquette] at (4.0, 0.6) {$\vect{z}$};
94 + \node[etiquette] at (8.0, 2.6) {$\hat{\vect{x}}$};
95 + % braces
96 + \draw[decorate, decoration={brace, mirror, amplitude=5pt}, thick]
97 + (-0.4,-3.6) -- (3.4,-3.6)
98 + node[midway, below=7pt, font=\small]
99 + {encoder $f_{\mathrm{enc}}$};
100 + \draw[decorate, decoration={brace, mirror, amplitude=5pt}, thick]
101 + (4.6,-3.6) -- (8.4,-3.6)
102 + node[midway, below=7pt, font=\small]
103 + {decoder $f_{\mathrm{dec}}$};
104 + \node[font=\small] at (4.0, -2.1) {latent code};
105 + \end{tikzpicture}
106 + \caption{The autoencoder: an hourglass architecture. The encoder
107 + compresses the input $\vect{x}$ into a latent code $\vect{z}$ (violet)
108 + of much smaller dimension; the decoder reconstructs
109 + $\hat{\vect{x}}$ from $\vect{z}$. Training minimizes the reconstruction
110 + loss~\eqref{eq:gen-ae-loss}.}
111 + \label{fig:gen-ae}
112 +\end{figure}
113 +
114 +\begin{remark}\label{rem:gen-ae-not-generative}
115 +A plain autoencoder is \emph{not} yet a generative model in the sense of
116 +Definition~\ref{def:gen-model}: nothing constrains the geometry of the
117 +latent space, so decoding an arbitrary $\vect{z}$ generally does not
118 +produce a realistic sample. Imposing a probabilistic structure on the
119 +latent space is precisely the contribution of the variational autoencoder.
120 +\end{remark}
121 +
122 +% ============================================================================
123 +\section{Variational Autoencoders}\label{sec:gen-vae}
124 +
125 +The variational autoencoder (VAE) of Kingma and
126 +Welling~\cite{kingma2014vae} turns the autoencoder into a proper
127 +latent-variable model. It posits a prior $p(\vect{z}) =
128 +\mathcal{N}(\vect{0}, \mat{I})$ and a decoder that defines a conditional
129 +likelihood $p_\theta(\vect{x} \mid \vect{z})$, so that the model density is
130 +\begin{equation}
131 + p_\theta(\vect{x})
132 + = \int p_\theta(\vect{x} \mid \vect{z})\, p(\vect{z})\, d\vect{z} .
133 + \label{eq:gen-lvm}
134 +\end{equation}
135 +The integral in \eqref{eq:gen-lvm} is intractable, so the VAE introduces an
136 +\emph{approximate posterior} $q_\phi(\vect{z} \mid \vect{x}) =
137 +\mathcal{N}\bigl(\vect{\mu}_\phi(\vect{x}),
138 +\operatorname{diag}(\vect{\sigma}_\phi^2(\vect{x}))\bigr)$ — the
139 +probabilistic encoder — and maximizes a lower bound on the log-likelihood.
140 +
141 +\subsection{The evidence lower bound}
142 +
143 +For any $q_\phi$, the log-likelihood decomposes exactly as
144 +\begin{equation}
145 + \log p_\theta(\vect{x})
146 + = \Loss_{\mathrm{ELBO}}(\theta, \phi; \vect{x})
147 + + \KL\bigl(q_\phi(\vect{z} \mid \vect{x}) \,\big\Vert\,
148 + p_\theta(\vect{z} \mid \vect{x})\bigr) ,
149 + \label{eq:gen-elbo-decomp}
150 +\end{equation}
151 +and since the Kullback--Leibler divergence is non-negative, the first term
152 +— the \emph{evidence lower bound} (ELBO) — bounds $\log p_\theta(\vect{x})$
153 +from below:
154 +\begin{equation}
155 + \Loss_{\mathrm{ELBO}}(\theta, \phi; \vect{x})
156 + = \underbrace{\E_{q_\phi(\vect{z} \mid \vect{x})}
157 + \bigl[\log p_\theta(\vect{x} \mid \vect{z})\bigr]}_{\text{reconstruction}}
158 + \;-\;
159 + \underbrace{\KL\bigl(q_\phi(\vect{z} \mid \vect{x})
160 + \,\big\Vert\, p(\vect{z})\bigr)}_{\text{regularization}} .
161 + \label{eq:gen-elbo}
162 +\end{equation}
163 +The two terms of \eqref{eq:gen-elbo} recover, and generalize, the two
164 +ingredients of the plain autoencoder: the first is a (probabilistic)
165 +reconstruction objective, while the second pulls the encoder's output
166 +distribution toward the prior, organizing the latent space so that samples
167 +$\vect{z} \sim \mathcal{N}(\vect{0},\mat{I})$ decode to realistic data.
168 +By \eqref{eq:gen-elbo-decomp}, maximizing the ELBO simultaneously raises
169 +the likelihood and tightens the posterior approximation.
170 +
171 +For diagonal Gaussians the regularization term of \eqref{eq:gen-elbo} has
172 +a closed form. With $q_\phi = \mathcal{N}(\vect{\mu},
173 +\operatorname{diag}(\vect{\sigma}^2))$ and $p = \mathcal{N}(\vect{0},
174 +\mat{I})$ in $m$ dimensions,
175 +\begin{equation}
176 + \KL\bigl(q_\phi(\vect{z} \mid \vect{x}) \,\big\Vert\,
177 + \mathcal{N}(\vect{0},\mat{I})\bigr)
178 + = -\frac{1}{2}\sum_{j=1}^{m}
179 + \bigl(1 + \log \sigma_j^2 - \mu_j^2 - \sigma_j^2\bigr) ,
180 + \label{eq:gen-kl}
181 +\end{equation}
182 +so this term costs nothing to estimate: it is computed analytically from
183 +the encoder outputs.
184 +
185 +\subsection{The reparameterization trick}
186 +
187 +One obstacle remains: the reconstruction term of \eqref{eq:gen-elbo} is an
188 +expectation over $\vect{z} \sim q_\phi(\vect{z} \mid \vect{x})$, and the
189 +sampling operation blocks the gradient with respect to $\phi$. The
190 +\emph{reparameterization trick}~\cite{kingma2014vae} rewrites the sample as
191 +a deterministic, differentiable function of the parameters plus exogenous
192 +noise:
193 +\begin{equation}
194 + \vect{z}
195 + = \vect{\mu}_\phi(\vect{x})
196 + + \vect{\sigma}_\phi(\vect{x}) \odot \vect{\varepsilon},
197 + \qquad
198 + \vect{\varepsilon} \sim \mathcal{N}(\vect{0}, \mat{I}) .
199 + \label{eq:gen-reparam}
200 +\end{equation}
201 +The randomness now enters only through $\vect{\varepsilon}$, which does not
202 +depend on $\phi$; gradients flow through $\vect{\mu}_\phi$ and
203 +$\vect{\sigma}_\phi$ by the ordinary chain rule, yielding an unbiased,
204 +low-variance pathwise estimator of $\nabla_\phi \Loss_{\mathrm{ELBO}}$.
205 +Figure~\ref{fig:gen-vae} traces the full computation.
206 +
207 +\begin{figure}[ht]
208 + \centering
209 + \begin{tikzpicture}[scale=0.88, transform shape]
210 + \node[blocinput, minimum width=1cm] (x) at (0,0) {$\vect{x}$};
211 + \node[blochidden, minimum width=1.9cm, minimum height=1.9cm]
212 + (enc) at (2.3,0)
213 + {Encoder\\ $q_\phi(\vect{z}\mid\vect{x})$};
214 + \node[bloc, minimum width=1.15cm] (mu) at (4.9, 1.0) {$\vect{\mu}$};
215 + \node[bloc, minimum width=1.15cm] (sg) at (4.9,-1.0) {$\log\vect{\sigma}^2$};
216 + \node[bloc, minimum width=1.6cm, densely dashed]
217 + (eps) at (4.9,-2.7)
218 + {$\vect{\varepsilon}\sim\mathcal{N}(\vect{0},\mat{I})$};
219 + \node[op] (odot) at (7.0,-1.0) {$\odot$};
220 + \node[op] (plus) at (8.1, 0) {$+$};
221 + \node[mem, minimum width=1cm] (z) at (9.4,0) {$\vect{z}$};
222 + \node[blochidden, minimum width=1.9cm, minimum height=1.9cm]
223 + (dec) at (11.7,0)
224 + {Decoder\\ $p_\theta(\vect{x}\mid\vect{z})$};
225 + \node[blocoutput, minimum width=1cm] (xh) at (14.0,0) {$\hat{\vect{x}}$};
226 + % arrows
227 + \draw[fleche] (x) -- (enc);
228 + \draw[fleche] ([yshift=5.5mm]enc.east) -- ++(0.35,0) |- (mu.west);
229 + \draw[fleche] ([yshift=-5.5mm]enc.east) -- ++(0.35,0) |- (sg.west);
230 + \draw[fleche] (sg) -- node[etiquette, above]{$\exp(\cdot/2)$} (odot);
231 + \draw[fleche] (eps.east) -| (odot);
232 + \draw[fleche] (odot) -- (plus);
233 + \draw[fleche] (mu.east) -| (plus);
234 + \draw[fleche] (plus) -- (z);
235 + \draw[fleche] (z) -- (dec);
236 + \draw[fleche] (dec) -- (xh);
237 + \end{tikzpicture}
238 + \caption{The variational autoencoder with the reparameterization
239 + trick~\eqref{eq:gen-reparam}. The encoder outputs the parameters
240 + $(\vect{\mu}, \log\vect{\sigma}^2)$ of the approximate posterior; the
241 + latent sample is assembled as $\vect{z} = \vect{\mu} + \vect{\sigma}
242 + \odot \vect{\varepsilon}$, so that gradients flow through
243 + $\vect{\mu}$ and $\vect{\sigma}$ while the stochasticity is confined
244 + to the exogenous noise $\vect{\varepsilon}$ (dashed).}
245 + \label{fig:gen-vae}
246 +\end{figure}
247 +
248 +With the reparameterization in place, a VAE training step is ordinary
249 +stochastic gradient ascent on the ELBO, summarized in
250 +Algorithm~\ref{alg:gen-vae}: encode, sample through
251 +\eqref{eq:gen-reparam}, decode, and differentiate the two terms of
252 +\eqref{eq:gen-elbo} jointly with respect to both parameter sets.
253 +
254 +\begin{algorithm}[htbp]
255 + \caption{VAE training step (stochastic gradient ascent on the ELBO)}
256 + \label{alg:gen-vae}
257 + \begin{algorithmic}[1]
258 + \Require minibatch $\{\vect{x}^{(i)}\}_{i=1}^{m}$, encoder parameters
259 + $\phi$, decoder parameters $\theta$, learning rate $\eta$
260 + \For{$i = 1, \dots, m$}
261 + \State $\bigl(\vect{\mu}^{(i)}, \log\vect{\sigma}^{2(i)}\bigr)
262 + \gets \mathrm{Encoder}_\phi\!\left(\vect{x}^{(i)}\right)$
263 + \State sample $\vect{\varepsilon}^{(i)} \sim
264 + \mathcal{N}(\vect{0}, \mat{I})$;\quad
265 + $\vect{z}^{(i)} \gets \vect{\mu}^{(i)}
266 + + \vect{\sigma}^{(i)} \odot \vect{\varepsilon}^{(i)}$
267 + \Comment{reparameterization \eqref{eq:gen-reparam}}
268 + \State $\hat{\vect{x}}^{(i)} \gets
269 + \mathrm{Decoder}_\theta\!\left(\vect{z}^{(i)}\right)$
270 + \State $\Loss^{(i)} \gets
271 + \log p_\theta\!\left(\vect{x}^{(i)} \mid \vect{z}^{(i)}\right)
272 + - \KL\!\left( q_\phi(\vect{z} \mid \vect{x}^{(i)})
273 + \,\Vert\, p(\vect{z}) \right)$
274 + \Comment{ELBO \eqref{eq:gen-elbo}, KL \eqref{eq:gen-kl}}
275 + \EndFor
276 + \State $\Loss \gets \frac{1}{m}\sum_{i=1}^{m} \Loss^{(i)}$
277 + \State $\theta \gets \theta + \eta\, \nabla_\theta \Loss$;\quad
278 + $\phi \gets \phi + \eta\, \nabla_\phi \Loss$
279 + \Comment{ascent; gradients by backpropagation}
280 + \end{algorithmic}
281 +\end{algorithm}
282 +
283 +\begin{remark}\label{rem:gen-vae-blur}
284 +Because the reconstruction term of \eqref{eq:gen-elbo} is typically a
285 +Gaussian (squared-error) likelihood averaged over the posterior, VAE
286 +samples tend to be slightly blurry: the decoder learns to output the
287 +conditional \emph{mean} of all plausible reconstructions. This is the
288 +characteristic failure mode recorded in
289 +Table~\ref{tab:gen-comparison}.
290 +\end{remark}
291 +
292 +% ============================================================================
293 +\section{Generative Adversarial Networks}\label{sec:gen-gan}
294 +
295 +Generative adversarial networks (GANs), introduced by Goodfellow et
296 +al.~\cite{goodfellow2014gan}, abandon explicit densities altogether. A
297 +\emph{generator} $G$ maps noise $\vect{z} \sim p_{\vect{z}} =
298 +\mathcal{N}(\vect{0},\mat{I})$ to a sample $G(\vect{z})$, and a
299 +\emph{discriminator} $D(\vect{x}) \in [0,1]$ estimates the probability
300 +that its input came from the data rather than from $G$. The two networks
301 +play a two-player \emph{minimax game}:
302 +\begin{equation}
303 + \min_G \max_D \; V(D, G)
304 + = \E_{\vect{x} \sim p_{\mathrm{data}}}\bigl[\log D(\vect{x})\bigr]
305 + + \E_{\vect{z} \sim p_{\vect{z}}}
306 + \bigl[\log\bigl(1 - D(G(\vect{z}))\bigr)\bigr] .
307 + \label{eq:gen-minimax}
308 +\end{equation}
309 +The discriminator is trained to tell real from fake; the generator is
310 +trained to fool it. Figure~\ref{fig:gen-gan} shows the adversarial
311 +arrangement: crucially, the generator never sees the data directly — its
312 +only training signal is the gradient that flows \emph{through} the
313 +discriminator.
314 +
315 +\begin{figure}[ht]
316 + \centering
317 + \begin{tikzpicture}[scale=0.92, transform shape]
318 + \node[bloc, minimum width=1.9cm] (z) at (0,1.0)
319 + {$\vect{z}\sim\mathcal{N}(\vect{0},\mat{I})$};
320 + \node[blochidden, minimum width=2.2cm, minimum height=1.1cm]
321 + (G) at (3.2,1.0) {Generator\\ $G$};
322 + \node[bloc, minimum width=2.3cm] (xf) at (6.5,1.0)
323 + {fake $\tilde{\vect{x}} = G(\vect{z})$};
324 + \node[blocinput, minimum width=2.3cm] (xr) at (6.5,-1.0)
325 + {real $\vect{x}\sim p_{\mathrm{data}}$};
326 + \node[blochidden, minimum width=2.4cm, minimum height=1.1cm]
327 + (D) at (10.0,0) {Discriminator\\ $D$};
328 + \node[blocoutput, minimum width=1.7cm] (out) at (13.2,0)
329 + {real / fake};
330 + \draw[fleche] (z) -- (G);
331 + \draw[fleche] (G) -- (xf);
332 + \draw[fleche] (xf.east) -- ++(0.55,0) |- ([yshift=3mm]D.west);
333 + \draw[fleche] (xr.east) -- ++(0.55,0) |- ([yshift=-3mm]D.west);
334 + \draw[fleche] (D) -- (out);
335 + % adversarial gradient (dashed), routed above the fake-sample block
336 + \draw[flechep, draw=coutput!80!black]
337 + (D.north) -- ++(0,1.45) -| (G.north)
338 + node[pos=0.25, above, etiquette, text=coutput!80!black]
339 + {adversarial gradient $\nabla_{\theta_G} \Loss$};
340 + \end{tikzpicture}
341 + \caption{The generative adversarial network. The generator maps noise
342 + $\vect{z}$ to a fake sample $\tilde{\vect{x}}$; the discriminator
343 + receives both real and fake samples and outputs the probability that
344 + its input is real. The generator's only learning signal is the
345 + adversarial gradient (dashed, red) backpropagated through the
346 + discriminator, per the minimax objective~\eqref{eq:gen-minimax}.}
347 + \label{fig:gen-gan}
348 +\end{figure}
349 +
350 +\subsection{The optimal discriminator and the Jensen--Shannon divergence}
351 +
352 +The minimax game \eqref{eq:gen-minimax} has a precise
353 +distribution-matching interpretation.
354 +
355 +\begin{theorem}[Optimal discriminator]\label{thm:gen-dstar}
356 +Let $p_g$ denote the distribution of $G(\vect{z})$ for fixed $G$. The
357 +discriminator maximizing $V(D,G)$ in \eqref{eq:gen-minimax} is
358 +\begin{equation}
359 + D^{*}(\vect{x})
360 + = \frac{p_{\mathrm{data}}(\vect{x})}
361 + {p_{\mathrm{data}}(\vect{x}) + p_g(\vect{x})} ,
362 + \label{eq:gen-dstar}
363 +\end{equation}
364 +and substituting $D^{*}$ into $V$ gives the generator's effective
365 +objective
366 +\begin{equation}
367 + C(G) = \max_D V(D,G)
368 + = -\log 4
369 + + 2\,\mathrm{JSD}\bigl(p_{\mathrm{data}} \,\big\Vert\, p_g\bigr) ,
370 + \label{eq:gen-jsd}
371 +\end{equation}
372 +where $\mathrm{JSD}$ is the Jensen--Shannon divergence. Hence $C(G)$ is
373 +minimized if and only if $p_g = p_{\mathrm{data}}$, where $D^{*} \equiv
374 +\tfrac{1}{2}$ and $C(G) = -\log 4$.
375 +\end{theorem}
376 +
377 +\begin{proof}[Proof sketch]
378 +For fixed $G$, $V(D,G) = \int \bigl[p_{\mathrm{data}}(\vect{x}) \log
379 +D(\vect{x}) + p_g(\vect{x}) \log(1 - D(\vect{x}))\bigr] d\vect{x}$;
380 +pointwise maximization of $a \log t + b \log(1-t)$ over $t \in (0,1)$
381 +yields $t^{*} = a/(a+b)$, which is \eqref{eq:gen-dstar}. Substituting back
382 +and completing each term to a KL divergence against the mixture
383 +$(p_{\mathrm{data}} + p_g)/2$ gives \eqref{eq:gen-jsd};
384 +see~\cite{goodfellow2014gan}.
385 +\end{proof}
386 +
387 +\subsection{Training in practice: non-saturating and Wasserstein losses}
388 +
389 +Early in training, $D$ rejects fakes easily, $D(G(\vect{z})) \approx 0$,
390 +and the generator's term $\log(1 - D(G(\vect{z})))$ in
391 +\eqref{eq:gen-minimax} saturates — its gradient vanishes exactly when the
392 +generator most needs guidance. The standard remedy, proposed already
393 +in~\cite{goodfellow2014gan}, is the \emph{non-saturating} generator loss:
394 +instead of minimizing $\E[\log(1 - D(G(\vect{z})))]$, the generator
395 +maximizes
396 +\begin{equation}
397 + \Loss_G^{\mathrm{NS}}
398 + = \E_{\vect{z} \sim p_{\vect{z}}}\bigl[\log D(G(\vect{z}))\bigr] ,
399 + \label{eq:gen-nonsat}
400 +\end{equation}
401 +which has the same fixed points but provides strong gradients precisely
402 +when the discriminator is confident. The resulting estimation procedure,
403 +Algorithm~\ref{alg:gen-gan}, alternates $k$ ascent steps on the
404 +discriminator's objective with one non-saturating update of the generator.
405 +
406 +\begin{algorithm}[htbp]
407 + \caption{GAN alternating training with the non-saturating generator
408 + loss}
409 + \label{alg:gen-gan}
410 + \begin{algorithmic}[1]
411 + \Require generator $G$ (parameters $\theta_g$), discriminator $D$
412 + (parameters $\theta_d$), discriminator steps $k$, batch size $m$,
413 + learning rates $\eta_d, \eta_g$
414 + \While{not converged}
415 + \For{$j = 1, \dots, k$} \Comment{discriminator updates}
416 + \State sample $\{\vect{x}^{(1)}, \dots, \vect{x}^{(m)}\}$ from the
417 + data, $\{\vect{z}^{(1)}, \dots, \vect{z}^{(m)}\}$ from
418 + $p_{\vect{z}}$
419 + \State $\Loss_D \gets \frac{1}{m}\sum_{i=1}^{m}
420 + \Bigl[ \log D\!\left(\vect{x}^{(i)}\right)
421 + + \log\!\left(1 - D\!\left(G(\vect{z}^{(i)})\right)\right)
422 + \Bigr]$
423 + \Comment{value of \eqref{eq:gen-minimax}}
424 + \State $\theta_d \gets \theta_d + \eta_d\,
425 + \nabla_{\theta_d} \Loss_D$
426 + \Comment{gradient \emph{ascent}}
427 + \EndFor
428 + \State sample $\{\vect{z}^{(1)}, \dots, \vect{z}^{(m)}\}$ from
429 + $p_{\vect{z}}$
430 + \State $\Loss_G^{\mathrm{NS}} \gets \frac{1}{m}\sum_{i=1}^{m}
431 + \log D\!\left(G(\vect{z}^{(i)})\right)$
432 + \Comment{non-saturating loss \eqref{eq:gen-nonsat}}
433 + \State $\theta_g \gets \theta_g + \eta_g\,
434 + \nabla_{\theta_g} \Loss_G^{\mathrm{NS}}$
435 + \EndWhile
436 + \end{algorithmic}
437 +\end{algorithm}
438 +
439 +A deeper pathology is that the Jensen--Shannon divergence in
440 +\eqref{eq:gen-jsd} is poorly behaved when $p_{\mathrm{data}}$ and $p_g$
441 +have (nearly) disjoint supports — as is typical for high-dimensional data
442 +concentrated on low-dimensional manifolds — contributing to unstable
443 +dynamics and \emph{mode collapse}, in which $G$ maps many latent vectors
444 +onto a few high-scoring outputs. The Wasserstein GAN (WGAN) replaces the
445 +JSD with the Wasserstein-1 distance, which by Kantorovich--Rubinstein
446 +duality is
447 +\begin{equation}
448 + W\bigl(p_{\mathrm{data}}, p_g\bigr)
449 + = \sup_{\lVert f \rVert_{L} \le 1}\;
450 + \E_{\vect{x} \sim p_{\mathrm{data}}}\bigl[f(\vect{x})\bigr]
451 + - \E_{\vect{x} \sim p_g}\bigl[f(\vect{x})\bigr] ,
452 + \label{eq:gen-wgan}
453 +\end{equation}
454 +where the supremum ranges over 1-Lipschitz functions. A \emph{critic}
455 +network (a discriminator without the final sigmoid) approximates the
456 +supremum in \eqref{eq:gen-wgan}, with the Lipschitz constraint enforced by
457 +weight clipping or, in later refinements, a gradient penalty. Because $W$
458 +remains finite and provides usable gradients even for disjoint supports,
459 +WGAN training is markedly more stable and less prone to mode collapse.
460 +
461 +% ============================================================================
462 +\section{Autoregressive Models and a Comparison of Families}
463 +\label{sec:gen-ar}
464 +
465 +A third route to generation requires neither latent variables nor an
466 +adversary: factor the joint density exactly by the chain rule,
467 +\begin{equation}
468 + p_\theta(\vect{x})
469 + = \prod_{i=1}^{d} p_\theta\bigl(x_i \mid x_1, \dots, x_{i-1}\bigr) ,
470 + \label{eq:gen-ar}
471 +\end{equation}
472 +and train a network to model each conditional by maximum likelihood — the
473 +factorization already encountered for sequence models, now applied to
474 +arbitrary data (pixels in raster order, audio samples, discrete tokens).
475 +Training is stable and parallelizable, and the likelihood
476 +\eqref{eq:gen-ar} is exact; the price is sampling, which is inherently
477 +sequential — one dimension at a time. Table~\ref{tab:gen-comparison}
478 +summarizes the trade-offs among the three families
479 +(see also~\cite{goodfellow2016book}).
480 +
481 +\begin{table}[ht]
482 + \centering
483 + \small
484 + \begin{tabular}{@{}l l l l l@{}}
485 + \toprule
486 + Family & Training objective & Sampling & Likelihood & Typical failure \\
487 + \midrule
488 + VAE & ELBO \eqref{eq:gen-elbo}, stable
489 + & one pass, fast & lower bound & blurry samples \\
490 + GAN & minimax \eqref{eq:gen-minimax}, unstable
491 + & one pass, fast & implicit (none) & mode collapse \\
492 + Autoregressive & exact MLE \eqref{eq:gen-ar}, stable
493 + & sequential, slow & exact & slow sampling \\
494 + \bottomrule
495 + \end{tabular}
496 + \caption{Comparison of the three classical generative families. Each
497 + optimizes a different surrogate of the same goal, $p_\theta \approx
498 + p_{\mathrm{data}}$, and each pays for its strengths with a
499 + characteristic weakness.}
500 + \label{tab:gen-comparison}
501 +\end{table}
502 +
503 +% ============================================================================
504 +\section{Toward Diffusion Models}\label{sec:gen-diffusion}
505 +
506 +The current state of the art in image, audio and video synthesis belongs
507 +to a fourth family that turns generation into \emph{iterative denoising}.
508 +A fixed forward process gradually destroys the data with Gaussian noise
509 +over $T$ steps, according to a variance schedule $\beta_1, \dots,
510 +\beta_T$:
511 +\begin{equation}
512 + q(\vect{x}_t \mid \vect{x}_{t-1})
513 + = \mathcal{N}\bigl(\vect{x}_t;\;
514 + \sqrt{1 - \beta_t}\, \vect{x}_{t-1},\; \beta_t \mat{I}\bigr) .
515 + \label{eq:gen-diff-forward}
516 +\end{equation}
517 +Iterating \eqref{eq:gen-diff-forward} and writing $\alpha_t = 1 -
518 +\beta_t$, $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$, the noisy state at
519 +any timestep is available in closed form directly from the clean sample:
520 +\begin{equation}
521 + q(\vect{x}_t \mid \vect{x}_0)
522 + = \mathcal{N}\bigl(\vect{x}_t;\;
523 + \sqrt{\bar{\alpha}_t}\, \vect{x}_0,\;
524 + (1 - \bar{\alpha}_t)\, \mat{I}\bigr)
525 + \;\Longleftrightarrow\;
526 + \vect{x}_t
527 + = \sqrt{\bar{\alpha}_t}\, \vect{x}_0
528 + + \sqrt{1 - \bar{\alpha}_t}\, \vect{\varepsilon},
529 + \quad \vect{\varepsilon} \sim \mathcal{N}(\vect{0}, \mat{I}) .
530 + \label{eq:gen-diff-closed}
531 +\end{equation}
532 +As $t \to T$, $\bar{\alpha}_t \to 0$ and the data dissolves into pure
533 +noise. Generation runs the process in reverse: a learned Markov chain
534 +$p_\theta(\vect{x}_{t-1} \mid \vect{x}_t)$ starts from
535 +$\mathcal{N}(\vect{0},\mat{I})$ and removes the noise step by step, and a
536 +variational argument in the spirit of \eqref{eq:gen-elbo} reduces its
537 +training to a strikingly simple regression — predict the noise
538 +$\vect{\varepsilon}$ injected in \eqref{eq:gen-diff-closed}:
539 +\begin{equation}
540 + \Loss_{\mathrm{simple}}
541 + = \E_{t,\, \vect{x}_0,\, \vect{\varepsilon}}
542 + \Bigl[\bigl\lVert \vect{\varepsilon}
543 + - \vect{\varepsilon}_\theta(\vect{x}_t, t)
544 + \bigr\rVert_2^2\Bigr] .
545 + \label{eq:gen-diff-loss}
546 +\end{equation}
547 +Diffusion models thus combine the stable, likelihood-based training of the
548 +VAE lineage with sample quality surpassing GANs, at the cost of an
549 +iterative sampling procedure — a trade-off that a rapidly evolving
550 +literature (accelerated samplers, latent-space diffusion, flow matching,
551 +consistency models) continues to push toward few-step and even one-step
552 +generation.
added chapters/11-advanced-generative.tex +741 −0
@@ -0,0 +1,741 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 11 : Normalizing Flows, Diffusion and Autoregressive Models
6 +% (chapters/11-advanced-generative.tex)
7 +% ============================================================================
8 +\chapter{Normalizing Flows, Diffusion and Autoregressive Models}
9 +\label{chap:advgen}
10 +
11 +The previous chapter introduced generative models whose likelihood is either
12 +approximated from below (the variational autoencoder) or never written down
13 +at all (the generative adversarial network). This chapter completes the
14 +generative landscape with three families that take the opposite stance:
15 +they insist on an \emph{explicit} probabilistic account of the data.
16 +\emph{Normalizing flows} construct an invertible map between noise and data
17 +and obtain the exact likelihood from the change-of-variables formula.
18 +\emph{Diffusion models} destroy data with a fixed Markovian noising process
19 +and learn to reverse it, optimizing a variational bound that collapses to a
20 +strikingly simple denoising objective. \emph{Deep autoregressive models}
21 +factorize the joint density exactly with the chain rule and reduce
22 +generation to next-element prediction. For each family we give the defining
23 +equations, the estimation algorithm in pseudocode, and a graphical
24 +representation of the computational structure; we close with a brief look
25 +at energy-based models, which unify several of these views.
26 +
27 +% ============================================================================
28 +\section{Normalizing Flows}
29 +\label{sec:agen-flows}
30 +
31 +\begin{definition}[Normalizing flow]\label{def:agen-flow}
32 +A \emph{normalizing flow} is a diffeomorphism $T \colon \R^{D} \to \R^{D}$
33 +— an invertible, differentiable map with differentiable inverse — that
34 +transports a simple \emph{base density} $p_Z$ (typically the standard
35 +Gaussian $\mathcal{N}(\vect{0}, \mat{I})$) into a complex data density:
36 +$\vect{x} = T(\vect{z})$ with $\vect{z} \sim p_Z$.
37 +\end{definition}
38 +
39 +Because $T$ is invertible, the density of $\vect{x}$ is not merely bounded
40 +or approximated — it is \emph{known exactly}, through the
41 +change-of-variables formula:
42 +\begin{equation}
43 + \log p_X(\vect{x})
44 + = \log p_Z\!\big(T^{-1}(\vect{x})\big)
45 + + \log \left| \det \frac{\partial T^{-1}(\vect{x})}{\partial \vect{x}}
46 + \right|.
47 + \label{eq:agen-cov}
48 +\end{equation}
49 +The Jacobian determinant measures how the map locally contracts or expands
50 +volume; it is precisely the correction that keeps total probability mass
51 +equal to one. In practice a single map is too rigid, so the flow is built
52 +as a composition of $K$ simpler bijections,
53 +$T = f_K \circ f_{K-1} \circ \dots \circ f_1$, generating a chain
54 +$\vect{z}_0 \sim p_Z$, $\vect{z}_k = f_k(\vect{z}_{k-1})$,
55 +$\vect{x} = \vect{z}_K$. Log-determinants then simply add along the chain
56 +\cite{rezende2015}:
57 +\begin{equation}
58 + \log p_X(\vect{x})
59 + = \log p_Z(\vect{z}_0)
60 + - \sum_{k=1}^{K} \log \left| \det
61 + \mat{J}_{f_k}(\vect{z}_{k-1}) \right|,
62 + \qquad
63 + \mat{J}_{f_k} = \frac{\partial f_k}{\partial \vect{z}_{k-1}}.
64 + \label{eq:agen-comp}
65 +\end{equation}
66 +Training maximizes the exact log-likelihood \eqref{eq:agen-comp} by
67 +running the chain backwards (density evaluation); sampling runs it
68 +forwards from Gaussian noise. Figure~\ref{fig:agen-flow} shows the
69 +architecture: a stack of bijective blocks, each contributing one log-det
70 +term to the likelihood.
71 +
72 +\begin{figure}[htbp]
73 + \centering
74 + \begin{tikzpicture}[node distance=7mm]
75 + % --- chain of bijections ---
76 + \node[mem, minimum width=10mm] (z0) {$\vect{z}_0$};
77 + \node[bloc, minimum width=10mm, right=of z0] (f1) {$f_1$};
78 + \node[mem, minimum width=10mm, right=of f1] (z1) {$\vect{z}_1$};
79 + \node[right=5.5mm of z1] (dots) {$\cdots$};
80 + \node[mem, minimum width=13mm, right=5.5mm of dots] (zk1) {$\vect{z}_{K-1}$};
81 + \node[bloc, minimum width=10mm, right=of zk1] (fk) {$f_K$};
82 + \node[blocoutput, minimum width=10mm, right=of fk] (x) {$\vect{x}$};
83 + \draw[fleche] (z0) -- (f1);
84 + \draw[fleche] (f1) -- (z1);
85 + \draw[fleche] (z1) -- (dots);
86 + \draw[fleche] (dots) -- (zk1);
87 + \draw[fleche] (zk1) -- (fk);
88 + \draw[fleche] (fk) -- (x);
89 + % --- accumulating log-det annotations ---
90 + \node[etiquette, below=2.5mm of f1]
91 + {$-\log\big|\det \mat{J}_{f_1}\big|$};
92 + \node[etiquette, below=2.5mm of fk]
93 + {$-\log\big|\det \mat{J}_{f_K}\big|$};
94 + \node[etiquette, below=2.5mm of dots] {$\cdots$};
95 + % --- direction labels ---
96 + \draw[fleche, black!50]
97 + ($(z0.south)+(0,-1.15)$) -- ($(x.south)+(0,-1.15)$)
98 + node[midway, below, etiquette] {generation $\vect{z}_0 \to \vect{x}$};
99 + \draw[flechep, black!50]
100 + ($(x.north)+(0,1.3)$) -- ($(z0.north)+(0,1.3)$)
101 + node[midway, above, etiquette]
102 + {density evaluation $\vect{x} \to \vect{z}_0$ via $f_k^{-1}$};
103 + % --- base density curve above z0 ---
104 + \begin{scope}[shift={($(z0.north)+(0,0.28)$)}]
105 + \draw[black!40] (-0.75,0) -- (0.75,0);
106 + \draw[cinput, thick]
107 + plot[domain=-0.72:0.72, samples=41] (\x, {0.55*exp(-11*\x*\x)});
108 + \end{scope}
109 + % --- warped density curve above x ---
110 + \begin{scope}[shift={($(x.north)+(0,0.28)$)}]
111 + \draw[black!40] (-0.75,0) -- (0.75,0);
112 + \draw[coutput, thick]
113 + plot[domain=-0.72:0.72, samples=61]
114 + (\x, {0.45*exp(-26*(\x-0.33)^2) + 0.55*exp(-26*(\x+0.28)^2)});
115 + \end{scope}
116 + \end{tikzpicture}
117 + \caption{A normalizing flow as a chain of $K$ bijections. Sampling runs
118 + left to right, from the Gaussian base density (blue curve) to the data
119 + density (red curve); exact density evaluation runs right to left through
120 + the inverses $f_k^{-1}$, accumulating the log-det terms of
121 + \eqref{eq:agen-comp}. Every intermediate variable $\vect{z}_k$ keeps the
122 + full dimension $D$: flows never compress.}
123 + \label{fig:agen-flow}
124 +\end{figure}
125 +
126 +The whole design problem of flows is to make each
127 +$\det \mat{J}_{f_k}$ computable in $O(D)$ rather than $O(D^{3})$ without
128 +destroying expressiveness \cite{rezende2015}. The canonical solution is
129 +the \emph{affine coupling layer} of Real~NVP \cite{dinh2017}. Split the
130 +input into two halves, $\vect{x} = (\vect{x}_{1:d}, \vect{x}_{d+1:D})$;
131 +the layer leaves the first half untouched and applies to the second an
132 +affine transformation \emph{whose parameters are computed from the first}:
133 +\begin{equation}
134 + \vect{y}_{1:d} = \vect{x}_{1:d},
135 + \qquad
136 + \vect{y}_{d+1:D} = \vect{x}_{d+1:D} \odot
137 + \exp\!\big(s(\vect{x}_{1:d})\big) + t(\vect{x}_{1:d}),
138 + \label{eq:agen-coupling}
139 +\end{equation}
140 +where $s, t \colon \R^{d} \to \R^{D-d}$ are arbitrary neural networks —
141 +they are never inverted, so they can be as deep as desired. The inverse is
142 +immediate:
143 +\begin{equation}
144 + \vect{x}_{1:d} = \vect{y}_{1:d},
145 + \qquad
146 + \vect{x}_{d+1:D} = \big(\vect{y}_{d+1:D} - t(\vect{y}_{1:d})\big)
147 + \odot \exp\!\big(-s(\vect{y}_{1:d})\big).
148 + \label{eq:agen-coupling-inv}
149 +\end{equation}
150 +Because $\vect{y}_{1:d}$ does not depend on $\vect{x}_{d+1:D}$, the
151 +Jacobian of \eqref{eq:agen-coupling} is block-triangular with a diagonal
152 +lower-right block, and its log-determinant is a plain sum — no determinant
153 +computation at all:
154 +\begin{equation}
155 + \log \left| \det \mat{J} \right|
156 + = \sum_{j=1}^{D-d} s\big(\vect{x}_{1:d}\big)_j.
157 + \label{eq:agen-coupling-logdet}
158 +\end{equation}
159 +Alternating which half is transformed from layer to layer (and permuting
160 +or squeezing dimensions between couplings) lets every coordinate
161 +eventually influence every other.
162 +
163 +\begin{remark}[Expressiveness versus tractability]\label{rem:agen-tradeoff}
164 +Equations \eqref{eq:agen-coupling}--\eqref{eq:agen-coupling-logdet} are one
165 +point on a general trade-off. Triangular-Jacobian constructions — coupling
166 +layers, and the masked autoregressive flows in which
167 +$x_i = z_i\, \sigma_i(\vect{x}_{1:i-1}) + \mu_i(\vect{x}_{1:i-1})$ — buy an
168 +$O(D)$ log-determinant at the cost of restricting how information mixes in
169 +a single layer; depth restores expressiveness. Unconstrained maps would be
170 +maximally expressive per layer, but their $O(D^{3})$ determinants are
171 +unusable at scale. Flows also require the latent dimension to equal the
172 +data dimension, in sharp contrast with the bottlenecked autoencoders of
173 +the previous chapter.
174 +\end{remark}
175 +
176 +Estimation is pure maximum likelihood on \eqref{eq:agen-comp}:
177 +
178 +\begin{algorithm}[htbp]
179 + \caption{Maximum-likelihood training of a normalizing flow}
180 + \label{alg:agen-flow}
181 + \begin{algorithmic}[1]
182 + \Require dataset $\mathcal{D}$, bijections $f_1, \dots, f_K$ with
183 + parameters $\theta$, base density $p_Z$, learning rate $\eta$
184 + \Repeat
185 + \State sample a minibatch $\{\vect{x}^{(1)}, \dots,
186 + \vect{x}^{(m)}\} \subset \mathcal{D}$
187 + \For{each $\vect{x}^{(i)}$ in the minibatch}
188 + \State $\vect{z}_K \gets \vect{x}^{(i)}$; \quad $\ell^{(i)} \gets 0$
189 + \For{$k = K, K-1, \dots, 1$}
190 + \Comment{inverse pass: data $\to$ noise}
191 + \State $\vect{z}_{k-1} \gets f_k^{-1}(\vect{z}_k)$
192 + \State $\ell^{(i)} \gets \ell^{(i)}
193 + - \log\big|\det \mat{J}_{f_k}(\vect{z}_{k-1})\big|$
194 + \EndFor
195 + \State $\ell^{(i)} \gets \ell^{(i)} + \log p_Z(\vect{z}_0)$
196 + \Comment{$\ell^{(i)} = \log p_X(\vect{x}^{(i)})$ by
197 + \eqref{eq:agen-comp}}
198 + \EndFor
199 + \State $\Loss(\theta) \gets -\frac{1}{m}\sum_{i=1}^{m} \ell^{(i)}$
200 + \State $\theta \gets \theta - \eta \, \nabla_\theta \Loss(\theta)$
201 + \Until{convergence}
202 + \end{algorithmic}
203 +\end{algorithm}
204 +
205 +% ============================================================================
206 +\section{Diffusion Models}
207 +\label{sec:agen-diffusion}
208 +
209 +Diffusion models \cite{ho2020} take a fundamentally different route to an
210 +explicit likelihood: instead of building an invertible map, they
211 +\emph{destroy} the data with a fixed stochastic process and learn only the
212 +reversal. The approach currently dominates image, audio and video
213 +synthesis, and its training objective is among the simplest in all of deep
214 +learning.
215 +
216 +\subsection{The forward (noising) process}
217 +
218 +Fix a number of steps $T$ (typically $10^{3}$) and a variance schedule
219 +$\beta_1, \dots, \beta_T \in (0,1)$. The \emph{forward process} is a
220 +Markov chain that gradually replaces signal with Gaussian noise:
221 +\begin{equation}
222 + q(\vect{x}_t \mid \vect{x}_{t-1})
223 + = \mathcal{N}\!\big(\vect{x}_t;\,
224 + \sqrt{1-\beta_t}\,\vect{x}_{t-1},\; \beta_t \mat{I}\big),
225 + \qquad
226 + q(\vect{x}_{1:T} \mid \vect{x}_0)
227 + = \prod_{t=1}^{T} q(\vect{x}_t \mid \vect{x}_{t-1}).
228 + \label{eq:agen-forward}
229 +\end{equation}
230 +Nothing in \eqref{eq:agen-forward} is learned. The scaling
231 +$\sqrt{1-\beta_t}$ is chosen so that variance is preserved: if
232 +$\vect{x}_{t-1}$ has identity covariance, so does $\vect{x}_t$.
233 +
234 +\begin{property}[Closed-form marginal]\label{prop:agen-marginal}
235 +Let $\alpha_t = 1 - \beta_t$ and
236 +$\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$. Then the forward chain can be
237 +jumped in a single step from $\vect{x}_0$ to any $\vect{x}_t$:
238 +\begin{equation}
239 + q(\vect{x}_t \mid \vect{x}_0)
240 + = \mathcal{N}\!\big(\vect{x}_t;\,
241 + \sqrt{\bar{\alpha}_t}\,\vect{x}_0,\;
242 + (1-\bar{\alpha}_t)\,\mat{I}\big)
243 + \;\Longleftrightarrow\;
244 + \vect{x}_t = \sqrt{\bar{\alpha}_t}\,\vect{x}_0
245 + + \sqrt{1-\bar{\alpha}_t}\;\vect{\varepsilon},
246 + \quad \vect{\varepsilon} \sim \mathcal{N}(\vect{0},\mat{I}).
247 + \label{eq:agen-closed}
248 +\end{equation}
249 +\end{property}
250 +
251 +\begin{proof}[Proof sketch]
252 +Compose two steps of \eqref{eq:agen-forward}:
253 +$\vect{x}_t = \sqrt{\alpha_t \alpha_{t-1}}\, \vect{x}_{t-2} +
254 +\sqrt{\alpha_t(1-\alpha_{t-1})}\, \vect{\varepsilon}' +
255 +\sqrt{1-\alpha_t}\, \vect{\varepsilon}''$. The two independent Gaussian
256 +terms merge into a single Gaussian with variance
257 +$\alpha_t(1-\alpha_{t-1}) + (1-\alpha_t) = 1 - \alpha_t\alpha_{t-1}$;
258 +induction over $t$ gives \eqref{eq:agen-closed}.
259 +\end{proof}
260 +
261 +Since $\bar{\alpha}_T \approx 0$ for sensible schedules,
262 +$q(\vect{x}_T \mid \vect{x}_0) \approx \mathcal{N}(\vect{0}, \mat{I})$: at
263 +the end of the chain, every trace of the data is gone.
264 +Figure~\ref{fig:agen-schedule} compares the two standard schedules through
265 +the signal coefficient $\bar{\alpha}_t$.
266 +
267 +\begin{figure}[htbp]
268 + \centering
269 + \begin{tikzpicture}
270 + \begin{axis}[
271 + width=0.62\textwidth, height=0.36\textwidth,
272 + xlabel={$t/T$}, ylabel={$\bar{\alpha}_t$},
273 + xmin=0, xmax=1, ymin=0, ymax=1.02,
274 + legend style={font=\small, at={(0.97,0.97)}, anchor=north east},
275 + grid=major, grid style={black!12},
276 + ]
277 + \addplot[cinput, thick, domain=0:1, samples=120]
278 + {exp(-(0.1*x + 9.95*x^2))};
279 + \addlegendentry{linear $\beta_t$ schedule}
280 + \addplot[coutput, thick, domain=0:1, samples=120]
281 + {cos((x+0.008)/1.008 * 90)^2};
282 + \addlegendentry{cosine schedule}
283 + \end{axis}
284 + \end{tikzpicture}
285 + \caption{Remaining signal fraction $\bar{\alpha}_t$ along the forward
286 + process, for the linear schedule of \cite{ho2020}
287 + ($\beta_1 = 10^{-4}$ to $\beta_T = 0.02$, $T = 1000$) and the cosine
288 + schedule. The linear schedule destroys most of the signal in the first
289 + third of the chain; the cosine schedule spreads the destruction more
290 + evenly over time.}
291 + \label{fig:agen-schedule}
292 +\end{figure}
293 +
294 +\subsection{The reverse (generative) process}
295 +
296 +Generation runs the chain backwards: start from pure noise
297 +$p(\vect{x}_T) = \mathcal{N}(\vect{0}, \mat{I})$ and denoise step by step
298 +with a \emph{learned} Markov chain,
299 +\begin{equation}
300 + p_\theta(\vect{x}_{t-1} \mid \vect{x}_t)
301 + = \mathcal{N}\!\big(\vect{x}_{t-1};\,
302 + \vect{\mu}_\theta(\vect{x}_t, t),\; \sigma_t^2 \mat{I}\big).
303 + \label{eq:agen-reverse}
304 +\end{equation}
305 +The structure of the two chains is summarized in
306 +Figure~\ref{fig:agen-diffusion}. Training maximizes a variational lower
307 +bound on $\log p_\theta(\vect{x}_0)$, exactly as in the VAE, and the bound
308 +decomposes over timesteps into a sum of Kullback--Leibler divergences
309 +between Gaussians:
310 +\begin{equation}
311 +\begin{split}
312 + \Loss_{\mathrm{VLB}}
313 + = \E_q\Big[
314 + &\underbrace{\KL\big(q(\vect{x}_T \mid \vect{x}_0)
315 + \,\|\, p(\vect{x}_T)\big)}_{\Loss_T} \\[-2pt]
316 + &+ \sum_{t=2}^{T}
317 + \underbrace{\KL\big(q(\vect{x}_{t-1} \mid \vect{x}_t, \vect{x}_0)
318 + \,\|\, p_\theta(\vect{x}_{t-1} \mid \vect{x}_t)\big)}_{\Loss_{t-1}}
319 + \underbrace{-\, \log p_\theta(\vect{x}_0 \mid \vect{x}_1)}_{\Loss_0}
320 + \Big].
321 +\end{split}
322 + \label{eq:agen-elbo}
323 +\end{equation}
324 +Each middle term compares the model's reverse step with the \emph{true}
325 +posterior of the forward chain conditioned on the clean data — which is
326 +itself Gaussian and available in closed form:
327 +\begin{align}
328 + \tilde{\vect{\mu}}_t(\vect{x}_t, \vect{x}_0)
329 + &= \frac{\sqrt{\bar{\alpha}_{t-1}}\,\beta_t}{1-\bar{\alpha}_t}\,
330 + \vect{x}_0
331 + + \frac{\sqrt{\alpha_t}\,(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_t}\,
332 + \vect{x}_t,
333 + \label{eq:agen-postmean}\\
334 + \tilde{\beta}_t
335 + &= \frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t}\,\beta_t.
336 + \label{eq:agen-postvar}
337 +\end{align}
338 +
339 +\begin{figure}[htbp]
340 + \centering
341 + \begin{tikzpicture}[
342 + dnode/.style={bloc, minimum width=12.5mm, minimum height=9mm},
343 + ]
344 + \node[dnode, fill=black!2] (x0) at (0,0) {$\vect{x}_0$};
345 + \node[dnode, fill=black!12] (x1) at (2.9,0) {$\vect{x}_1$};
346 + \node (xd) at (5.8,0) {$\cdots$};
347 + \node[dnode, fill=black!30] (xt) at (8.7,0) {$\vect{x}_{T-1}$};
348 + \node[dnode, fill=black!45] (xT) at (11.6,0) {$\vect{x}_T$};
349 + % forward arrows (top, solid, blue)
350 + \foreach \a/\b in {x0/x1, x1/xd, xd/xt, xt/xT}
351 + \draw[fleche, cinput!80!black]
352 + ([yshift=1mm]\a.north east) to[out=35, in=145]
353 + ([yshift=1mm]\b.north west);
354 + \node[etiquette, text=cinput!80!black] at (5.8, 1.35)
355 + {forward: $q(\vect{x}_t \mid \vect{x}_{t-1})$ — fixed};
356 + % reverse arrows (bottom, dashed, red)
357 + \foreach \a/\b in {x1/x0, xd/x1, xt/xd, xT/xt}
358 + \draw[flechep, coutput!85!black]
359 + ([yshift=-1mm]\a.south west) to[out=215, in=-35]
360 + ([yshift=-1mm]\b.south east);
361 + \node[etiquette, text=coutput!85!black] at (5.8, -1.35)
362 + {reverse: $p_\theta(\vect{x}_{t-1} \mid \vect{x}_t)$ — learned};
363 + % end labels
364 + \node[etiquette, above=1.5mm of x0] {data};
365 + \node[etiquette, above=1.5mm of xT]
366 + {$\approx \mathcal{N}(\vect{0},\mat{I})$};
367 + \end{tikzpicture}
368 + \caption{The two Markov chains of a diffusion model. The fixed forward
369 + chain (solid blue, eq.~\eqref{eq:agen-forward}) progressively noises the
370 + data — suggested by the darkening blocks — until only Gaussian noise
371 + remains. The learned reverse chain (dashed red,
372 + eq.~\eqref{eq:agen-reverse}) denoises step by step; each reverse step is
373 + trained to match the tractable posterior
374 + \eqref{eq:agen-postmean}--\eqref{eq:agen-postvar}.}
375 + \label{fig:agen-diffusion}
376 +\end{figure}
377 +
378 +\subsection{Noise prediction and the simple loss}
379 +
380 +The decisive reparameterization of \cite{ho2020} is to make the network
381 +predict not the mean $\tilde{\vect{\mu}}_t$ but the \emph{noise}
382 +$\vect{\varepsilon}$ that was added in \eqref{eq:agen-closed}. Substituting
383 +$\vect{x}_0 = (\vect{x}_t - \sqrt{1-\bar{\alpha}_t}\,\vect{\varepsilon})
384 +/\sqrt{\bar{\alpha}_t}$ into \eqref{eq:agen-postmean} gives the model mean
385 +\begin{equation}
386 + \vect{\mu}_\theta(\vect{x}_t, t)
387 + = \frac{1}{\sqrt{\alpha_t}}
388 + \left( \vect{x}_t
389 + - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\,
390 + \vect{\varepsilon}_\theta(\vect{x}_t, t) \right),
391 + \label{eq:agen-mu}
392 +\end{equation}
393 +and the weighted sum of KL terms in \eqref{eq:agen-elbo}, once the
394 +time-dependent weights are dropped, collapses to a plain denoising
395 +regression:
396 +\begin{equation}
397 + \Loss_{\mathrm{simple}}(\theta)
398 + = \E_{t,\, \vect{x}_0,\, \vect{\varepsilon}}
399 + \Big[ \big\| \vect{\varepsilon}
400 + - \vect{\varepsilon}_\theta\big(
401 + \sqrt{\bar{\alpha}_t}\,\vect{x}_0
402 + + \sqrt{1-\bar{\alpha}_t}\,\vect{\varepsilon},\; t\big)
403 + \big\|^2 \Big],
404 + \qquad
405 + t \sim \mathcal{U}\{1,\dots,T\},\;
406 + \vect{\varepsilon} \sim \mathcal{N}(\vect{0},\mat{I}).
407 + \label{eq:agen-simple}
408 +\end{equation}
409 +Train a network (in practice a U-Net) to guess the noise hidden in a
410 +randomly noised example, at a random timestep — that is the entire
411 +training procedure, Algorithm~\ref{alg:agen-ddpm-train}. Sampling
412 +(Algorithm~\ref{alg:agen-ddpm-sample}) then applies \eqref{eq:agen-mu}
413 +from $t = T$ down to $1$, re-injecting fresh noise $\sigma_t \vect{z}$ at
414 +every step except the last.
415 +
416 +\begin{algorithm}[htbp]
417 + \caption{DDPM training \cite{ho2020}}
418 + \label{alg:agen-ddpm-train}
419 + \begin{algorithmic}[1]
420 + \Require data distribution $q(\vect{x}_0)$, schedule
421 + $\{\beta_t\}_{t=1}^T$, network $\vect{\varepsilon}_\theta$,
422 + learning rate $\eta$
423 + \Repeat
424 + \State $\vect{x}_0 \sim q(\vect{x}_0)$;\quad
425 + $t \sim \mathcal{U}\{1, \dots, T\}$;\quad
426 + $\vect{\varepsilon} \sim \mathcal{N}(\vect{0}, \mat{I})$
427 + \State $\vect{x}_t \gets \sqrt{\bar{\alpha}_t}\,\vect{x}_0
428 + + \sqrt{1-\bar{\alpha}_t}\,\vect{\varepsilon}$
429 + \Comment{one-step jump, eq.~\eqref{eq:agen-closed}}
430 + \State $\theta \gets \theta - \eta\,\nabla_\theta
431 + \big\| \vect{\varepsilon}
432 + - \vect{\varepsilon}_\theta(\vect{x}_t, t) \big\|^2$
433 + \Comment{eq.~\eqref{eq:agen-simple}}
434 + \Until{convergence}
435 + \end{algorithmic}
436 +\end{algorithm}
437 +
438 +\begin{algorithm}[htbp]
439 + \caption{DDPM ancestral sampling \cite{ho2020}}
440 + \label{alg:agen-ddpm-sample}
441 + \begin{algorithmic}[1]
442 + \Require trained $\vect{\varepsilon}_\theta$, schedule
443 + $\{\beta_t\}_{t=1}^T$, variances $\sigma_t^2$
444 + (e.g.\ $\sigma_t^2 = \tilde{\beta}_t$)
445 + \State $\vect{x}_T \sim \mathcal{N}(\vect{0}, \mat{I})$
446 + \For{$t = T, T-1, \dots, 1$}
447 + \State $\vect{z} \sim \mathcal{N}(\vect{0}, \mat{I})$ if $t > 1$,
448 + else $\vect{z} \gets \vect{0}$
449 + \State $\vect{x}_{t-1} \gets \dfrac{1}{\sqrt{\alpha_t}}
450 + \left( \vect{x}_t - \dfrac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\,
451 + \vect{\varepsilon}_\theta(\vect{x}_t, t) \right)
452 + + \sigma_t \vect{z}$
453 + \Comment{eq.~\eqref{eq:agen-mu}}
454 + \EndFor
455 + \State \Return $\vect{x}_0$
456 + \end{algorithmic}
457 +\end{algorithm}
458 +
459 +\subsection{Faster sampling and guidance}
460 +
461 +Two refinements matter enormously in practice. First, the thousand-step
462 +chain can be shortened. The \emph{denoising diffusion implicit model}
463 +(DDIM) defines a non-Markovian family sharing the marginals
464 +\eqref{eq:agen-closed} — so the very same trained
465 +$\vect{\varepsilon}_\theta$ can be reused — with update
466 +\begin{equation}
467 + \vect{x}_{t-1}
468 + = \sqrt{\bar{\alpha}_{t-1}}
469 + \underbrace{\left(
470 + \frac{\vect{x}_t - \sqrt{1-\bar{\alpha}_t}\,
471 + \vect{\varepsilon}_\theta(\vect{x}_t, t)}
472 + {\sqrt{\bar{\alpha}_t}} \right)}_{\text{predicted } \vect{x}_0}
473 + + \sqrt{1-\bar{\alpha}_{t-1}-\sigma_t^2}\;
474 + \vect{\varepsilon}_\theta(\vect{x}_t, t)
475 + + \sigma_t \vect{z}_t;
476 + \label{eq:agen-ddim}
477 +\end{equation}
478 +setting $\sigma_t = 0$ makes sampling \emph{deterministic} and allows
479 +$10$--$50$ steps instead of $1000$. Second, conditional generation is
480 +sharpened by \emph{classifier-free guidance}: train one network with the
481 +condition $y$ randomly dropped, then extrapolate between the conditional
482 +and unconditional predictions at sampling time,
483 +\begin{equation}
484 + \tilde{\vect{\varepsilon}}_\theta(\vect{x}_t, y)
485 + = (1+w)\,\vect{\varepsilon}_\theta(\vect{x}_t, y)
486 + - w\,\vect{\varepsilon}_\theta(\vect{x}_t, \varnothing),
487 + \label{eq:agen-cfg}
488 +\end{equation}
489 +where the guidance scale $w > 0$ trades sample diversity for fidelity to
490 +the condition. Equation \eqref{eq:agen-cfg} is the workhorse of modern
491 +text-to-image systems.
492 +
493 +\subsection{The score-based view and continuous time}
494 +
495 +Diffusion admits an equivalent formulation in terms of the \emph{score
496 +function} $\nabla_{\vect{x}} \log p(\vect{x})$ — the direction of steepest
497 +ascent of log-density. Learning the score at multiple noise levels is
498 +possible without ever knowing $p$, through \emph{denoising score
499 +matching}:
500 +\begin{equation}
501 + \Loss_{\mathrm{DSM}}(\theta)
502 + = \frac{1}{L} \sum_{i=1}^{L} \lambda(\sigma_i)\,
503 + \E_{\vect{x},\, \tilde{\vect{x}} \sim
504 + \mathcal{N}(\vect{x}, \sigma_i^2 \mat{I})}
505 + \left[ \left\| \vect{s}_\theta(\tilde{\vect{x}}, \sigma_i)
506 + + \frac{\tilde{\vect{x}} - \vect{x}}{\sigma_i^2} \right\|^2
507 + \right],
508 + \label{eq:agen-score}
509 +\end{equation}
510 +whose minimizer satisfies $\vect{s}_\theta(\tilde{\vect{x}}, \sigma)
511 +\approx \nabla_{\tilde{\vect{x}}} \log p_\sigma(\tilde{\vect{x}})$. In the
512 +continuous-time limit the forward chain \eqref{eq:agen-forward} becomes a
513 +stochastic differential equation
514 +$d\vect{x} = f(\vect{x}, t)\, dt + g(t)\, d\vect{w}$, and generation
515 +solves its \emph{reverse-time} SDE, which involves exactly the score:
516 +\begin{equation}
517 + d\vect{x}
518 + = \big[ f(\vect{x}, t)
519 + - g(t)^2\, \nabla_{\vect{x}} \log p_t(\vect{x}) \big]\, dt
520 + + g(t)\, d\bar{\vect{w}}.
521 + \label{eq:agen-sde}
522 +\end{equation}
523 +The DDPM chain is a discretization of a variance-preserving SDE, the
524 +deterministic DDIM sampler \eqref{eq:agen-ddim} a discretization of the
525 +associated probability-flow ODE, and the noise predictor a scaled score,
526 +$\vect{s}_\theta(\vect{x}_t, t) =
527 +-\vect{\varepsilon}_\theta(\vect{x}_t, t)/\sqrt{1-\bar{\alpha}_t}$. One
528 +trained network therefore serves stochastic sampling, deterministic
529 +sampling and exact likelihood computation alike.
530 +
531 +% ============================================================================
532 +\section{Deep Autoregressive Models}
533 +\label{sec:agen-ar}
534 +
535 +The third family needs no latent variable and no invertibility: it writes
536 +the exact likelihood directly with the \emph{chain rule of probability}.
537 +For $\vect{x} = (x_1, \dots, x_n)$ in any fixed ordering,
538 +\begin{equation}
539 + p(\vect{x}) = \prod_{i=1}^{n}
540 + p\big(x_i \mid x_1, \dots, x_{i-1}\big).
541 + \label{eq:agen-chain}
542 +\end{equation}
543 +A single network models all conditionals at once; training is parallel
544 +maximum likelihood (every position is predicted from its prefix,
545 +teacher-forced), while sampling is inherently sequential —
546 +Algorithm~\ref{alg:agen-ar}. This paradigm, applied to discrete tokens, is
547 +next-token prediction, the objective behind GPT-style language models;
548 +applied to pixels and audio samples it yields PixelCNN and WaveNet.
549 +
550 +\paragraph{Masked convolutions.}
551 +PixelCNN models images pixel by pixel in raster order. To let a
552 +convolutional network evaluate all conditionals of \eqref{eq:agen-chain}
553 +in one parallel pass, each kernel is multiplied by a binary
554 +\emph{causality mask} $\mat{M}$ that zeroes every weight looking at the
555 +current or future positions:
556 +\begin{equation}
557 + \big( (\mat{M} \odot \mat{W}) * \vect{x} \big)_i
558 + = \sum_{j} (\mat{M} \odot \mat{W})_j \, x_{i+j},
559 + \qquad
560 + M_j =
561 + \begin{cases}
562 + 1 & \text{if position $j$ precedes the center,}\\
563 + 0 & \text{otherwise,}
564 + \end{cases}
565 + \label{eq:agen-mask}
566 +\end{equation}
567 +so the receptive field of output $i$ contains only $x_{<i}$ and the
568 +autoregressive property holds by construction.
569 +
570 +\paragraph{Dilated causal convolutions.}
571 +WaveNet \cite{oord2016wavenet} applies \eqref{eq:agen-chain} to raw audio,
572 +where useful context spans thousands of samples. Its building block is the
573 +\emph{dilated causal convolution}, which reads the past with holes of
574 +size $d$:
575 +\begin{equation}
576 + \big( \mat{W} *_d \vect{x} \big)_t
577 + = \sum_{k=0}^{K-1} w_k \, x_{t - d\cdot k},
578 + \label{eq:agen-dilated}
579 +\end{equation}
580 +combined with a gated activation unit that borrows the multiplicative
581 +gating of recurrent networks:
582 +\begin{equation}
583 + \vect{z} = \tanh\!\big(\mat{W}_f *_d \vect{x}\big)
584 + \odot \sigma\!\big(\mat{W}_g *_d \vect{x}\big).
585 + \label{eq:agen-gated}
586 +\end{equation}
587 +Stacking $L$ layers with doubling dilations $d_\ell = 2^{\ell-1}$ and
588 +kernel size $K = 2$ makes the receptive field grow \emph{exponentially}
589 +with depth,
590 +\begin{equation}
591 + r = 1 + (K-1) \sum_{\ell=1}^{L} d_\ell
592 + = 1 + \sum_{\ell=1}^{L} 2^{\ell-1}
593 + = 2^{L},
594 + \label{eq:agen-rf}
595 +\end{equation}
596 +while the parameter count grows only linearly.
597 +Figure~\ref{fig:agen-wavenet} shows the resulting computation: the
598 +highlighted binary tree gathers all $16 = 2^{4}$ past inputs into a single
599 +output after just four layers.
600 +
601 +\begin{figure}[htbp]
602 + \centering
603 + \begin{tikzpicture}[
604 + snode/.style={circle, draw=black!55, fill=black!8,
605 + minimum size=3.4mm, inner sep=0pt},
606 + anode/.style={snode, fill=chidden!70, draw=chidden!85!black},
607 + hl/.style={draw=chidden!85!black, line width=1.05pt},
608 + ]
609 + % ---- nodes: row 0 = input, rows 1..4 = dilations 1,2,4,8 ----
610 + \foreach \row/\yy in {0/0, 1/1.05, 2/2.1, 3/3.15, 4/4.2}
611 + \foreach \t in {0,...,15}
612 + \node[snode] (n\row-\t) at (0.78*\t, \yy) {};
613 + % ---- all causal edges (gray) ----
614 + \foreach \row/\d in {1/1, 2/2, 3/4, 4/8} {
615 + \pgfmathtruncatemacro{\prev}{\row-1}
616 + \foreach \t in {0,...,15} {
617 + \draw[black!20] (n\prev-\t) -- (n\row-\t);
618 + \pgfmathtruncatemacro{\s}{\t-\d}
619 + \ifnum\s>-1
620 + \draw[black!20] (n\prev-\s) -- (n\row-\t);
621 + \fi
622 + }
623 + }
624 + % ---- highlighted binary tree reaching the last output ----
625 + \foreach \t in {1,3,5,7,9,11,13,15} {
626 + \pgfmathtruncatemacro{\s}{\t-1}
627 + \draw[hl] (n0-\t) -- (n1-\t);
628 + \draw[hl] (n0-\s) -- (n1-\t);
629 + }
630 + \foreach \t in {3,7,11,15} {
631 + \pgfmathtruncatemacro{\s}{\t-2}
632 + \draw[hl] (n1-\t) -- (n2-\t);
633 + \draw[hl] (n1-\s) -- (n2-\t);
634 + }
635 + \foreach \t in {7,15} {
636 + \pgfmathtruncatemacro{\s}{\t-4}
637 + \draw[hl] (n2-\t) -- (n3-\t);
638 + \draw[hl] (n2-\s) -- (n3-\t);
639 + }
640 + \draw[hl] (n3-15) -- (n4-15);
641 + \draw[hl] (n3-7) -- (n4-15);
642 + % ---- redraw the tree nodes on top ----
643 + \foreach \t in {0,...,15} \node[anode] at (n0-\t) {};
644 + \foreach \t in {1,3,5,7,9,11,13,15} \node[anode] at (n1-\t) {};
645 + \foreach \t in {3,7,11,15} \node[anode] at (n2-\t) {};
646 + \foreach \t in {7,15} \node[anode] at (n3-\t) {};
647 + \node[anode] at (n4-15) {};
648 + % ---- row labels ----
649 + \node[etiquette, anchor=east] at (-0.45, 0) {input};
650 + \node[etiquette, anchor=east] at (-0.45, 1.05) {dilation $d=1$};
651 + \node[etiquette, anchor=east] at (-0.45, 2.1) {dilation $d=2$};
652 + \node[etiquette, anchor=east] at (-0.45, 3.15) {dilation $d=4$};
653 + \node[etiquette, anchor=east] at (-0.45, 4.2) {dilation $d=8$};
654 + % ---- output arrow ----
655 + \draw[fleche, coutput!85!black] (n4-15) -- ++(0, 0.75)
656 + node[above, etiquette, text=coutput!85!black]
657 + {$p(x_{t+1} \mid x_{t-15}, \dots, x_t)$};
658 + \end{tikzpicture}
659 + \caption{WaveNet's stack of dilated causal convolutions
660 + (eq.~\eqref{eq:agen-dilated}) with kernel size $K=2$ and dilations
661 + $1, 2, 4, 8$. Gray edges show all connections; the highlighted binary
662 + tree is the receptive field of the final output — all $2^{4} = 16$ past
663 + samples reach it after only $4$ layers, as predicted by
664 + \eqref{eq:agen-rf}.}
665 + \label{fig:agen-wavenet}
666 +\end{figure}
667 +
668 +\begin{algorithm}[htbp]
669 + \caption{Autoregressive sampling}
670 + \label{alg:agen-ar}
671 + \begin{algorithmic}[1]
672 + \Require trained conditional model
673 + $p_\theta(x_i \mid \vect{x}_{<i})$, length $n$
674 + \For{$i = 1, 2, \dots, n$}
675 + \State compute the distribution
676 + $p_\theta(x_i \mid x_1, \dots, x_{i-1})$
677 + \Comment{one forward pass}
678 + \State sample $x_i \sim p_\theta(x_i \mid x_1, \dots, x_{i-1})$
679 + \EndFor
680 + \State \Return $\vect{x} = (x_1, \dots, x_n)$
681 + \end{algorithmic}
682 +\end{algorithm}
683 +
684 +\begin{remark}[Exactness versus sampling speed]\label{rem:agen-ar}
685 +Among all generative families, autoregressive models and flows are the
686 +only ones with \emph{exact} likelihoods. The price is paid at sampling
687 +time: generation by Algorithm~\ref{alg:agen-ar} costs one network
688 +evaluation per element, whereas a flow samples in a single pass. Diffusion
689 +sits in between, with tens to thousands of passes depending on the sampler
690 +\eqref{eq:agen-ddim}.
691 +\end{remark}
692 +
693 +% ============================================================================
694 +\section{Energy-Based Models: A Unifying Coda}
695 +\label{sec:agen-ebm}
696 +
697 +Many of the threads above meet in the \emph{energy-based model} (EBM),
698 +which drops all structural constraints and defines a density directly
699 +through a scalar energy network $E_\theta$:
700 +\begin{equation}
701 + p_\theta(\vect{x}) = \frac{e^{-E_\theta(\vect{x})}}{Z(\theta)},
702 + \qquad
703 + Z(\theta) = \int e^{-E_\theta(\vect{x})}\, d\vect{x}.
704 + \label{eq:agen-ebm}
705 +\end{equation}
706 +The partition function $Z(\theta)$ is intractable, but the
707 +maximum-likelihood gradient takes a contrastive form that never requires
708 +$Z$ itself:
709 +\begin{equation}
710 + \nabla_\theta \log p_\theta(\vect{x})
711 + = -\nabla_\theta E_\theta(\vect{x})
712 + + \E_{\vect{x}' \sim p_\theta}
713 + \big[ \nabla_\theta E_\theta(\vect{x}') \big]
714 + \label{eq:agen-ebmgrad}
715 +\end{equation}
716 +— push the energy of real data down (positive phase), push the energy of
717 +model samples up (negative phase). The negative samples $\vect{x}'$ are
718 +drawn by Markov chain Monte Carlo, typically \emph{Langevin dynamics},
719 +which follows the energy gradient with injected noise:
720 +\begin{equation}
721 + \vect{x}_{k+1} = \vect{x}_k
722 + - \frac{\eta}{2} \nabla_{\vect{x}} E_\theta(\vect{x}_k)
723 + + \sqrt{\eta}\; \vect{\varepsilon}_k,
724 + \qquad \vect{\varepsilon}_k \sim \mathcal{N}(\vect{0}, \mat{I}).
725 + \label{eq:agen-langevin}
726 +\end{equation}
727 +
728 +\begin{remark}[One object, many views]\label{rem:agen-unify}
729 +Since $\nabla_{\vect{x}} \log p_\theta(\vect{x}) =
730 +-\nabla_{\vect{x}} E_\theta(\vect{x})$, sampling an EBM by
731 +\eqref{eq:agen-langevin} is exactly \emph{score following} — and the noise
732 +predictor of a diffusion model is a scaled score estimate,
733 +$\nabla_{\vect{x}_t} \log p(\vect{x}_t) =
734 +-\vect{\varepsilon}_\theta(\vect{x}_t, t)/\sqrt{1-\bar{\alpha}_t}$.
735 +Diffusion models can thus be read as EBMs whose sampling chain has been
736 +fixed, annealed and amortized. Historically, the same positive/negative
737 +phase structure of \eqref{eq:agen-ebmgrad} appeared in the Boltzmann
738 +machines and restricted Boltzmann machines discussed earlier in this book,
739 +trained by contrastive divergence; the modern generative families of this
740 +chapter are, in that sense, their descendants.
741 +\end{remark}
added chapters/12-bio-inspired.tex +926 −0
@@ -0,0 +1,926 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 12 : Biologically Inspired Networks (chapters/12-bio-inspired.tex)
6 +% ============================================================================
7 +
8 +\chapter{Biologically Inspired Networks: Spiking Neurons, Self-Organizing
9 + Maps and Reservoir Computing}\label{chap:bio}
10 +
11 +Every architecture examined so far shares one commitment: a differentiable
12 +map from inputs to outputs, trained by gradient descent on a global
13 +objective. That commitment is a modelling choice, not a law of nature, and
14 +the brain does not appear to honour it. Cortical neurons communicate with
15 +discrete, asynchronous electrical events rather than real-valued
16 +activations; synapses are modified by locally available information rather
17 +than by a gradient routed backwards from a distant loss; and much of the
18 +structure of sensory cortex emerges from unsupervised competition rather
19 +than from labelled supervision.
20 +
21 +This chapter develops three families that each relax one of those
22 +commitments while remaining mathematically precise. \emph{Spiking neural
23 +networks} replace the real-valued activation by a temporal train of events,
24 +and the gradient by a local, timing-dependent plasticity rule.
25 +\emph{Self-organizing maps} replace supervised error correction by
26 +competitive learning, and obtain a topology-preserving projection of the
27 +data as an emergent property. \emph{Reservoir computing} keeps the
28 +recurrent architecture but refuses to train it, learning only a linear
29 +readout — which reduces the entire training problem to convex regression.
30 +For each family we give the state equations, an estimation algorithm in
31 +pseudocode, and a figure of the mechanism.
32 +
33 +% ----------------------------------------------------------------------------
34 +\section{Spiking Neural Networks}
35 +% ----------------------------------------------------------------------------
36 +
37 +Spiking networks have been called the third generation of neural network
38 +models \cite{maass1997}: the first generation computes with thresholded
39 +binary units, the second with continuous activations, and the third with
40 +sequences of events in continuous time. A spiking neuron does not output a
41 +number; it outputs a set of firing times $\{t^{(1)}, t^{(2)}, \dots\}$,
42 +conventionally written as a sum of Dirac impulses,
43 +\begin{equation}
44 + S(t) \;=\; \sum_{f} \delta\!\left(t - t^{(f)}\right),
45 + \label{eq:bio-spiketrain}
46 +\end{equation}
47 +so that the activation of a neuron is a distribution rather than a scalar.
48 +Information may then be carried by how \emph{often} a neuron fires, or by
49 +\emph{when} it fires — a distinction with no counterpart in the
50 +architectures of the preceding chapters.
51 +
52 +\subsection{The Leaky Integrate-and-Fire Neuron}
53 +
54 +The standard computational unit abstracts the cell membrane as a leaky
55 +capacitor charged by the incoming synaptic current.
56 +
57 +\begin{definition}[Leaky integrate-and-fire neuron]\label{def:bio-lif}
58 +A leaky integrate-and-fire (LIF) neuron with membrane time constant
59 +$\tau_m$, resting potential $V_{\mathrm{rest}}$, membrane resistance $R_m$
60 +and threshold $V_{\mathrm{th}}$ evolves according to
61 +\begin{equation}
62 + \tau_m \, \frac{\mathrm{d}V(t)}{\mathrm{d}t}
63 + \;=\; -\bigl(V(t) - V_{\mathrm{rest}}\bigr) \;+\; R_m\, I(t),
64 + \label{eq:bio-lif-ode}
65 +\end{equation}
66 +supplemented by the fire-and-reset rule
67 +\begin{equation}
68 + \text{if } V(t^-) \ge V_{\mathrm{th}}:
69 + \quad
70 + \text{emit a spike at } t,
71 + \quad
72 + V(t^+) \leftarrow V_{\mathrm{reset}},
73 + \quad
74 + V \equiv V_{\mathrm{reset}} \text{ on } [t,\, t + t_{\mathrm{ref}}],
75 + \label{eq:bio-lif-reset}
76 +\end{equation}
77 +where $t_{\mathrm{ref}}$ is the absolute refractory period.
78 +\end{definition}
79 +
80 +Equation~\eqref{eq:bio-lif-ode} is a first-order linear ODE, so its
81 +behaviour under a constant input current $I(t) \equiv I_0$ is available in
82 +closed form. Starting from $V(0) = V_{\mathrm{reset}}$,
83 +\begin{equation}
84 + V(t)
85 + \;=\;
86 + V_{\mathrm{rest}} + R_m I_0
87 + \;+\;
88 + \bigl(V_{\mathrm{reset}} - V_{\mathrm{rest}} - R_m I_0\bigr)
89 + \, e^{-t/\tau_m},
90 + \label{eq:bio-lif-sol}
91 +\end{equation}
92 +an exponential relaxation towards the asymptote
93 +$V_\infty = V_{\mathrm{rest}} + R_m I_0$. Two regimes follow immediately.
94 +If $V_\infty < V_{\mathrm{th}}$ the neuron is \emph{subthreshold} and never
95 +fires, however long we wait: the input is filtered out entirely. If
96 +$V_\infty > V_{\mathrm{th}}$, solving \eqref{eq:bio-lif-sol} for the
97 +threshold crossing gives the interspike interval
98 +\begin{equation}
99 + T_{\mathrm{ISI}}
100 + \;=\;
101 + t_{\mathrm{ref}}
102 + \;+\;
103 + \tau_m \,
104 + \ln\!\left(
105 + \frac{R_m I_0 - (V_{\mathrm{reset}} - V_{\mathrm{rest}})}
106 + {R_m I_0 - (V_{\mathrm{th}} - V_{\mathrm{rest}})}
107 + \right),
108 + \label{eq:bio-lif-isi}
109 +\end{equation}
110 +and hence the current-to-frequency transfer function, the \emph{f--I curve}
111 +of the neuron,
112 +\begin{equation}
113 + f(I_0) \;=\; \frac{1}{T_{\mathrm{ISI}}}
114 + \;=\;
115 + \left[
116 + t_{\mathrm{ref}} + \tau_m \ln\!\left(
117 + \frac{R_m I_0 - V_{\mathrm{reset}} + V_{\mathrm{rest}}}
118 + {R_m I_0 - V_{\mathrm{th}} + V_{\mathrm{rest}}}
119 + \right)
120 + \right]^{-1}.
121 + \label{eq:bio-lif-fi}
122 +\end{equation}
123 +The refractory term in \eqref{eq:bio-lif-fi} imposes the saturation
124 +$f \le 1/t_{\mathrm{ref}}$: unlike a ReLU, a spiking neuron has a hard
125 +maximum firing rate. Figure~\ref{fig:bio-lif-trace} traces the
126 +corresponding membrane dynamics.
127 +
128 +\begin{figure}[htbp]
129 + \centering
130 + \begin{tikzpicture}
131 + \begin{axis}[
132 + width=13.4cm, height=6.2cm,
133 + xmin=0, xmax=42, ymin=-0.08, ymax=1.70,
134 + xlabel={time $t$ (ms)},
135 + ylabel={membrane potential $V(t)$},
136 + ylabel style={font=\small}, xlabel style={font=\small},
137 + tick label style={font=\scriptsize},
138 + axis lines=left,
139 + ytick={0,1}, yticklabels={$V_{\mathrm{reset}}$, $V_{\mathrm{th}}$},
140 + xtick={0,10,20,30,40},
141 + clip=false,
142 + ]
143 + % --- refractory shading (drawn first, behind everything) ---
144 + \fill[black!8] (axis cs:10.986,0) rectangle (axis cs:12.986,1.40);
145 + \fill[black!8] (axis cs:23.972,0) rectangle (axis cs:25.972,1.40);
146 + \fill[black!8] (axis cs:36.958,0) rectangle (axis cs:38.958,1.40);
147 + % --- threshold and asymptote ---
148 + \draw[dashed, black!65, thick] (axis cs:0,1) -- (axis cs:42,1);
149 + \draw[dotted, black!50, thick] (axis cs:0,1.5) -- (axis cs:42,1.5);
150 + % --- spikes and resets (unrolled: \foreach variables do not survive
151 + % the deferred execution of drawing commands inside an axis) ---
152 + \draw[coutput, line width=1.1pt]
153 + (axis cs:10.986,0) -- (axis cs:10.986,1.40);
154 + \draw[coutput, line width=1.1pt]
155 + (axis cs:23.972,0) -- (axis cs:23.972,1.40);
156 + \draw[coutput, line width=1.1pt]
157 + (axis cs:36.958,0) -- (axis cs:36.958,1.40);
158 + % --- charging ramps: V = 1.5 (1 - e^{-(t-t_0)/10}) ---
159 + \addplot[cinput, line width=1.1pt, domain=0:10.986, samples=60]
160 + {1.5*(1-exp(-x/10))};
161 + \addplot[cinput, line width=1.1pt, domain=12.986:23.972, samples=60]
162 + {1.5*(1-exp(-(x-12.986)/10))};
163 + \addplot[cinput, line width=1.1pt, domain=25.972:36.958, samples=60]
164 + {1.5*(1-exp(-(x-25.972)/10))};
165 + \addplot[cinput, line width=1.1pt, domain=38.958:42, samples=30]
166 + {1.5*(1-exp(-(x-38.958)/10))};
167 + % --- refractory plateaus ---
168 + \addplot[cinput, line width=1.1pt, domain=10.986:12.986, samples=2]{0};
169 + \addplot[cinput, line width=1.1pt, domain=23.972:25.972, samples=2]{0};
170 + \addplot[cinput, line width=1.1pt, domain=36.958:38.958, samples=2]{0};
171 + % --- annotations ---
172 + \node[etiquette, anchor=west, text=coutput]
173 + at (axis cs:11.25,1.20) {spike};
174 + \node[etiquette, anchor=south] at (axis cs:11.99,0.04)
175 + {$t_{\mathrm{ref}}$};
176 + \node[etiquette, anchor=west, fill=white, inner sep=1.5pt]
177 + at (axis cs:0.8,1.5) {asymptote $V_\infty = R_m I_0$, never reached};
178 + \end{axis}
179 + \end{tikzpicture}
180 + \caption{Membrane potential of an LIF neuron \eqref{eq:bio-lif-ode} driven
181 + by a constant supra-threshold current, with $\tau_m = 10$~ms,
182 + $R_m I_0 = 1.5\,V_{\mathrm{th}}$,
183 + $V_{\mathrm{rest}} = V_{\mathrm{reset}} = 0$ and
184 + $t_{\mathrm{ref}} = 2$~ms. The potential relaxes exponentially towards
185 + $V_\infty$, but crosses the dashed threshold first: a spike (red) is
186 + emitted, the potential is reset, and the shaded refractory window clamps
187 + it before integration resumes. The period predicted by
188 + \eqref{eq:bio-lif-isi} is
189 + $t_{\mathrm{ref}} + \tau_m \ln 3 \approx 12.99$~ms, a firing rate of
190 + about $77$~Hz.}
191 + \label{fig:bio-lif-trace}
192 +\end{figure}
193 +
194 +For simulation and for training, \eqref{eq:bio-lif-ode} is discretized with
195 +a fixed step $\Delta t$. Writing $\beta = e^{-\Delta t / \tau_m}$ for the
196 +per-step decay, taking $V_{\mathrm{rest}} = 0$, and folding the synaptic
197 +weights $\vect{w}$ into the input current, one obtains the recurrence used
198 +by every modern spiking-network framework:
199 +\begin{align}
200 + V[t] &= \beta\, V[t-1]
201 + \;+\; \vect{w}\transp \vect{S}_{\mathrm{in}}[t]
202 + \;-\; S[t-1]\, V_{\mathrm{th}},
203 + \label{eq:bio-lif-discrete}\\[2pt]
204 + S[t] &= \Theta\!\bigl(V[t] - V_{\mathrm{th}}\bigr),
205 + \qquad
206 + \Theta(u) = \begin{cases} 1, & u \ge 0,\\ 0, & u < 0,\end{cases}
207 + \label{eq:bio-lif-heaviside}
208 +\end{align}
209 +where $\vect{S}_{\mathrm{in}}[t] \in \{0,1\}^{n}$ collects the presynaptic
210 +spikes arriving at step $t$. The last term of \eqref{eq:bio-lif-discrete}
211 +implements a \emph{soft reset} by subtraction, which preserves the
212 +supra-threshold excess; the alternative \emph{hard reset} sets
213 +$V[t] \leftarrow 0$ and discards it. Note the structural resemblance
214 +between \eqref{eq:bio-lif-discrete} and the state update of a gated
215 +recurrent unit: an LIF layer is a recurrent network whose forget gate is
216 +frozen at the constant $\beta$ and whose output nonlinearity is a step.
217 +
218 +\begin{remark}[Biophysically complete models]\label{rem:bio-hh}
219 +The LIF neuron is a caricature. The Hodgkin--Huxley model instead resolves
220 +the ionic conductances explicitly, through four coupled state variables:
221 +\begin{equation}
222 + C_m \frac{\mathrm{d}V}{\mathrm{d}t}
223 + = I_{\mathrm{ext}}
224 + - \bar{g}_{\mathrm{Na}}\, m^{3} h \,(V - E_{\mathrm{Na}})
225 + - \bar{g}_{\mathrm{K}}\, n^{4} (V - E_{\mathrm{K}})
226 + - \bar{g}_{L} (V - E_{L}),
227 + \label{eq:bio-hh}
228 +\end{equation}
229 +each gating variable $x \in \{m, h, n\}$ obeying first-order kinetics
230 +$\dot{x} = \bigl(x_\infty(V) - x\bigr)/\tau_x(V)$. Here the spike is an
231 +emergent consequence of sodium activation rather than an imposed threshold
232 +rule — at a cost of roughly $10^{3}$ floating-point operations per
233 +millisecond of simulated time.
234 +\end{remark}
235 +
236 +\begin{remark}[The Izhikevich compromise]\label{rem:bio-izh}
237 +A two-dimensional reduction of \eqref{eq:bio-hh} retains most of the
238 +qualitative firing repertoire at about $13$ operations per millisecond:
239 +\begin{equation}
240 + \frac{\mathrm{d}v}{\mathrm{d}t} = 0.04\, v^{2} + 5v + 140 - u + I,
241 + \qquad
242 + \frac{\mathrm{d}u}{\mathrm{d}t} = a\,(b v - u),
243 + \label{eq:bio-izh}
244 +\end{equation}
245 +with the after-spike reset $v \leftarrow c$, $u \leftarrow u + d$ whenever
246 +$v \ge 30$~mV. The four parameters $(a,b,c,d)$ select among regular
247 +spiking, bursting, chattering and fast-spiking regimes.
248 +\end{remark}
249 +
250 +\subsection{Neural Coding}
251 +
252 +Because a spike train is a point process, the question of what the code
253 +\emph{is} becomes explicit. Under \emph{rate coding} the transmitted
254 +quantity is the count obtained by integrating the spike train
255 +\eqref{eq:bio-spiketrain} over a window of length $T$,
256 +\begin{equation}
257 + r \;=\; \frac{1}{T}\int_{t}^{t+T} S(u)\,\mathrm{d}u
258 + \;=\; \frac{\bigl|\{f : t \le t^{(f)} < t+T\}\bigr|}{T},
259 + \label{eq:bio-ratecode}
260 +\end{equation}
261 +which recovers a real-valued activation and makes spiking networks
262 +approximately equivalent to conventional ones, at the price of a latency
263 +and an energy budget proportional to the spike count. Under \emph{temporal}
264 +or time-to-first-spike coding the quantity is the latency $t^{(1)}$ itself,
265 +so a single spike per neuron suffices — the regime in which neuromorphic
266 +hardware achieves its energy advantage. Phase coding (spike time relative
267 +to a background oscillation) and rank-order coding (the permutation in
268 +which a population fires) are further alternatives.
269 +
270 +\subsection{Spike-Timing-Dependent Plasticity}
271 +
272 +The canonical learning rule for spiking networks is local in both space and
273 +time: a synapse is strengthened when it participates causally in a
274 +postsynaptic spike, and weakened when it fires just after one. Writing
275 +$\Delta t = t_{\mathrm{post}} - t_{\mathrm{pre}}$ for the signed latency of
276 +a pre/post spike pair, the classical additive exponential window is
277 +\begin{equation}
278 + \Delta w(\Delta t)
279 + \;=\;
280 + \begin{cases}
281 + \phantom{-}A_{+}\exp\!\left(-\dfrac{\Delta t}{\tau_{+}}\right),
282 + & \Delta t > 0
283 + \qquad \text{(pre before post: potentiation)},\\[2.4ex]
284 + -A_{-}\exp\!\left(\dfrac{\Delta t}{\tau_{-}}\right),
285 + & \Delta t \le 0
286 + \qquad \text{(post before pre: depression)},
287 + \end{cases}
288 + \label{eq:bio-stdp}
289 +\end{equation}
290 +with time constants $\tau_{\pm}$ of the order of $20$~ms.
291 +Equation~\eqref{eq:bio-stdp} is an unsupervised Hebbian rule that requires
292 +no error signal, no target and no backward pass: it is a pure causality
293 +detector, and every quantity it needs is available at the synapse itself.
294 +Its shape is plotted in Figure~\ref{fig:bio-stdp}.
295 +
296 +\begin{figure}[htbp]
297 + \centering
298 + \begin{tikzpicture}
299 + \begin{axis}[
300 + width=12.4cm, height=6.2cm,
301 + xmin=-80, xmax=80, ymin=-0.85, ymax=1.22,
302 + axis lines=middle,
303 + xlabel={$\Delta t = t_{\mathrm{post}} - t_{\mathrm{pre}}$ (ms)},
304 + ylabel={$\Delta w$},
305 + xlabel style={font=\small,
306 + at={(axis description cs:0.5,-0.02)}, anchor=north},
307 + ylabel style={font=\small, at={(ticklabel* cs:1.0)},
308 + anchor=south east},
309 + tick label style={font=\scriptsize},
310 + xtick={-80,-40,40,80}, ytick={-0.6,-0.3,0.3,0.6,0.9},
311 + clip=false,
312 + ]
313 + \addplot[cgate, line width=1.2pt, domain=0.001:80, samples=120]
314 + {1.0*exp(-x/17)};
315 + \addplot[coutput, line width=1.2pt, domain=-80:-0.001, samples=120]
316 + {-0.6*exp(x/34)};
317 + \draw[cgate, dotted, thick] (axis cs:0,0) -- (axis cs:0, 1.0);
318 + \draw[coutput, dotted, thick] (axis cs:0,0) -- (axis cs:0,-0.6);
319 + \node[etiquette, anchor=west, text=cgate!55!black, align=left]
320 + at (axis cs:14,0.88)
321 + {LTP: pre before post,\\[-1pt]$A_{+}e^{-\Delta t/\tau_{+}}$};
322 + \node[etiquette, anchor=east, text=coutput!85!black, align=right]
323 + at (axis cs:-14,-0.56)
324 + {LTD: post before pre,\\[-1pt]$-A_{-}e^{\Delta t/\tau_{-}}$};
325 + \end{axis}
326 + \end{tikzpicture}
327 + \caption{The pair-based STDP window \eqref{eq:bio-stdp}, with
328 + $A_{+} = 1.0$, $\tau_{+} = 17$~ms, $A_{-} = 0.6$ and
329 + $\tau_{-} = 34$~ms. A presynaptic spike arriving shortly \emph{before} a
330 + postsynaptic one strengthens the synapse (green, right half-plane); the
331 + reverse order weakens it (red, left half-plane). The rule is
332 + discontinuous at $\Delta t = 0$ and decays to zero for
333 + $|\Delta t| \gg \tau_{\pm}$, so only near-coincident pairs are
334 + consequential. The asymmetry $A_{-}\tau_{-} > A_{+}\tau_{+}$ shown here
335 + makes the net effect of uncorrelated firing depressing, which keeps
336 + weights from drifting upwards.}
337 + \label{fig:bio-stdp}
338 +\end{figure}
339 +
340 +Implementing \eqref{eq:bio-stdp} literally would require storing all pairs
341 +of spike times. The standard online form instead keeps one low-pass
342 +\emph{eligibility trace} per terminal, decaying exponentially and
343 +incremented at every spike:
344 +\begin{align}
345 + \tau_{+}\frac{\mathrm{d}x_{\mathrm{pre}}}{\mathrm{d}t}
346 + &= -\,x_{\mathrm{pre}}
347 + + \sum_{f}\delta\!\left(t - t^{(f)}_{\mathrm{pre}}\right),
348 + \qquad
349 + \tau_{-}\frac{\mathrm{d}x_{\mathrm{post}}}{\mathrm{d}t}
350 + = -\,x_{\mathrm{post}}
351 + + \sum_{f}\delta\!\left(t - t^{(f)}_{\mathrm{post}}\right),
352 + \label{eq:bio-stdp-traces}\\[2pt]
353 + \frac{\mathrm{d}w}{\mathrm{d}t}
354 + &= A_{+}\, x_{\mathrm{pre}}(t)\, S_{\mathrm{post}}(t)
355 + \;-\; A_{-}\, x_{\mathrm{post}}(t)\, S_{\mathrm{pre}}(t).
356 + \label{eq:bio-stdp-online}
357 +\end{align}
358 +Equations \eqref{eq:bio-stdp-traces}--\eqref{eq:bio-stdp-online} reproduce
359 +\eqref{eq:bio-stdp} exactly for isolated spike pairs while requiring $O(1)$
360 +memory per synapse, which is why they, and not the pairwise window, are
361 +what neuromorphic hardware implements.
362 +Algorithm~\ref{alg:bio-snn-stdp} assembles the discretized simulation loop;
363 +substituting a richer neuron model such as \eqref{eq:bio-izh} changes only
364 +the membrane update on line~4, leaving the plasticity block untouched.
365 +
366 +\begin{algorithm}[htbp]
367 +\caption{Clocked simulation of an LIF layer with online STDP}
368 +\label{alg:bio-snn-stdp}
369 +\begin{algorithmic}[1]
370 +\Require step $\Delta t$; horizon $T$; weights
371 + $\mat{W} \in \R^{n_{\mathrm{post}} \times n_{\mathrm{pre}}}$; decays
372 + $\beta = e^{-\Delta t/\tau_m}$, $\beta_{\pm} = e^{-\Delta t/\tau_{\pm}}$;
373 + rates $A_{\pm}$; bound $w_{\max}$
374 +\State $\vect{V} \leftarrow \vect{0}$;\;
375 + $\vect{S}_{\mathrm{post}} \leftarrow \vect{0}$;\;
376 + $\vect{x}_{\mathrm{pre}} \leftarrow \vect{0}$;\;
377 + $\vect{x}_{\mathrm{post}} \leftarrow \vect{0}$;\;
378 + $\vect{\rho} \leftarrow \vect{0}$
379 + \Comment{$\vect{\rho}$: refractory counters}
380 +\For{$t = 1, \dots, T/\Delta t$}
381 + \State $\vect{S}_{\mathrm{pre}} \leftarrow$ presynaptic spikes at step $t$
382 + \State $\vect{V} \leftarrow \beta\,\vect{V}
383 + + \mat{W}\,\vect{S}_{\mathrm{pre}}
384 + - V_{\mathrm{th}}\,\vect{S}_{\mathrm{post}}$
385 + \Comment{leak, integrate, soft reset \eqref{eq:bio-lif-discrete}}
386 + \State $\vect{V} \leftarrow \vect{V} \odot \mathbf{1}[\vect{\rho} = 0]$
387 + \Comment{clamp refractory units \eqref{eq:bio-lif-reset}}
388 + \State $\vect{S}_{\mathrm{post}} \leftarrow
389 + \Theta(\vect{V} - V_{\mathrm{th}}) \odot \mathbf{1}[\vect{\rho} = 0]$
390 + \Comment{\eqref{eq:bio-lif-heaviside}}
391 + \State $\vect{\rho} \leftarrow \max\!\bigl(\vect{\rho} - 1,\, 0\bigr)
392 + + \lceil t_{\mathrm{ref}}/\Delta t \rceil\,
393 + \vect{S}_{\mathrm{post}}$
394 + \Statex \hspace{\algorithmicindent}\textit{plasticity: traces first, then
395 + the two-term update}
396 + \State $\vect{x}_{\mathrm{pre}} \leftarrow
397 + \beta_{+}\,\vect{x}_{\mathrm{pre}} + \vect{S}_{\mathrm{pre}}$;\quad
398 + $\vect{x}_{\mathrm{post}} \leftarrow
399 + \beta_{-}\,\vect{x}_{\mathrm{post}} + \vect{S}_{\mathrm{post}}$
400 + \Comment{\eqref{eq:bio-stdp-traces}}
401 + \State $\mat{W} \leftarrow \mat{W}
402 + + A_{+}\,\vect{S}_{\mathrm{post}}\,\vect{x}_{\mathrm{pre}}\transp
403 + - A_{-}\,\vect{x}_{\mathrm{post}}\,\vect{S}_{\mathrm{pre}}\transp$
404 + \Comment{rule \eqref{eq:bio-stdp-online}}
405 + \State $\mat{W} \leftarrow \mathrm{clip}\bigl(\mat{W},\, 0,\, w_{\max}\bigr)$
406 + \Comment{additive STDP is otherwise unbounded}
407 +\EndFor
408 +\State \Return $\mat{W}$
409 +\end{algorithmic}
410 +\end{algorithm}
411 +
412 +\subsection{Surrogate Gradients}
413 +
414 +STDP is unsupervised and cannot by itself solve a supervised task. To train
415 +a deep spiking network by gradient descent one must differentiate
416 +\eqref{eq:bio-lif-heaviside}, and the derivative of the Heaviside step is a
417 +Dirac impulse: zero almost everywhere, unbounded at the threshold. Gradient
418 +descent through it is impossible. The \emph{surrogate gradient} method
419 +keeps the step function in the forward pass but substitutes a smooth,
420 +bounded pseudo-derivative in the backward pass, for instance the
421 +fast-sigmoid derivative
422 +\begin{equation}
423 + \frac{\partial S}{\partial V}
424 + \;\approx\;
425 + \sigma'_{\mathrm{surr}}(V)
426 + \;=\;
427 + \frac{1}{\bigl(1 + \gamma\,|V - V_{\mathrm{th}}|\bigr)^{2}},
428 + \qquad \gamma > 0,
429 + \label{eq:bio-surrogate}
430 +\end{equation}
431 +which peaks at the threshold and decays away from it, so that only neurons
432 +close to firing receive appreciable credit. With \eqref{eq:bio-surrogate}
433 +in place the discretized network \eqref{eq:bio-lif-discrete} becomes an
434 +ordinary recurrent network over the time axis, trainable by
435 +backpropagation through time (Algorithm~\ref{alg:bio-surrogate}). The
436 +steepness $\gamma$ controls a bias--variance trade-off: large $\gamma$
437 +approaches the true, useless derivative, while small $\gamma$ leaks
438 +gradient to neurons far from threshold.
439 +
440 +\begin{algorithm}[htbp]
441 +\caption{Surrogate-gradient training of a deep spiking network}
442 +\label{alg:bio-surrogate}
443 +\begin{algorithmic}[1]
444 +\Require encoded input spike trains $\vect{S}^{(0)}[1{:}T]$; targets
445 + $\vect{y}$; layers $\ell = 1, \dots, L$; steepness $\gamma$; rate $\eta$
446 +\For{each mini-batch}
447 + \State $\vect{V}^{(\ell)} \leftarrow \vect{0}$,\;
448 + $\vect{S}^{(\ell)} \leftarrow \vect{0}$ for all $\ell$
449 + \For{$t = 1, \dots, T$}
450 + \Comment{forward pass: the exact step function is retained}
451 + \For{$\ell = 1, \dots, L$}
452 + \State $\vect{V}^{(\ell)}[t] \leftarrow
453 + \beta\, \vect{V}^{(\ell)}[t-1]
454 + + \mat{W}^{(\ell)} \vect{S}^{(\ell-1)}[t]
455 + - V_{\mathrm{th}}\, \vect{S}^{(\ell)}[t-1]$
456 + \State $\vect{S}^{(\ell)}[t] \leftarrow
457 + \Theta\bigl(\vect{V}^{(\ell)}[t] - V_{\mathrm{th}}\bigr)$
458 + \EndFor
459 + \EndFor
460 + \State $\hat{\vect{y}} \leftarrow
461 + \softmax\bigl(\textstyle\sum_{t} \vect{V}^{(L)}[t]\bigr)$
462 + \Comment{or a spike-count readout \eqref{eq:bio-ratecode}}
463 + \State $\Loss \leftarrow \mathrm{CE}(\hat{\vect{y}}, \vect{y})$
464 + \State backpropagate through time, replacing every factor
465 + $\partial \vect{S}^{(\ell)}[t] / \partial \vect{V}^{(\ell)}[t]$
466 + by $\sigma'_{\mathrm{surr}}$ of \eqref{eq:bio-surrogate}
467 + \State $\mat{W}^{(\ell)} \leftarrow \mat{W}^{(\ell)}
468 + - \eta\, \nabla_{\mat{W}^{(\ell)}} \Loss$ for all $\ell$
469 +\EndFor
470 +\end{algorithmic}
471 +\end{algorithm}
472 +
473 +% ----------------------------------------------------------------------------
474 +\section{Self-Organizing Maps}
475 +% ----------------------------------------------------------------------------
476 +
477 +The self-organizing map \cite{kohonen1990} is an unsupervised, competitive
478 +algorithm that projects a high-dimensional data space onto a
479 +low-dimensional discrete lattice — usually a two-dimensional grid — while
480 +preserving topology: inputs close in $\R^{n}$ are mapped to units close on
481 +the lattice. Unlike the projection produced by an autoencoder, the output
482 +space is prescribed and discrete, and the mapping is learned without any
483 +gradient.
484 +
485 +Each unit $i$ of the lattice carries two objects: a fixed position
486 +$\vect{r}_i$ \emph{on the lattice}, and a trainable codebook vector
487 +$\vect{m}_i \in \R^{n}$ living \emph{in the data space}. Learning
488 +alternates two steps for each presented sample $\vect{x}(t)$.
489 +
490 +\paragraph{Competition.} The best matching unit (BMU) is the nearest
491 +codebook vector,
492 +\begin{equation}
493 + c(t) \;=\; \argmin_{i} \bigl\| \vect{x}(t) - \vect{m}_i(t) \bigr\|,
494 + \label{eq:bio-som-bmu}
495 +\end{equation}
496 +a nearest-prototype rule identical to the assignment step of $k$-means.
497 +
498 +\paragraph{Cooperation.} What distinguishes the SOM from $k$-means is that
499 +the winner does not adapt alone. A neighbourhood kernel, measured in
500 +\emph{lattice} coordinates rather than data coordinates, spreads the update
501 +to nearby units:
502 +\begin{equation}
503 + h_{ci}(t)
504 + \;=\;
505 + \exp\!\left(
506 + -\,\frac{\bigl\|\vect{r}_c - \vect{r}_i\bigr\|^{2}}{2\,\sigma^{2}(t)}
507 + \right)
508 + \;\in\; (0, 1],
509 + \label{eq:bio-som-neigh}
510 +\end{equation}
511 +so that $h_{cc} = 1$ and $h_{ci}$ decays with lattice distance. The
512 +codebook update is then a shrinkage of every unit towards the current
513 +sample, weighted by that kernel:
514 +\begin{equation}
515 + \vect{m}_i(t+1)
516 + \;=\;
517 + \vect{m}_i(t)
518 + \;+\;
519 + \alpha(t)\, h_{ci}(t)\,
520 + \bigl[\vect{x}(t) - \vect{m}_i(t)\bigr].
521 + \label{eq:bio-som-update}
522 +\end{equation}
523 +Because \eqref{eq:bio-som-update} couples lattice neighbours to a common
524 +data point, units adjacent on the grid are dragged towards adjacent regions
525 +of the data space. This coupling, and not any explicit constraint, is what
526 +produces topology preservation.
527 +
528 +Both the learning rate and the neighbourhood radius are annealed,
529 +\begin{equation}
530 + \alpha(t) = \alpha_0\, e^{-t/\lambda_{\alpha}},
531 + \qquad
532 + \sigma(t) = \sigma_0\, e^{-t/\lambda_{\sigma}},
533 + \label{eq:bio-som-schedule}
534 +\end{equation}
535 +which splits training into two phases with distinct roles. While $\sigma$
536 +is large the kernel \eqref{eq:bio-som-neigh} is nearly flat and the whole
537 +map moves almost rigidly, unfolding into the data cloud and fixing the
538 +global ordering. As $\sigma$ shrinks towards one lattice unit the updates
539 +become local and the phase turns into fine quantization. Ordering, once
540 +achieved, is never undone — which is the practical reason the annealing
541 +must be slow. Figure~\ref{fig:bio-som} shows one such update.
542 +
543 +\begin{figure}[htbp]
544 + \centering
545 + \begin{tikzpicture}[font=\small]
546 + % ---- input vector -------------------------------------------------
547 + \node[ninput, minimum size=9mm] (x) at (-1.1, 4.5) {$\vect{x}$};
548 + \foreach \r in {1,...,5}{
549 + \draw[black!35, -{Stealth[length=1.6mm]}] (x) -- (1.02, 1.5*\r);
550 + }
551 + \node[etiquette, anchor=north, align=center, text width=2.6cm]
552 + at (-1.1, 3.9) {compared with\\all units,
553 + \eqref{eq:bio-som-bmu}};
554 + % ---- neighbourhood radius (drawn first, so the opaque unit fills
555 + % hide the arcs that would otherwise cross their labels) -------
556 + \draw[dashed, cgate!70!black, thick] (3.0,3.0) circle (2.25cm);
557 + \node[etiquette, fill=white, inner sep=1.5pt, text=cgate!50!black]
558 + at (3.58, 0.83) {$\sigma(t)$};
559 + % ---- lattice ------------------------------------------------------
560 + \foreach \c/\r/\h/\f in {%
561 + 1/1/0.64/53, 1/2/0.80/64, 1/3/0.64/53, 1/4/0.33/31, 1/5/0.11/16,
562 + 2/1/0.80/64, 2/3/0.80/64, 2/4/0.41/37, 2/5/0.14/17,
563 + 3/1/0.64/53, 3/2/0.80/64, 3/3/0.64/53, 3/4/0.33/31, 3/5/0.11/16,
564 + 4/1/0.33/31, 4/2/0.41/37, 4/3/0.33/31, 4/4/0.17/20, 4/5/0.06/12,
565 + 5/1/0.11/16, 5/2/0.14/17, 5/3/0.11/16, 5/4/0.06/12, 5/5/0.02/9}{
566 + \node[neuron, minimum size=9mm, font=\tiny,
567 + fill=chidden!\f, draw=chidden!80!black]
568 + (u\c\r) at (1.5*\c, 1.5*\r) {\h};
569 + }
570 + % ---- best matching unit -------------------------------------------
571 + \node[neuron, minimum size=9mm, font=\tiny, fill=chidden!78,
572 + draw=coutput, line width=1.1pt] (bmu) at (3.0, 3.0) {1.00};
573 + % ---- profile of the update along the winner's row -----------------
574 + \foreach \c/\l in {1/0.80, 2/1.00, 3/0.80, 4/0.41, 5/0.14}{
575 + \draw[cinput!85!black, line width=2.4pt]
576 + (1.5*\c - \l*0.5, 0.15) -- (1.5*\c + \l*0.5, 0.15);
577 + }
578 + \node[etiquette, anchor=east, align=right, text width=2.9cm]
579 + at (0.5, 0.15) {update size\\$\alpha(t)\,h_{ci}(t)$};
580 + % ---- legend -------------------------------------------------------
581 + \node[etiquette, anchor=west, align=left] at (8.2, 3.15)
582 + {\textcolor{coutput}{thick ring}:\\BMU $c$};
583 + \node[etiquette, anchor=west, align=left] at (8.2, 2.15)
584 + {numbers: $h_{ci}(t)$};
585 + \end{tikzpicture}
586 + \caption{One SOM update on a $5 \times 5$ lattice. The sample $\vect{x}$
587 + is compared with every codebook vector, and the winner $c$ (red ring) is
588 + the nearest one, \eqref{eq:bio-som-bmu}. The number inside each unit is
589 + its neighbourhood weight $h_{ci}$ from \eqref{eq:bio-som-neigh} with
590 + $\sigma = 1.5$ lattice units, and the fill intensity is proportional to
591 + it; the dashed circle marks the radius $\sigma(t)$. The bars at the
592 + bottom give the profile of the update magnitude
593 + \eqref{eq:bio-som-update} along the winner's row: the whole
594 + neighbourhood moves towards $\vect{x}$, by an amount decaying with
595 + lattice distance. Note that distances in \eqref{eq:bio-som-neigh} are
596 + measured on the lattice, whereas the competition
597 + \eqref{eq:bio-som-bmu} is decided in the data space.}
598 + \label{fig:bio-som}
599 +\end{figure}
600 +
601 +A batch variant removes the dependence on presentation order by recomputing
602 +every codebook vector as a kernel-weighted mean of the assigned samples,
603 +\begin{equation}
604 + \vect{m}_i
605 + \;=\;
606 + \frac{\sum_{t} h_{c(t)\,i}\; \vect{x}(t)}
607 + {\sum_{t} h_{c(t)\,i}},
608 + \label{eq:bio-som-batch}
609 +\end{equation}
610 +which is to \eqref{eq:bio-som-update} what Lloyd's algorithm is to online
611 +$k$-means. Map quality is assessed by two complementary scalars, the
612 +quantization error and the topographic error,
613 +\begin{equation}
614 + \mathrm{QE} = \frac{1}{N}\sum_{t=1}^{N}
615 + \bigl\|\vect{x}(t) - \vect{m}_{c(t)}\bigr\|,
616 + \qquad
617 + \mathrm{TE} = \frac{1}{N}\sum_{t=1}^{N}
618 + \mathbf{1}\!\left[
619 + \vect{r}_{c_1(t)} \text{ and } \vect{r}_{c_2(t)}
620 + \text{ non-adjacent}
621 + \right],
622 + \label{eq:bio-som-errors}
623 +\end{equation}
624 +where $c_1$ and $c_2$ denote the first and second best matching units. The
625 +pair matters: QE alone is minimized by abandoning topology altogether, and
626 +TE alone by collapsing every codebook vector onto the same point.
627 +Algorithm~\ref{alg:bio-som} states the complete online procedure; replacing
628 +its inner loop by \eqref{eq:bio-som-batch} yields the batch variant.
629 +
630 +\begin{algorithm}[htbp]
631 +\caption{Self-organizing map — online training}
632 +\label{alg:bio-som}
633 +\begin{algorithmic}[1]
634 +\Require data $\{\vect{x}(t)\}_{t=1}^{N}$; lattice positions
635 + $\{\vect{r}_i\}_{i=1}^{M}$; $\alpha_0$, $\sigma_0$, $\lambda_\alpha$,
636 + $\lambda_\sigma$; epochs $E$
637 +\State initialize $\vect{m}_i$ by random sampling of the data, or on the
638 + plane of the two leading principal components
639 +\For{$e = 1, \dots, E$}
640 + \For{each sample $\vect{x}(t)$, in random order}
641 + \State $c \leftarrow \argmin_{i}\|\vect{x}(t) - \vect{m}_i\|$
642 + \Comment{competition \eqref{eq:bio-som-bmu}}
643 + \State $\alpha \leftarrow \alpha_0 e^{-t/\lambda_\alpha}$;\quad
644 + $\sigma \leftarrow \sigma_0 e^{-t/\lambda_\sigma}$
645 + \Comment{annealing \eqref{eq:bio-som-schedule}}
646 + \For{each unit $i = 1, \dots, M$}
647 + \State $h \leftarrow \exp\bigl(-\|\vect{r}_c - \vect{r}_i\|^{2} /
648 + (2\sigma^{2})\bigr)$
649 + \Comment{lattice distance \eqref{eq:bio-som-neigh}}
650 + \State $\vect{m}_i \leftarrow \vect{m}_i
651 + + \alpha\, h\, \bigl(\vect{x}(t) - \vect{m}_i\bigr)$
652 + \Comment{cooperation \eqref{eq:bio-som-update}}
653 + \EndFor
654 + \EndFor
655 + \State monitor QE and TE of \eqref{eq:bio-som-errors}
656 +\EndFor
657 +\State \Return $\{\vect{m}_i\}_{i=1}^{M}$
658 +\end{algorithmic}
659 +\end{algorithm}
660 +
661 +% ----------------------------------------------------------------------------
662 +\section{Reservoir Computing and Echo State Networks}
663 +% ----------------------------------------------------------------------------
664 +
665 +Training a recurrent network by backpropagation through time is expensive
666 +and numerically delicate. Reservoir computing takes a radical shortcut:
667 +keep a large, sparse, randomly initialized and permanently frozen recurrent
668 +network — the reservoir — and train only a linear readout on top of its
669 +states \cite{jaeger2001}. The reservoir serves as a fixed,
670 +high-dimensional, nonlinear expansion of the input history, and the
671 +learning problem that remains is linear regression.
672 +
673 +The echo state network state update, with leak rate $\alpha \in (0,1]$, is
674 +\begin{equation}
675 + \vect{h}_t
676 + \;=\;
677 + (1 - \alpha)\,\vect{h}_{t-1}
678 + \;+\;
679 + \alpha\,\tanh\!\bigl(
680 + \mat{W}_{\mathrm{in}}\,\vect{u}_t
681 + + \mat{W}\,\vect{h}_{t-1}
682 + + \vect{b}
683 + \bigr),
684 + \label{eq:bio-esn-state}
685 +\end{equation}
686 +and the trained readout is the affine map
687 +\begin{equation}
688 + \vect{y}_t \;=\; \mat{W}_{\mathrm{out}}\,
689 + \begin{bmatrix} 1 \\ \vect{u}_t \\ \vect{h}_t \end{bmatrix}.
690 + \label{eq:bio-esn-readout}
691 +\end{equation}
692 +Only $\mat{W}_{\mathrm{out}}$ is learned: $\mat{W}_{\mathrm{in}}$,
693 +$\mat{W}$ and $\vect{b}$ are drawn once at random and never touched. For
694 +this to be sensible the reservoir must forget, so that its state is
695 +asymptotically a function of the input history alone and not of an
696 +arbitrary initial condition.
697 +
698 +\begin{property}[Echo state property]\label{prop:bio-esp}
699 +The reservoir \eqref{eq:bio-esn-state} has the echo state property when,
700 +for any two initial states $\vect{h}_0$ and $\vect{h}'_0$ driven by the
701 +same input sequence, $\|\vect{h}_t - \vect{h}'_t\| \to 0$ as
702 +$t \to \infty$. A sufficient condition is $\sigma_{\max}(\mat{W}) < 1$; in
703 +practice one uses the far less conservative spectral-radius heuristic
704 +\begin{equation}
705 + \rho(\mat{W}) \;=\; \max_{i} \bigl|\lambda_i(\mat{W})\bigr| \;<\; 1,
706 + \label{eq:bio-esn-spectral}
707 +\end{equation}
708 +tuning $\rho$ just below unity. The reservoir is then a \emph{fading
709 +memory} filter: larger $\rho$ lengthens the memory, smaller $\rho$ shortens
710 +it, and $\rho$ slightly above one often still works on strongly driven
711 +tasks.
712 +\end{property}
713 +
714 +Condition \eqref{eq:bio-esn-spectral} is enforced by construction rather
715 +than by optimization. One draws a sparse random matrix
716 +$\mat{W}_{\mathrm{raw}}$ — typically one to five per cent of entries
717 +nonzero — computes its spectral radius numerically, and rescales:
718 +\begin{equation}
719 + \mat{W}
720 + \;\leftarrow\;
721 + \frac{\rho^{\star}}{\rho(\mat{W}_{\mathrm{raw}})}\; \mat{W}_{\mathrm{raw}},
722 + \qquad \rho^{\star} \lesssim 1 .
723 + \label{eq:bio-esn-rescale}
724 +\end{equation}
725 +
726 +The decisive consequence of freezing the recurrence is that the training
727 +objective becomes convex. Collecting the extended states of all retained
728 +time steps into $\mat{H} \in \R^{(1 + n_u + n_h) \times T}$ and the targets
729 +into $\mat{Y} \in \R^{n_y \times T}$, the ridge-regularized least-squares
730 +solution is available in closed form:
731 +\begin{equation}
732 + \mat{W}_{\mathrm{out}}
733 + \;=\;
734 + \argmin_{\mat{W}}
735 + \left\{
736 + \bigl\|\mat{W}\mat{H} - \mat{Y}\bigr\|_{F}^{2}
737 + + \lambda \bigl\|\mat{W}\bigr\|_{F}^{2}
738 + \right\}
739 + \;=\;
740 + \mat{Y}\mat{H}\transp
741 + \bigl(\mat{H}\mat{H}\transp + \lambda \mat{I}\bigr)^{-1}.
742 + \label{eq:bio-esn-ridge}
743 +\end{equation}
744 +There is no local minimum, no learning rate, no vanishing gradient and no
745 +backward pass through time: one matrix inversion of size
746 +$(1 + n_u + n_h)$ replaces the entire training procedure. The ridge term is
747 +not optional — reservoir states are strongly correlated, so
748 +$\mat{H}\mat{H}\transp$ is badly conditioned.
749 +Algorithm~\ref{alg:bio-esn} gives the complete recipe, and
750 +Figure~\ref{fig:bio-esn} the architecture.
751 +
752 +\begin{figure}[htbp]
753 + \centering
754 + \begin{tikzpicture}[font=\small]
755 + % ---- reservoir boundary -------------------------------------------
756 + \draw[dashed, black!55, thick]
757 + (4,0) ellipse [x radius=2.4cm, y radius=1.8cm];
758 + % ---- reservoir units ----------------------------------------------
759 + \node[nhidden, minimum size=6mm] (n1) at (2.55, 0.55) {};
760 + \node[nhidden, minimum size=6mm] (n2) at (2.55,-0.65) {};
761 + \node[nhidden, minimum size=6mm] (n3) at (3.60, 1.15) {};
762 + \node[nhidden, minimum size=6mm] (n4) at (3.75,-1.15) {};
763 + \node[nhidden, minimum size=6mm] (n5) at (4.10, 0.05) {};
764 + \node[nhidden, minimum size=6mm] (n6) at (4.95, 1.05) {};
765 + \node[nhidden, minimum size=6mm] (n7) at (5.15,-0.85) {};
766 + \node[nhidden, minimum size=6mm] (n8) at (5.55, 0.10) {};
767 + % ---- fixed sparse recurrent connections ---------------------------
768 + \begin{scope}[black!40, -{Stealth[length=1.6mm]}, semithick]
769 + \draw (n1) -- (n5);
770 + \draw (n5) to[bend left=15] (n3);
771 + \draw (n3) -- (n6);
772 + \draw (n6) to[bend left=12] (n8);
773 + \draw (n8) -- (n7);
774 + \draw (n7) -- (n4);
775 + \draw (n4) to[bend left=12] (n2);
776 + \draw (n2) -- (n1);
777 + \draw (n5) to[bend right=15] (n7);
778 + \draw (n6) to[bend left=18] (n5);
779 + \draw (n2) to[bend right=12] (n5);
780 + \draw (n5) to[out=-115, in=-65, looseness=6] (n5);
781 + \end{scope}
782 + % ---- input --------------------------------------------------------
783 + \node[ninput, minimum size=9mm] (u) at (0,0) {$\vect{u}_t$};
784 + \draw[fleche, black!55] (u) -- (n1);
785 + \draw[fleche, black!55] (u) -- (n2);
786 + \node[etiquette, anchor=east] at (1.68, 0.80)
787 + {$\mat{W}_{\mathrm{in}}$ (fixed)};
788 + % ---- readout ------------------------------------------------------
789 + \node[noutput, minimum size=9mm] (y) at (8.7,0) {$\vect{y}_t$};
790 + \draw[fleche, coutput, line width=1.1pt] (n6) -- (y);
791 + \draw[fleche, coutput, line width=1.1pt] (n8) -- (y);
792 + \draw[fleche, coutput, line width=1.1pt] (n7) -- (y);
793 + \node[etiquette, anchor=south, text=coutput!85!black] at (7.4, 0.80)
794 + {$\mat{W}_{\mathrm{out}}$ (\textbf{trained})};
795 + % ---- titles -------------------------------------------------------
796 + \node[anchor=south, font=\small] at (4, 2.05)
797 + {Reservoir: $n_h$ fixed random units};
798 + \node[etiquette, anchor=north, align=center] at (4, -2.05)
799 + {$\mat{W}$ sparse, random, rescaled to
800 + $\rho(\mat{W}) = \rho^{\star} < 1$, \eqref{eq:bio-esn-rescale}};
801 + \end{tikzpicture}
802 + \caption{An echo state network. The input weights
803 + $\mat{W}_{\mathrm{in}}$ and the sparse recurrent matrix $\mat{W}$ (grey)
804 + are drawn at random and \emph{never trained}; they merely expand the
805 + input history into a high-dimensional dynamical state
806 + \eqref{eq:bio-esn-state}, whose fading memory is guaranteed by the
807 + spectral-radius condition \eqref{eq:bio-esn-spectral}. Only the readout
808 + $\mat{W}_{\mathrm{out}}$ (red) is learned, and it is learned in closed
809 + form by ridge regression \eqref{eq:bio-esn-ridge}. For legibility only
810 + three readout connections are drawn; in fact every reservoir state, and
811 + the input itself, feeds the readout \eqref{eq:bio-esn-readout}.}
812 + \label{fig:bio-esn}
813 +\end{figure}
814 +
815 +\begin{algorithm}[htbp]
816 +\caption{Echo state network — construction and closed-form training}
817 +\label{alg:bio-esn}
818 +\begin{algorithmic}[1]
819 +\Require sequence $\{(\vect{u}_t, \vect{y}^{\star}_t)\}_{t=1}^{T}$;
820 + reservoir size $n_h$; sparsity $s$; target radius $\rho^{\star}$; leak
821 + $\alpha$; ridge $\lambda$; washout $T_0$
822 +\Statex \textit{1. build the reservoir, once, at random}
823 +\State draw $\mat{W}_{\mathrm{in}} \in \R^{n_h \times n_u}$ with i.i.d.\
824 + entries uniform on $[-a, a]$
825 +\State draw $\mat{W}_{\mathrm{raw}} \in \R^{n_h \times n_h}$ with a fraction
826 + $s$ of nonzero i.i.d.\ entries
827 +\State $\mat{W} \leftarrow
828 + \bigl(\rho^{\star}/\rho(\mat{W}_{\mathrm{raw}})\bigr)\,
829 + \mat{W}_{\mathrm{raw}}$
830 + \Comment{enforce \eqref{eq:bio-esn-spectral} by
831 + \eqref{eq:bio-esn-rescale}}
832 +\Statex \textit{2. drive the reservoir and harvest its states}
833 +\State $\vect{h}_0 \leftarrow \vect{0}$
834 +\For{$t = 1, \dots, T$}
835 + \State $\vect{h}_t \leftarrow (1-\alpha)\,\vect{h}_{t-1}
836 + + \alpha \tanh\bigl(\mat{W}_{\mathrm{in}}\vect{u}_t
837 + + \mat{W}\vect{h}_{t-1} + \vect{b}\bigr)$
838 + \Comment{\eqref{eq:bio-esn-state}}
839 + \If{$t > T_0$}
840 + \State append $[\,1;\, \vect{u}_t;\, \vect{h}_t\,]$ as a column of
841 + $\mat{H}$, and $\vect{y}^{\star}_t$ as a column of $\mat{Y}$
842 + \EndIf
843 +\EndFor
844 +\Statex \textit{3. solve one convex problem}
845 +\State $\mat{W}_{\mathrm{out}} \leftarrow \mat{Y}\mat{H}\transp
846 + \bigl(\mat{H}\mat{H}\transp + \lambda\mat{I}\bigr)^{-1}$
847 + \Comment{ridge regression \eqref{eq:bio-esn-ridge}}
848 +\State \Return $\mat{W}_{\mathrm{in}}$, $\mat{W}$, $\mat{W}_{\mathrm{out}}$
849 +\end{algorithmic}
850 +\end{algorithm}
851 +
852 +The discarded prefix of length $T_0$ in Algorithm~\ref{alg:bio-esn} is not
853 +a detail: by Property~\ref{prop:bio-esp} the influence of $\vect{h}_0$
854 +decays only asymptotically, so the earliest states are contaminated by the
855 +arbitrary initial condition and would bias the regression. Echo state
856 +networks are competitive on chaotic time-series prediction, at a training
857 +cost orders of magnitude below that of a fully trained recurrent network.
858 +
859 +\begin{remark}[Liquid state machines]\label{rem:bio-lsm}
860 +The liquid state machine is the spiking counterpart of the echo state
861 +network: the reservoir is a recurrent population of leaky
862 +integrate-and-fire neurons in the sense of Definition~\ref{def:bio-lif} —
863 +the liquid — and the readout is trained on its instantaneous state.
864 +Real-time universal computing power follows from two conditions: a
865 +\emph{separation property} for the liquid, meaning that different input
866 +histories drive measurably different states, and an \emph{approximation
867 +property} for the readout. The idea has outlived its original setting in
868 +\emph{physical} reservoir computing, where the fixed nonlinear dynamics are
869 +supplied by photonic, memristive or even mechanical substrates rather than
870 +by simulated neurons.
871 +\end{remark}
872 +
873 +% ----------------------------------------------------------------------------
874 +\section{What Is Learned, What Is Fixed, What Emerges}
875 +% ----------------------------------------------------------------------------
876 +
877 +The three families of this chapter occupy three distinct positions on the
878 +question of where the adaptivity of a network should reside;
879 +Table~\ref{tab:bio-comparison} summarizes them.
880 +
881 +\begin{table}[htbp]
882 + \centering
883 + \small
884 + \caption{The three biologically inspired paradigms compared. None of them
885 + uses a global gradient of a supervised loss in its native form.}
886 + \label{tab:bio-comparison}
887 + \begin{tabular}{@{}lllll@{}}
888 + \toprule
889 + & \textbf{Fixed} & \textbf{Learned} & \textbf{Learning signal}
890 + & \textbf{Emergent property} \\
891 + \midrule
892 + SNN with STDP
893 + & neuron dynamics
894 + & all synapses
895 + & local spike timing
896 + & causality detection \\
897 + SOM
898 + & lattice topology
899 + & codebook $\vect{m}_i$
900 + & competition, no error
901 + & topology preservation \\
902 + ESN
903 + & $\mat{W}_{\mathrm{in}}$, $\mat{W}$
904 + & readout only
905 + & convex least squares
906 + & fading memory \\
907 + \bottomrule
908 + \end{tabular}
909 +\end{table}
910 +
911 +\begin{remark}[The common thread]\label{rem:bio-thread}
912 +Each paradigm buys something by giving up end-to-end differentiability.
913 +Spiking networks buy energy: under temporal coding a neuron may transmit
914 +its message with a single event, which is why these models map so naturally
915 +onto event-driven neuromorphic silicon. Self-organizing maps buy
916 +interpretability: the lattice is a fixed, inspectable output space, so the
917 +learned representation can be read off directly rather than probed.
918 +Reservoir computing buys convexity, and with it a training procedure that
919 +is exact, fast and free of the pathologies of backpropagation through time.
920 +The price, uniformly, is expressive power on large supervised problems —
921 +precisely where the gradient-trained architectures of the preceding
922 +chapters remain unmatched. The surrogate-gradient method of
923 +\eqref{eq:bio-surrogate} is the most successful attempt so far to have
924 +both, restoring differentiability to the spiking forward pass without
925 +abandoning its event-driven semantics.
926 +\end{remark}
added chapters/13-emerging.tex +731 −0
@@ -0,0 +1,731 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and Graphical
3 +% Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Chapter 13 : Emerging Architectures (chapters/13-emerging.tex)
6 +% ============================================================================
7 +\chapter{Emerging Architectures: Capsules, Neural ODEs, Memory-Augmented
8 + Networks and KANs}\label{chap:emerging}
9 +
10 +The architectures of the preceding chapters share a common template: fixed
11 +computational graphs whose edges carry scalar weights and whose nodes apply
12 +fixed nonlinearities. This final chapter examines four families that each
13 +break the template along a different axis. \emph{Capsule networks} replace
14 +scalar neurons by vector-valued units routed by iterative agreement.
15 +\emph{Neural ordinary differential equations} replace discrete layers by a
16 +continuous-time flow whose depth is chosen by a numerical solver.
17 +\emph{Memory-augmented networks} couple a neural controller to an external,
18 +differentiable random-access memory. \emph{Kolmogorov--Arnold networks}
19 +move the learnable nonlinearity from the nodes onto the edges themselves.
20 +In every case we follow the discipline of this book: the defining
21 +equations, the estimation algorithm, and a faithful graphical
22 +representation.
23 +
24 +% ============================================================================
25 +\section{Capsule Networks and Routing by Agreement}
26 +\label{sec:em-capsules}
27 +
28 +Convolutional networks achieve tolerance to small translations through
29 +pooling, but in doing so they discard the precise spatial relationships
30 +between parts — a face detector built on max-pooling responds to a jumble
31 +of eyes and mouth almost as strongly as to a correctly arranged face.
32 +Capsule networks \cite{sabour2017} address this by making the network's
33 +elementary unit a \emph{vector} rather than a scalar.
34 +
35 +\begin{definition}[Capsule]\label{def:em-capsule}
36 +A \emph{capsule} is a group of neurons whose activity vector
37 +$\vect{u} \in \R^{d}$ encodes the instantiation parameters (pose,
38 +deformation, texture) of an entity, and whose length
39 +$\lVert\vect{u}\rVert \in [0,1)$ encodes the probability that the entity
40 +is present in the input.
41 +\end{definition}
42 +
43 +\subsection{Prediction, coupling and squashing}
44 +
45 +Let $\vect{u}_i$ be the output of capsule $i$ in layer $\ell$. Capsule $i$
46 +casts a \emph{vote} for the pose of each capsule $j$ in layer $\ell+1$
47 +through a learned transformation matrix $\mat{W}_{ij}$, and the votes are
48 +aggregated with data-dependent \emph{coupling coefficients} $c_{ij}$:
49 +\begin{align}
50 + \hat{\vect{u}}_{j|i} &= \mat{W}_{ij}\,\vect{u}_i,
51 + \label{eq:em-caps-pred}\\
52 + \vect{s}_j &= \sum_i c_{ij}\,\hat{\vect{u}}_{j|i}.
53 + \label{eq:em-caps-sum}
54 +\end{align}
55 +The couplings are a softmax over \emph{routing logits} $b_{ij}$,
56 +initialized to zero, so that each lower capsule distributes a unit budget
57 +of attention across the capsules of the next layer:
58 +\begin{equation}
59 + c_{ij} \;=\; \frac{\exp(b_{ij})}{\sum_k \exp(b_{ik})}.
60 + \label{eq:em-caps-coupling}
61 +\end{equation}
62 +The total input $\vect{s}_j$ is converted to an output vector by the
63 +\emph{squashing} nonlinearity, which preserves orientation while mapping
64 +length into $[0,1)$ so it can act as a probability:
65 +\begin{equation}
66 + \vect{v}_j \;=\;
67 + \frac{\lVert\vect{s}_j\rVert^2}{1+\lVert\vect{s}_j\rVert^2}\,
68 + \frac{\vect{s}_j}{\lVert\vect{s}_j\rVert}.
69 + \label{eq:em-caps-squash}
70 +\end{equation}
71 +
72 +\subsection{Dynamic routing as an estimation algorithm}
73 +
74 +The logits are refined by \emph{routing by agreement}: a vote that aligns
75 +with the emerging consensus $\vect{v}_j$ has its coupling reinforced,
76 +\begin{equation}
77 + b_{ij} \;\leftarrow\; b_{ij} + \hat{\vect{u}}_{j|i} \cdot \vect{v}_j,
78 + \label{eq:em-caps-agree}
79 +\end{equation}
80 +which is a soft clustering in pose space: parts whose predicted poses
81 +agree are assigned to the same whole. Algorithm~\ref{alg:em-routing}
82 +summarizes the procedure of \cite{sabour2017}, typically run for $r=3$
83 +iterations between consecutive capsule layers.
84 +
85 +\begin{algorithm}[t]
86 +\caption{Dynamic routing by agreement \cite{sabour2017}}
87 +\label{alg:em-routing}
88 +\begin{algorithmic}[1]
89 +\Require prediction vectors $\hat{\vect{u}}_{j|i}$ from
90 + \eqref{eq:em-caps-pred}, number of iterations $r$
91 +\Ensure output capsules $\vect{v}_j$
92 +\State $b_{ij} \gets 0$ for all $i, j$
93 +\For{$t = 1, \dots, r$}
94 + \State $c_{ij} \gets \softmax_j(b_{ij})$
95 + \Comment{coupling coefficients, \eqref{eq:em-caps-coupling}}
96 + \State $\vect{s}_j \gets \sum_i c_{ij}\,\hat{\vect{u}}_{j|i}$
97 + \Comment{weighted vote aggregation, \eqref{eq:em-caps-sum}}
98 + \State $\vect{v}_j \gets
99 + \dfrac{\lVert\vect{s}_j\rVert^2}{1+\lVert\vect{s}_j\rVert^2}\,
100 + \dfrac{\vect{s}_j}{\lVert\vect{s}_j\rVert}$
101 + \Comment{squash, \eqref{eq:em-caps-squash}}
102 + \State $b_{ij} \gets b_{ij} + \hat{\vect{u}}_{j|i}\cdot\vect{v}_j$
103 + \Comment{agreement update, \eqref{eq:em-caps-agree}}
104 +\EndFor
105 +\State \Return $\vect{v}_j$
106 +\end{algorithmic}
107 +\end{algorithm}
108 +
109 +Figure~\ref{fig:em-capsules} shows two capsule layers after routing has
110 +converged: the width and opacity of each arrow is proportional to the
111 +coupling coefficient $c_{ij}$, so the diagram makes visible the
112 +part--whole assignment that \eqref{eq:em-caps-agree} computes.
113 +
114 +\begin{figure}[t]
115 + \centering
116 + \begin{tikzpicture}[scale=1.0]
117 + % primary capsules
118 + \node[nhidden, minimum size=9mm] (u1) at (0, 1.7) {$\vect{u}_1$};
119 + \node[nhidden, minimum size=9mm] (u2) at (0, 0.0) {$\vect{u}_2$};
120 + \node[nhidden, minimum size=9mm] (u3) at (0,-1.7) {$\vect{u}_3$};
121 + % output capsules
122 + \node[noutput, minimum size=10mm] (v1) at (5.6, 0.9) {$\vect{v}_1$};
123 + \node[noutput, minimum size=10mm] (v2) at (5.6,-0.9) {$\vect{v}_2$};
124 + % couplings: widths proportional to c_ij
125 + \draw[fleche, line width=1.9pt, cinput!90!black]
126 + (u1) -- (v1)
127 + node[pos=0.40, above=1pt, sloped, font=\scriptsize, text=black]
128 + {$\hat{\vect{u}}_{1|1}$};
129 + \draw[fleche, line width=0.5pt, cinput!90!black, opacity=0.30]
130 + (u1) -- (v2);
131 + \draw[fleche, line width=1.5pt, cinput!90!black, opacity=0.85]
132 + (u2) -- (v1);
133 + \draw[fleche, line width=0.7pt, cinput!90!black, opacity=0.40]
134 + (u2) -- (v2);
135 + \draw[fleche, line width=0.5pt, cinput!90!black, opacity=0.30]
136 + (u3) -- (v1);
137 + \draw[fleche, line width=1.9pt, cinput!90!black]
138 + (u3) -- (v2);
139 + % annotations
140 + \node[etiquette, align=center] at (0, 2.85)
141 + {primary capsules\\(layer $\ell$)};
142 + \node[etiquette, align=center] at (5.6, 2.05)
143 + {output capsules\\(layer $\ell+1$)};
144 + \node[etiquette, align=left, anchor=west] at (6.55, 0.9)
145 + {$\lVert\vect{v}_1\rVert \approx 1$:\\entity present};
146 + \node[etiquette, align=left, anchor=west] at (6.55,-0.9)
147 + {$\lVert\vect{v}_2\rVert \approx 1$:\\entity present};
148 + \node[etiquette, align=center] at (2.8,-2.75)
149 + {arrow width $\propto$ coupling coefficient $c_{ij}$
150 + of \eqref{eq:em-caps-coupling}};
151 + \end{tikzpicture}
152 + \caption{Two capsule layers after routing by agreement
153 + (Algorithm~\ref{alg:em-routing}). Each primary capsule sends a
154 + prediction vector $\hat{\vect{u}}_{j|i} = \mat{W}_{ij}\vect{u}_i$ to
155 + every output capsule; iterating
156 + \eqref{eq:em-caps-coupling}--\eqref{eq:em-caps-agree} concentrates the
157 + couplings (arrow width) on the wholes whose pose the parts agree on:
158 + here $\vect{u}_1, \vect{u}_2$ are routed to $\vect{v}_1$ and
159 + $\vect{u}_3$ to $\vect{v}_2$.}
160 + \label{fig:em-capsules}
161 +\end{figure}
162 +
163 +\subsection{Margin loss}
164 +
165 +Since capsule lengths are probabilities, classification uses a per-class
166 +\emph{margin loss}: with $T_k = 1$ if class $k$ is present and $0$
167 +otherwise,
168 +\begin{equation}
169 + L_k \;=\; T_k \max\bigl(0,\, m^{+} - \lVert\vect{v}_k\rVert\bigr)^2
170 + \;+\; \lambda\,(1-T_k)\max\bigl(0,\,
171 + \lVert\vect{v}_k\rVert - m^{-}\bigr)^2,
172 + \label{eq:em-caps-margin}
173 +\end{equation}
174 +with $m^{+}=0.9$, $m^{-}=0.1$ and $\lambda = 0.5$ in \cite{sabour2017};
175 +the total loss $\Loss = \sum_k L_k$ is complemented by a small
176 +reconstruction penalty that regularizes the capsule poses.
177 +
178 +\begin{remark}
179 +Routing replaces pooling: instead of discarding position information, the
180 +network performs an explicit, differentiable assignment of parts to
181 +wholes. The price is computational — routing is an inner-loop iteration
182 +per layer and per example — and capsule networks have not yet scaled
183 +beyond mid-sized benchmarks, but the equivariance principle they embody
184 +continues to influence architecture design.
185 +\end{remark}
186 +
187 +% ============================================================================
188 +\section{Neural Ordinary Differential Equations}
189 +\label{sec:em-node}
190 +
191 +\subsection{From residual blocks to continuous depth}
192 +
193 +A residual block computes
194 +$\vect{h}_{t+1} = \vect{h}_t + f(\vect{h}_t, \theta_t)$. Reading the layer
195 +index $t$ as time, this is exactly one step of the explicit Euler scheme
196 +with unit step size. Neural ODEs \cite{chen2018} take the continuous
197 +limit: the hidden state is defined as the solution of an initial value
198 +problem,
199 +\begin{equation}
200 + \frac{d\vect{h}(t)}{dt} = f\bigl(\vect{h}(t), t, \theta\bigr),
201 + \qquad
202 + \vect{h}(t_1) = \vect{h}(t_0)
203 + + \int_{t_0}^{t_1} f\bigl(\vect{h}(t), t, \theta\bigr)\,dt,
204 + \label{eq:em-node-dynamics}
205 +\end{equation}
206 +evaluated by a black-box solver
207 +$\vect{h}(t_1) = \mathrm{ODESolve}(\vect{h}(t_0), f, t_0, t_1, \theta)$
208 +such as an adaptive Runge--Kutta method.
209 +
210 +\begin{remark}[ResNet as Euler discretization]\label{rem:em-euler}
211 +With step size $\Delta t$, the Euler scheme applied to
212 +\eqref{eq:em-node-dynamics} reads
213 +\begin{equation}
214 + \vect{h}_{t+1} \;=\; \vect{h}_t
215 + + \Delta t\, f(\vect{h}_t, t, \theta),
216 + \label{eq:em-node-euler}
217 +\end{equation}
218 +which for $\Delta t = 1$ is precisely the residual block of a ResNet. A
219 +neural ODE is thus a residual network whose number of layers — the number
220 +of solver steps — is chosen adaptively at run time, trading accuracy
221 +against compute without retraining.
222 +\end{remark}
223 +
224 +\subsection{The adjoint method}
225 +
226 +Naively backpropagating through every internal solver step would store all
227 +intermediate states. The \emph{adjoint sensitivity method} avoids this.
228 +Define the adjoint state as the sensitivity of the loss to the hidden
229 +state at each instant,
230 +\begin{equation}
231 + \vect{a}(t) \;=\; \frac{\partial \Loss}{\partial \vect{h}(t)}.
232 + \label{eq:em-node-adjoint}
233 +\end{equation}
234 +The adjoint obeys its own linear ODE, integrated \emph{backwards} in time
235 +from $\vect{a}(t_1) = \partial\Loss/\partial\vect{h}(t_1)$:
236 +\begin{equation}
237 + \frac{d\vect{a}(t)}{dt}
238 + \;=\; -\,\vect{a}(t)\transp\,
239 + \frac{\partial f\bigl(\vect{h}(t), t, \theta\bigr)}{\partial \vect{h}},
240 + \label{eq:em-node-adjoint-ode}
241 +\end{equation}
242 +and the parameter gradient is obtained by a single quadrature along the
243 +same backward pass:
244 +\begin{equation}
245 + \frac{d\Loss}{d\theta}
246 + \;=\; -\int_{t_1}^{t_0} \vect{a}(t)\transp\,
247 + \frac{\partial f\bigl(\vect{h}(t), t, \theta\bigr)}{\partial \theta}
248 + \,dt.
249 + \label{eq:em-node-grad}
250 +\end{equation}
251 +In practice one concatenates the state, the adjoint and the accumulating
252 +gradient into a single \emph{augmented state} and makes one backward
253 +solver call:
254 +\begin{equation}
255 + \frac{d}{dt}
256 + \begin{bmatrix} \vect{h}(t)\\[2pt] \vect{a}(t)\\[2pt]
257 + \dfrac{d\Loss}{d\theta}(t) \end{bmatrix}
258 + =
259 + \begin{bmatrix} f(\vect{h}, t, \theta)\\[2pt]
260 + -\vect{a}\transp\,\partial f/\partial\vect{h}\\[2pt]
261 + -\vect{a}\transp\,\partial f/\partial\theta \end{bmatrix},
262 + \label{eq:em-node-augmented}
263 +\end{equation}
264 +where the vector--Jacobian products are computed by ordinary reverse-mode
265 +automatic differentiation of $f$ alone.
266 +Algorithm~\ref{alg:em-adjoint} assembles the full gradient computation.
267 +
268 +\begin{property}[Constant memory in depth]\label{prop:em-memory}
269 +Training with
270 +\eqref{eq:em-node-adjoint-ode}--\eqref{eq:em-node-augmented} requires
271 +storing only the endpoint states, so the memory cost of a neural ODE is
272 +$O(1)$ in the effective depth, whereas backpropagation through an
273 +$L$-layer residual network stores $O(L)$ activations. The hidden
274 +trajectory $\vect{h}(t)$ needed inside \eqref{eq:em-node-adjoint-ode} is
275 +recovered on the fly by integrating \eqref{eq:em-node-dynamics}
276 +backwards alongside the adjoint.
277 +\end{property}
278 +
279 +\begin{algorithm}[t]
280 +\caption{Adjoint-method gradient for a neural ODE \cite{chen2018}}
281 +\label{alg:em-adjoint}
282 +\begin{algorithmic}[1]
283 +\Require dynamics $f_\theta$, initial state $\vect{h}(t_0)$, loss $\Loss$
284 +\State $\vect{h}(t_1) \gets
285 + \mathrm{ODESolve}\bigl(\vect{h}(t_0), f, t_0, t_1, \theta\bigr)$
286 + \Comment{forward solve of \eqref{eq:em-node-dynamics}}
287 +\State $\vect{a}(t_1) \gets \partial\Loss/\partial\vect{h}(t_1)$
288 + \Comment{terminal adjoint, \eqref{eq:em-node-adjoint}}
289 +\State $\vect{s}(t_1) \gets
290 + \bigl[\vect{h}(t_1),\ \vect{a}(t_1),\ \vect{0}\bigr]$
291 + \Comment{augmented state}
292 +\State $\bigl[\vect{h}(t_0), \vect{a}(t_0), d\Loss/d\theta\bigr] \gets
293 + \mathrm{ODESolve}\bigl(\vect{s}(t_1),
294 + \text{dynamics \eqref{eq:em-node-augmented}}, t_1, t_0\bigr)$
295 + \Comment{backward solve}
296 +\State \Return $d\Loss/d\theta$ \ and \
297 + $\vect{a}(t_0) = \partial\Loss/\partial\vect{h}(t_0)$
298 +\end{algorithmic}
299 +\end{algorithm}
300 +
301 +Figure~\ref{fig:em-node} contrasts the two views of depth: the discrete
302 +Euler/ResNet staircase of \eqref{eq:em-node-euler} against the smooth flow
303 +of \eqref{eq:em-node-dynamics}, drawn over the vector field
304 +$f(h) = 0.8\,h\,(1-h/3)$ that both are following.
305 +
306 +\begin{figure}[t]
307 + \centering
308 + \begin{tikzpicture}
309 + \begin{axis}[
310 + width=0.82\textwidth, height=6.6cm,
311 + xlabel={$t$ (depth)}, ylabel={$h(t)$},
312 + xmin=-0.15, xmax=4.45, ymin=0, ymax=3.15,
313 + legend style={at={(0.97,0.06)}, anchor=south east,
314 + font=\scriptsize, draw=black!30},
315 + tick label style={font=\scriptsize},
316 + label style={font=\small},
317 + ]
318 + % faint vector field of the dynamics f(h)=0.8 h (1-h/3)
319 + \foreach \t in {0.1,0.6,1.1,1.6,2.1,2.6,3.1,3.6,4.1}{
320 + \foreach \h in {0.35,0.75,1.15,1.55,1.95,2.35,2.75}{
321 + \edef\temp{\noexpand\draw[-{Stealth[length=1.0mm]},
322 + black!25, thin]
323 + (axis cs:\t,\h) --
324 + (axis cs:{\t+0.16},{\h+0.16*0.8*\h*(1-\h/3)});}
325 + \temp
326 + }
327 + }
328 + % continuous ODE trajectory (logistic solution)
329 + \addplot[cinput, very thick, smooth, domain=0:4.3, samples=80]
330 + {3/(1+5*exp(-0.8*x))};
331 + \addlegendentry{continuous flow, \eqref{eq:em-node-dynamics}}
332 + % Euler / ResNet steps, dt = 1
333 + \addplot[chidden!80!black, thick, dashed,
334 + mark=*, mark options={solid, fill=chidden}]
335 + coordinates {(0,0.5) (1,0.8333) (2,1.3148) (3,1.9056)
336 + (4,2.4617)};
337 + \addlegendentry{Euler / ResNet steps, $\Delta t = 1$,
338 + \eqref{eq:em-node-euler}}
339 + \end{axis}
340 + \end{tikzpicture}
341 + \caption{A neural ODE as the continuous limit of a residual network.
342 + Grey arrows: the learned vector field $f(h,t,\theta)$. Blue: the
343 + exact trajectory of \eqref{eq:em-node-dynamics}. Orange: the Euler
344 + discretization \eqref{eq:em-node-euler} with $\Delta t = 1$, i.e.\ a
345 + four-block residual network following the same field with
346 + accumulating discretization error. An adaptive solver places as many
347 + evaluations as the requested tolerance demands.}
348 + \label{fig:em-node}
349 +\end{figure}
350 +
351 +% ============================================================================
352 +\section{Memory-Augmented Networks: Neural Turing Machines}
353 +\label{sec:em-ntm}
354 +
355 +Recurrent networks store all their knowledge in a fixed-size hidden state.
356 +Neural Turing Machines \cite{graves2014} decouple computation from
357 +storage: a controller network (typically an LSTM) interacts with an
358 +external memory matrix $\mat{M}_t \in \R^{N\times W}$ ($N$ locations of
359 +width $W$) through read and write \emph{heads} that are differentiable
360 +end-to-end, so the whole system is trained by ordinary gradient descent.
361 +
362 +\subsection{Reading and writing}
363 +
364 +Every interaction is mediated by a normalized attention weighting
365 +$\vect{w}_t \in \R^N$, $\sum_i w_t(i) = 1$, over memory locations. Reading
366 +returns the weighted average of the rows,
367 +\begin{equation}
368 + \vect{r}_t \;=\; \sum_{i=1}^{N} w_t(i)\,\mat{M}_t(i),
369 + \label{eq:em-ntm-read}
370 +\end{equation}
371 +and writing decomposes, like an LSTM gate pair, into an erase followed by
372 +an add, with $\vect{e}_t \in [0,1]^W$ and $\vect{a}_t \in \R^W$ emitted by
373 +the controller:
374 +\begin{align}
375 + \tilde{\mat{M}}_t(i) &= \mat{M}_{t-1}(i) \odot
376 + \bigl[\vect{1} - w_t(i)\,\vect{e}_t\bigr],
377 + \label{eq:em-ntm-erase}\\
378 + \mat{M}_t(i) &= \tilde{\mat{M}}_t(i) + w_t(i)\,\vect{a}_t.
379 + \label{eq:em-ntm-add}
380 +\end{align}
381 +
382 +\subsection{The differentiable addressing pipeline}
383 +
384 +The weighting $\vect{w}_t$ is produced by a four-stage pipeline that
385 +blends content-based and location-based addressing. First, the controller
386 +emits a key $\vect{k}_t$ compared to every row by cosine similarity,
387 +\begin{equation}
388 + K[\vect{u},\vect{v}] \;=\;
389 + \frac{\vect{u}\cdot\vect{v}}{\lVert\vect{u}\rVert\,
390 + \lVert\vect{v}\rVert},
391 + \label{eq:em-ntm-cosine}
392 +\end{equation}
393 +sharpened by a scalar $\beta_t > 0$ and normalized:
394 +\begin{equation}
395 + w_t^{c}(i) \;=\;
396 + \frac{\exp\bigl(\beta_t\,K[\vect{k}_t, \mat{M}_t(i)]\bigr)}
397 + {\sum_j \exp\bigl(\beta_t\,K[\vect{k}_t, \mat{M}_t(j)]\bigr)}.
398 + \label{eq:em-ntm-content}
399 +\end{equation}
400 +The content weighting is then interpolated with the previous weighting by
401 +a gate $g_t \in [0,1]$,
402 +\begin{equation}
403 + \vect{w}_t^{g} \;=\; g_t\,\vect{w}_t^{c}
404 + + (1-g_t)\,\vect{w}_{t-1},
405 + \label{eq:em-ntm-interp}
406 +\end{equation}
407 +rotated by a circular convolution with a shift distribution $\vect{s}_t$
408 +(location-based addressing, enabling ``move one slot to the right''),
409 +\begin{equation}
410 + \tilde{w}_t(i) \;=\; \sum_{j=0}^{N-1} w_t^{g}(j)\,
411 + s_t\bigl((i-j) \bmod N\bigr),
412 + \label{eq:em-ntm-shift}
413 +\end{equation}
414 +and finally re-sharpened with $\gamma_t \geq 1$ to undo the blurring
415 +introduced by the convolution:
416 +\begin{equation}
417 + w_t(i) \;=\;
418 + \frac{\tilde{w}_t(i)^{\gamma_t}}{\sum_j \tilde{w}_t(j)^{\gamma_t}}.
419 + \label{eq:em-ntm-sharpen}
420 +\end{equation}
421 +Algorithm~\ref{alg:em-ntm} chains
422 +\eqref{eq:em-ntm-cosine}--\eqref{eq:em-ntm-sharpen} with the memory update
423 +\eqref{eq:em-ntm-erase}--\eqref{eq:em-ntm-add}; every operation is smooth
424 +in its inputs, so gradients flow from the task loss into the controller,
425 +the heads and the memory itself. The complete system is drawn in
426 +Figure~\ref{fig:em-ntm}.
427 +
428 +\begin{algorithm}[t]
429 +\caption{One NTM head at timestep $t$ \cite{graves2014}}
430 +\label{alg:em-ntm}
431 +\begin{algorithmic}[1]
432 +\Require controller outputs $\vect{k}_t, \beta_t, g_t, \vect{s}_t,
433 + \gamma_t$ (and $\vect{e}_t, \vect{a}_t$ for a write head),
434 + previous weighting $\vect{w}_{t-1}$, memory $\mat{M}_{t-1}$
435 +\State $w_t^c(i) \gets \softmax_i\bigl(\beta_t\,
436 + K[\vect{k}_t,\mat{M}_{t-1}(i)]\bigr)$
437 + \Comment{content addressing,
438 + \eqref{eq:em-ntm-cosine}--\eqref{eq:em-ntm-content}}
439 +\State $\vect{w}_t^{g} \gets g_t \vect{w}_t^{c}
440 + + (1-g_t)\,\vect{w}_{t-1}$
441 + \Comment{interpolation, \eqref{eq:em-ntm-interp}}
442 +\State $\tilde{w}_t(i) \gets \sum_j w^{g}_t(j)\,s_t((i-j)\bmod N)$
443 + \Comment{circular shift, \eqref{eq:em-ntm-shift}}
444 +\State $w_t(i) \gets \tilde{w}_t(i)^{\gamma_t} \big/
445 + \sum_j \tilde{w}_t(j)^{\gamma_t}$
446 + \Comment{sharpening, \eqref{eq:em-ntm-sharpen}}
447 +\If{write head}
448 + \State apply erase then add,
449 + \eqref{eq:em-ntm-erase}--\eqref{eq:em-ntm-add}
450 +\Else
451 + \State \Return read vector
452 + $\vect{r}_t = \sum_i w_t(i)\,\mat{M}_t(i)$,
453 + \eqref{eq:em-ntm-read}
454 +\EndIf
455 +\end{algorithmic}
456 +\end{algorithm}
457 +
458 +\begin{figure}[t]
459 + \centering
460 + \begin{tikzpicture}[scale=1.0]
461 + % controller
462 + \node[blochidden, minimum width=2.5cm, minimum height=1.5cm,
463 + align=center]
464 + (ctrl) at (0,0) {Controller\\(LSTM)};
465 + \draw[fleche] (-2.6,-0.45) node[left, font=\small] {$\vect{x}_t$}
466 + -- (-1.25,-0.45);
467 + \draw[fleche] (-1.25, 0.45)
468 + -- (-2.6, 0.45) node[left, font=\small] {$\vect{y}_t$};
469 + % heads
470 + \node[gate, minimum width=1.9cm] (wh) at (3.7, 1.5)
471 + {write head};
472 + \node[gate, minimum width=1.9cm] (rh) at (3.7,-1.5)
473 + {read head};
474 + \draw[fleche] (1.25, 0.45) -| (wh.south);
475 + \draw[fleche] (1.25,-0.45) -| (rh.north);
476 + \node[etiquette, anchor=south, align=center] at (1.85, 0.55)
477 + {$\vect{k}_t, \beta_t, g_t, \vect{s}_t,$\\
478 + $\gamma_t, \vect{e}_t, \vect{a}_t$};
479 + \node[etiquette, anchor=north] at (2.15,-0.55)
480 + {$\vect{k}_t, \beta_t, g_t, \vect{s}_t, \gamma_t$};
481 + % memory grid: 4 rows (locations N) x 5 columns (width W)
482 + \begin{scope}[shift={(6.3,-1.24)}]
483 + % highlighted row (addressed): row index 2 (third from bottom)
484 + \fill[cmem!35] (0, 1.24) rectangle (3.10, 1.86);
485 + \foreach \r in {0,1,2,3}{
486 + \foreach \c in {0,1,2,3,4}{
487 + \draw[black!60] (\c*0.62, \r*0.62)
488 + rectangle ++(0.62,0.62);
489 + }
490 + }
491 + \node[etiquette] at (1.55, 2.85)
492 + {memory $\mat{M}_t \in \R^{N\times W}$};
493 + \node[etiquette, anchor=west] at (3.22, 1.55) {$w_t$};
494 + \end{scope}
495 + % head <-> memory arrows (enter the addressed row's left edge)
496 + \draw[fleche, cgate!60!black] (wh.east)
497 + -- (5.9, 1.5) -- (5.9, 0.48) -- (6.28, 0.48);
498 + \draw[fleche, cgate!60!black] (rh.east)
499 + -- (5.9, -1.5) -- (5.9, 0.14) -- (6.28, 0.14);
500 + \node[etiquette, anchor=south, align=center] at (5.45, 1.58)
501 + {erase, add\\
502 + \eqref{eq:em-ntm-erase}--\eqref{eq:em-ntm-add}};
503 + \node[etiquette, anchor=north] at (5.42, -1.62)
504 + {address $w_t$};
505 + % read vector returned to controller (dashed)
506 + \draw[flechep, cmem!70!black]
507 + (rh.south) |- (0,-2.75)
508 + node[pos=0.75, above, etiquette] {read vector $\vect{r}_t$,
509 + \eqref{eq:em-ntm-read}}
510 + -- (0,-0.75);
511 + \end{tikzpicture}
512 + \caption{The Neural Turing Machine. The controller emits addressing
513 + parameters for each head; the write head modifies the memory by erase
514 + \eqref{eq:em-ntm-erase} then add \eqref{eq:em-ntm-add}, the read head
515 + returns the attention-weighted content of the addressed row (violet),
516 + and the read vector feeds back into the controller at the next step
517 + (dashed). Every operation is differentiable, so the system trains
518 + end-to-end by gradient descent.}
519 + \label{fig:em-ntm}
520 +\end{figure}
521 +
522 +\begin{remark}
523 +The Differentiable Neural Computer refines the NTM with dynamic slot
524 +allocation via usage vectors and a temporal link matrix that lets read
525 +heads replay memory in the order it was written. Although superseded in
526 +practice by Transformers — whose attention \emph{is} a form of
527 +content-based addressing \eqref{eq:em-ntm-content} over an internal
528 +memory of past tokens — the NTM/DNC line established the memory-augmented
529 +paradigm that today's retrieval-augmented models inherit.
530 +\end{remark}
531 +
532 +% ============================================================================
533 +\section{Kolmogorov--Arnold Networks}
534 +\label{sec:em-kan}
535 +
536 +\subsection{The representation theorem}
537 +
538 +\begin{theorem}[Kolmogorov--Arnold superposition]\label{thm:em-kart}
539 +Every continuous function $f : [0,1]^n \to \R$ can be written as
540 +\begin{equation}
541 + f(x_1, \dots, x_n) \;=\;
542 + \sum_{q=1}^{2n+1} \Phi_q\!\left(
543 + \sum_{p=1}^{n} \phi_{q,p}(x_p)\right),
544 + \label{eq:em-kan-kart}
545 +\end{equation}
546 +where the $\Phi_q : \R \to \R$ and $\phi_{q,p} : [0,1] \to \R$ are
547 +continuous \emph{univariate} functions.
548 +\end{theorem}
549 +
550 +Multivariate continuity thus reduces entirely to sums and compositions of
551 +one-dimensional functions — the only truly multivariate operation in
552 +\eqref{eq:em-kan-kart} is addition.
553 +
554 +\subsection{KAN layers}
555 +
556 +Kolmogorov--Arnold networks \cite{liu2024kan} turn this structure into an
557 +architecture. Where an MLP layer computes
558 +$\sigma(\mat{W}\vect{x} + \vect{b})$ — fixed nonlinearities on the nodes,
559 +learnable scalars on the edges — a KAN layer places a \emph{learnable
560 +univariate function on every edge} and reduces nodes to pure summation:
561 +\begin{equation}
562 + x_{\ell+1,\,j} \;=\; \sum_{i=1}^{n_\ell}
563 + \phi_{\ell,j,i}\bigl(x_{\ell,\,i}\bigr),
564 + \label{eq:em-kan-layer}
565 +\end{equation}
566 +and a full network is a composition of such function matrices
567 +$\Phi_\ell = \bigl(\phi_{\ell,j,i}\bigr)_{j,i}$:
568 +\begin{equation}
569 + \mathrm{KAN}(\vect{x}) \;=\;
570 + \bigl(\Phi_{L-1} \circ \cdots \circ \Phi_1 \circ
571 + \Phi_0\bigr)(\vect{x}).
572 + \label{eq:em-kan-compose}
573 +\end{equation}
574 +Each edge function is parameterized as a B-spline plus a smooth residual
575 +basis that keeps gradients well-behaved:
576 +\begin{equation}
577 + \phi(x) \;=\; w_b\,\mathrm{silu}(x)
578 + + w_s \sum_{i} c_i\,B_i(x),
579 + \label{eq:em-kan-spline}
580 +\end{equation}
581 +with learnable spline coefficients $c_i$ over a grid of knots. The
582 +authors of \cite{liu2024kan} stress that
583 +\eqref{eq:em-kan-layer}--\eqref{eq:em-kan-compose} generalize the exact
584 +depth-2, width-$(2n{+}1)$ form of Theorem~\ref{thm:em-kart} to arbitrary
585 +depths and widths. Figure~\ref{fig:em-kan} draws a small KAN with the
586 +learned univariate function displayed on each edge.
587 +
588 +\begin{figure}[t]
589 + \centering
590 + \begin{tikzpicture}[scale=1.0]
591 + % nodes
592 + \node[ninput] (x1) at (0, 1.0) {$x_1$};
593 + \node[ninput] (x2) at (0,-1.0) {$x_2$};
594 + \node[op, minimum size=7mm] (m1) at (3.4, 1.8) {$+$};
595 + \node[op, minimum size=7mm] (m2) at (3.4, 0.0) {$+$};
596 + \node[op, minimum size=7mm] (m3) at (3.4,-1.8) {$+$};
597 + \node[noutput, minimum size=9mm] (y) at (6.8, 0.0) {$y$};
598 + % edges first (below boxes)
599 + \draw[fleche, black!55] (x1) -- (m1);
600 + \draw[fleche, black!55] (x1) -- (m2);
601 + \draw[fleche, black!55] (x1) -- (m3);
602 + \draw[fleche, black!55] (x2) -- (m1);
603 + \draw[fleche, black!55] (x2) -- (m2);
604 + \draw[fleche, black!55] (x2) -- (m3);
605 + \draw[fleche, black!55] (m1) -- (y);
606 + \draw[fleche, black!55] (m2) -- (y);
607 + \draw[fleche, black!55] (m3) -- (y);
608 + % mini function boxes on edges (drawn after edges to sit on top)
609 + % straight-ish edges: boxes at pos ~0.4 ; crossing edges: pos ~0.72
610 + \node[draw=black!60, fill=white, minimum width=8mm,
611 + minimum height=5.5mm, inner sep=0.5pt] at (1.36, 1.32) {};
612 + \draw[cinput, thick]
613 + plot[domain=-0.30:0.30, samples=19]
614 + ({1.36+\x}, {1.32+0.14*sin(600*\x)});
615 + \node[draw=black!60, fill=white, minimum width=8mm,
616 + minimum height=5.5mm, inner sep=0.5pt] at (1.36, 0.60) {};
617 + \draw[cinput, thick]
618 + plot[domain=-0.30:0.30, samples=19]
619 + ({1.36+\x}, {0.60+1.1*\x*\x-0.09});
620 + \node[draw=black!60, fill=white, minimum width=8mm,
621 + minimum height=5.5mm, inner sep=0.5pt] at (2.448,-1.016) {};
622 + \draw[cinput, thick]
623 + plot[domain=-0.30:0.30, samples=19]
624 + ({2.448+\x}, {-1.016+0.13*tanh(6*\x)});
625 + \node[draw=black!60, fill=white, minimum width=8mm,
626 + minimum height=5.5mm, inner sep=0.5pt] at (2.448, 1.016) {};
627 + \draw[cinput, thick]
628 + plot[domain=-0.30:0.30, samples=19]
629 + ({2.448+\x}, {1.016-0.13*tanh(6*\x)});
630 + \node[draw=black!60, fill=white, minimum width=8mm,
631 + minimum height=5.5mm, inner sep=0.5pt] at (1.36,-0.60) {};
632 + \draw[cinput, thick]
633 + plot[domain=-0.30:0.30, samples=19]
634 + ({1.36+\x}, {-0.60+0.14*sin(300*\x)});
635 + \node[draw=black!60, fill=white, minimum width=8mm,
636 + minimum height=5.5mm, inner sep=0.5pt] at (1.36,-1.32) {};
637 + \draw[cinput, thick]
638 + plot[domain=-0.30:0.30, samples=19]
639 + ({1.36+\x}, {-1.32-1.1*\x*\x+0.09});
640 + % layer 2 boxes
641 + \node[draw=black!60, fill=white, minimum width=8mm,
642 + minimum height=5.5mm, inner sep=0.5pt] at (5.1, 0.90) {};
643 + \draw[coutput, thick]
644 + plot[domain=-0.30:0.30, samples=19]
645 + ({5.1+\x}, {0.90+0.13*tanh(6*\x)});
646 + \node[draw=black!60, fill=white, minimum width=8mm,
647 + minimum height=5.5mm, inner sep=0.5pt] at (5.1, 0.0) {};
648 + \draw[coutput, thick]
649 + plot[domain=-0.30:0.30, samples=19]
650 + ({5.1+\x}, {0.0+0.14*sin(600*\x)});
651 + \node[draw=black!60, fill=white, minimum width=8mm,
652 + minimum height=5.5mm, inner sep=0.5pt] at (5.1,-0.90) {};
653 + \draw[coutput, thick]
654 + plot[domain=-0.30:0.30, samples=19]
655 + ({5.1+\x}, {-0.90+1.1*\x*\x-0.09});
656 + % annotations
657 + \node[etiquette, align=center] at (0,-2.6)
658 + {inputs};
659 + \node[etiquette, align=center] at (3.4,-2.6)
660 + {summation nodes \eqref{eq:em-kan-layer}};
661 + \node[etiquette, align=center] at (6.8,-2.6)
662 + {output};
663 + \node[etiquette, align=center] at (3.4, 2.75)
664 + {a learnable univariate $\phi_{\ell,j,i}$
665 + \eqref{eq:em-kan-spline} on \emph{every} edge};
666 + \end{tikzpicture}
667 + \caption{A small Kolmogorov--Arnold network with $n_0 = 2$ inputs,
668 + $n_1 = 3$ hidden summation nodes and one output. Each edge carries
669 + its own learnable univariate function (inset curves), parameterized
670 + as a B-spline \eqref{eq:em-kan-spline}; the nodes only add their
671 + incoming values, exactly as in the superposition
672 + \eqref{eq:em-kan-kart}.}
673 + \label{fig:em-kan}
674 +\end{figure}
675 +
676 +\subsection{KANs versus MLPs}
677 +
678 +Table~\ref{tab:em-kan-mlp} summarizes the structural contrast. KANs trade
679 +the hardware-friendliness of dense matrix multiplication for
680 +interpretability: a trained spline can be plotted, pruned, and often
681 +symbolically identified ($\sin$, $x^2$, $\exp$), which has made KANs
682 +attractive for scientific and symbolic-regression tasks.
683 +
684 +\begin{table}[t]
685 + \centering
686 + \caption{Multilayer perceptrons versus Kolmogorov--Arnold networks.}
687 + \label{tab:em-kan-mlp}
688 + \begin{tabular}{@{}lll@{}}
689 + \toprule
690 + & MLP & KAN \\
691 + \midrule
692 + Nonlinearity & fixed, on nodes & learnable, on edges \\
693 + Edge parameters & scalar weights $w_{ij}$ &
694 + spline functions $\phi_{j,i}$ \\
695 + Node operation & $\sigma(\mat{W}\vect{x}+\vect{b})$ &
696 + summation \eqref{eq:em-kan-layer} \\
697 + Theoretical anchor & universal approximation &
698 + superposition \eqref{eq:em-kan-kart} \\
699 + Strengths & fast dense algebra, scales &
700 + small-scale accuracy, interpretable \\
701 + Weaknesses & opaque parameters &
702 + slower training, unproven at scale \\
703 + \bottomrule
704 + \end{tabular}
705 +\end{table}
706 +
707 +\begin{remark}[Grid extension]\label{rem:em-grid}
708 +The spline grid in \eqref{eq:em-kan-spline} can be refined during
709 +training: a KAN first fitted on a coarse grid is re-projected onto a
710 +finer one (a small least-squares problem per edge), increasing capacity
711 +exactly where resolution is needed without restarting optimization — a
712 +form of continuation in model space with no analogue in standard MLP
713 +training.
714 +\end{remark}
715 +
716 +% ============================================================================
717 +\section{Outlook}
718 +\label{sec:em-outlook}
719 +
720 +The four families of this chapter relax, in turn, each frozen ingredient
721 +of the classical neural network: the scalar unit (capsules), the discrete
722 +layer (neural ODEs), the fixed-size state (memory-augmented networks) and
723 +the fixed nonlinearity (KANs). None has displaced the Transformer as the
724 +dominant general-purpose architecture, but each has permanently enlarged
725 +the design space — routing lives on in mixture-of-experts gating,
726 +continuous-depth models in diffusion and flow-based generation, external
727 +memory in retrieval-augmented systems, and learnable univariate bases in
728 +scientific machine learning. The history of the field, from the
729 +perceptron onward, suggests that ideas of this kind rarely disappear;
730 +they wait for the scale, the hardware or the objective that lets them
731 +matter.
added main.pdf +0 −0

Binary file not shown.

added main.tex +511 −0
@@ -0,0 +1,511 @@
1 +% ============================================================================
2 +% Artificial Neural Networks — Methods, Equations and
3 +% Graphical Representations
4 +% Author : Simon-Pierre Boucher — contact@spboucher.ai
5 +% Book format (parts, preface, chapters, bibliography)
6 +% File : main.tex — preamble, cover, preface, abstract, TOC, bibliography
7 +% ============================================================================
8 +\documentclass[11pt,a4paper]{book}
9 +
10 +% ----------------------------------------------------------------------------
11 +% Encoding, language, typography
12 +% ----------------------------------------------------------------------------
13 +\usepackage[utf8]{inputenc}
14 +\usepackage[T1]{fontenc}
15 +\usepackage{lmodern}
16 +\usepackage[english]{babel}
17 +\usepackage{microtype}
18 +
19 +% ----------------------------------------------------------------------------
20 +% Mathematics
21 +% ----------------------------------------------------------------------------
22 +\usepackage{amsmath,amssymb,amsthm}
23 +\usepackage{mathtools}
24 +\usepackage{bm}
25 +
26 +% ----------------------------------------------------------------------------
27 +% Graphics: TikZ + pgfplots (no external images)
28 +% ----------------------------------------------------------------------------
29 +\usepackage{tikz}
30 +\usetikzlibrary{positioning,arrows.meta,calc,shapes.geometric,fit,
31 + backgrounds,decorations.pathreplacing}
32 +\usepackage{pgfplots}
33 +\pgfplotsset{compat=1.17}
34 +
35 +% ----------------------------------------------------------------------------
36 +% Algorithms
37 +% ----------------------------------------------------------------------------
38 +\usepackage{algorithm}
39 +\usepackage{algpseudocode}
40 +
41 +% ----------------------------------------------------------------------------
42 +% Page layout
43 +% ----------------------------------------------------------------------------
44 +\usepackage[a4paper,top=2.6cm,bottom=2.8cm,inner=2.8cm,outer=2.4cm,
45 + headheight=14.5pt]{geometry}
46 +\usepackage{fancyhdr}
47 +\usepackage{booktabs}
48 +\usepackage{array}
49 +
50 +\pagestyle{fancy}
51 +\fancyhf{}
52 +\fancyhead[LE]{\small\itshape\nouppercase{\leftmark}}
53 +\fancyhead[RO]{\small\itshape\nouppercase{\rightmark}}
54 +\fancyfoot[C]{\small\thepage}
55 +\renewcommand{\headrulewidth}{0.4pt}
56 +\fancypagestyle{plain}{\fancyhf{}\fancyfoot[C]{\small\thepage}%
57 + \renewcommand{\headrulewidth}{0pt}}
58 +
59 +% ----------------------------------------------------------------------------
60 +% Hyperlinks
61 +% ----------------------------------------------------------------------------
62 +\usepackage[colorlinks=true,
63 + linkcolor=blue!50!black,
64 + citecolor=green!40!black,
65 + urlcolor=violet!70!black,
66 + pdftitle={Artificial Neural Networks — Methods, Equations and
67 + Graphical Representations},
68 + pdfauthor={Simon-Pierre Boucher}]{hyperref}
69 +
70 +% ----------------------------------------------------------------------------
71 +% Named palette (used by ALL figures)
72 +% ----------------------------------------------------------------------------
73 +\definecolor{cinput}{RGB}{31,119,180} % blue — inputs
74 +\definecolor{chidden}{RGB}{255,127,14} % orange — hidden layers
75 +\definecolor{coutput}{RGB}{214,39,40} % red — outputs
76 +\definecolor{cgate}{RGB}{44,160,44} % green — gates
77 +\definecolor{cmem}{RGB}{148,103,189} % violet — memory / state
78 +
79 +% ----------------------------------------------------------------------------
80 +% Global TikZ styles (reused across all chapters)
81 +% ----------------------------------------------------------------------------
82 +\tikzset{
83 + neuron/.style = {circle, draw=black!70, minimum size=8mm,
84 + inner sep=0pt, font=\small},
85 + ninput/.style = {neuron, fill=cinput!25, draw=cinput!80!black},
86 + nhidden/.style = {neuron, fill=chidden!30, draw=chidden!80!black},
87 + noutput/.style = {neuron, fill=coutput!25, draw=coutput!80!black},
88 + bloc/.style = {rectangle, rounded corners=2pt, draw=black!70,
89 + fill=black!5, minimum width=2.1cm, minimum height=8mm,
90 + align=center, font=\small},
91 + blocinput/.style = {bloc, fill=cinput!15, draw=cinput!70!black},
92 + blochidden/.style = {bloc, fill=chidden!20, draw=chidden!70!black},
93 + blocoutput/.style = {bloc, fill=coutput!15, draw=coutput!70!black},
94 + gate/.style = {bloc, fill=cgate!20, draw=cgate!70!black,
95 + minimum width=9mm, minimum height=7mm},
96 + mem/.style = {bloc, fill=cmem!15, draw=cmem!70!black},
97 + op/.style = {circle, draw=black!70, fill=white, inner sep=1pt,
98 + minimum size=5.5mm, font=\small},
99 + fleche/.style = {-{Stealth[length=2.5mm]}, thick},
100 + flechep/.style = {fleche, dashed},
101 + etiquette/.style= {font=\scriptsize, text=black!60},
102 +}
103 +
104 +% ----------------------------------------------------------------------------
105 +% Theorem environments (numbered per chapter)
106 +% ----------------------------------------------------------------------------
107 +\theoremstyle{plain}
108 +\newtheorem{theorem}{Theorem}[chapter]
109 +\newtheorem{property}[theorem]{Property}
110 +\theoremstyle{definition}
111 +\newtheorem{definition}[theorem]{Definition}
112 +\theoremstyle{remark}
113 +\newtheorem{remark}[theorem]{Remark}
114 +
115 +% ----------------------------------------------------------------------------
116 +% Notation macros (absolute consistency across chapters)
117 +% ----------------------------------------------------------------------------
118 +\newcommand{\vect}[1]{\bm{#1}} % vector (bold)
119 +\newcommand{\mat}[1]{\bm{#1}} % matrix (bold)
120 +\newcommand{\Loss}{\mathcal{L}} % loss function
121 +\newcommand{\E}{\mathbb{E}} % expectation
122 +\newcommand{\R}{\mathbb{R}} % reals
123 +\newcommand{\KL}{D_{\mathrm{KL}}} % Kullback–Leibler divergence
124 +\DeclareMathOperator{\softmax}{softmax}
125 +\DeclareMathOperator*{\argmin}{arg\,min}
126 +\DeclareMathOperator*{\argmax}{arg\,max}
127 +\newcommand{\transp}{^{\top}} % transpose
128 +
129 +\numberwithin{equation}{chapter}
130 +
131 +% ============================================================================
132 +\begin{document}
133 +
134 +% ----------------------------------------------------------------------------
135 +% COVER / TITLE PAGE
136 +% ----------------------------------------------------------------------------
137 +\begin{titlepage}
138 + \centering
139 + \vspace*{1.2cm}
140 + {\scshape\Large The Deep Learning Series\par}
141 + \vspace{0.4cm}
142 + \rule{0.72\textwidth}{0.8pt}\par
143 + \vspace{0.9cm}
144 + {\Huge\bfseries Artificial Neural\\[0.25em] Networks\par}
145 + \vspace{0.55cm}
146 + {\Large Methods, Equations and Graphical Representations\par}
147 + \vspace{0.9cm}
148 + \rule{0.72\textwidth}{0.8pt}\par
149 + \vspace{1.3cm}
150 +
151 + % Decorative TikZ mini-network
152 + \begin{tikzpicture}[scale=0.9, transform shape]
153 + \foreach \i in {1,2,3}
154 + \node[ninput] (i\i) at (0, 1.5-\i*1.0) {};
155 + \foreach \j in {1,...,4}
156 + \node[nhidden] (h1\j) at (2.4, 2.0-\j*1.0) {};
157 + \foreach \j in {1,...,4}
158 + \node[nhidden] (h2\j) at (4.8, 2.0-\j*1.0) {};
159 + \foreach \k in {1,2}
160 + \node[noutput] (o\k) at (7.2, 1.0-\k*1.0) {};
161 + \foreach \i in {1,2,3} \foreach \j in {1,...,4}
162 + \draw[black!30, semithick] (i\i) -- (h1\j);
163 + \foreach \i in {1,...,4} \foreach \j in {1,...,4}
164 + \draw[black!30, semithick] (h1\i) -- (h2\j);
165 + \foreach \i in {1,...,4} \foreach \k in {1,2}
166 + \draw[black!30, semithick] (h2\i) -- (o\k);
167 + \end{tikzpicture}
168 + \par
169 + \vspace{1.5cm}
170 +
171 + {\Large\itshape Simon-Pierre Boucher\par}
172 + \vspace{0.35cm}
173 + {\large\href{mailto:contact@spboucher.ai}{\texttt{contact@spboucher.ai}}\par}
174 + \vfill
175 + {\large \today\par}
176 + \vspace*{0.8cm}
177 +\end{titlepage}
178 +
179 +\frontmatter
180 +
181 +% ----------------------------------------------------------------------------
182 +% PREFACE
183 +% ----------------------------------------------------------------------------
184 +\chapter*{Preface}
185 +\addcontentsline{toc}{chapter}{Preface}
186 +
187 +Neural networks have moved, in a single generation, from a contested
188 +scientific hypothesis to the engine of modern artificial intelligence. Yet
189 +the field is often taught as a collection of recipes, where the underlying
190 +mathematics is scattered across papers and the architectures are described
191 +by ambiguous sketches. This book was written against that fragmentation.
192 +Its organizing principle is simple and strict: \emph{every method is
193 +presented with rigorous equations and a faithful graphical representation}.
194 +Every figure in this book is drawn natively in TikZ or pgfplots — no
195 +imported images — so that each diagram is exactly as precise as the
196 +equations it illustrates.
197 +
198 +The book is self-contained and can serve as a standalone course text. It
199 +assumes only linear algebra, multivariate calculus and elementary
200 +probability. Notation is kept absolutely consistent from the first chapter
201 +to the last: vectors are bold lowercase ($\vect{x}$), matrices bold
202 +uppercase ($\mat{W}$), losses are written $\Loss$, and the Hadamard product
203 +is $\odot$ throughout.
204 +
205 +\vspace{0.6cm}
206 +\begin{flushright}
207 +\itshape Simon-Pierre Boucher\\
208 +\href{mailto:contact@spboucher.ai}{contact@spboucher.ai}
209 +\end{flushright}
210 +
211 +% ----------------------------------------------------------------------------
212 +% ABSTRACT
213 +% ----------------------------------------------------------------------------
214 +\chapter*{Abstract}
215 +\addcontentsline{toc}{chapter}{Abstract}
216 +
217 +This book offers a rigorous and self-contained treatment of the major
218 +methods of artificial neural networks, from Rosenblatt's perceptron to
219 +contemporary generative models. Each method is presented under a twofold
220 +requirement: on one hand, a complete mathematical formulation —
221 +forward-propagation equations, gradients, objective functions — in a
222 +notation unified across chapters; on the other hand, a native graphical
223 +representation (TikZ/pgfplots) that makes the underlying computational
224 +structure visible.
225 +
226 +Chapter~\ref{chap:foundations} lays the foundations: the artificial neuron,
227 +the perceptron, activation functions, the multilayer perceptron and the
228 +universal approximation theorem. Chapter~\ref{chap:training} covers
229 +training: backpropagation of gradients, adaptive optimizers (SGD, Momentum,
230 +Adam), learning-rate scheduling and initialization.
231 +Chapter~\ref{chap:regularization} addresses regularization: $L_1$/$L_2$
232 +penalties, dropout, batch and layer normalization, early stopping.
233 +Chapters~\ref{chap:cnn} and~\ref{chap:rnn} detail convolutional networks
234 +(convolution, pooling, residual blocks, receptive fields) and recurrent
235 +networks (RNN, LSTM, GRU, bidirectional and encoder--decoder
236 +architectures). Chapter~\ref{chap:attention} is devoted to the attention
237 +mechanism and the Transformer architecture, and
238 +Chapter~\ref{chap:modern} to its modern descendants: Vision Transformers,
239 +mixtures of experts, efficient attention and selective state-space models.
240 +Chapter~\ref{chap:gnn} develops graph neural networks (message passing,
241 +GCN, GraphSAGE, GAT, GIN) and Chapter~\ref{chap:energy} the energy-based
242 +family: Hopfield networks, Boltzmann machines, restricted Boltzmann
243 +machines and deep belief networks. Chapter~\ref{chap:generative} presents
244 +the core generative models — autoencoders, variational autoencoders and
245 +generative adversarial networks — while Chapter~\ref{chap:advgen} covers
246 +normalizing flows, diffusion models and deep autoregressive models.
247 +Finally, Chapter~\ref{chap:bio} treats biologically inspired computation
248 +(spiking networks, self-organizing maps, reservoir computing) and
249 +Chapter~\ref{chap:emerging} the emerging architectures: capsule networks,
250 +neural ordinary differential equations, memory-augmented networks and
251 +Kolmogorov--Arnold networks.
252 +
253 +\vspace{0.8cm}
254 +\noindent\textbf{Keywords}: neural networks, deep learning,
255 +backpropagation, regularization, convolutional networks, recurrent
256 +networks, LSTM, attention, Transformer, variational autoencoder, GAN,
257 +diffusion.
258 +
259 +% ----------------------------------------------------------------------------
260 +% TABLES
261 +% ----------------------------------------------------------------------------
262 +\tableofcontents
263 +
264 +\mainmatter
265 +
266 +% ----------------------------------------------------------------------------
267 +% CHAPTERS
268 +% ----------------------------------------------------------------------------
269 +\part{Foundations and Learning}
270 +\include{chapters/01-foundations}
271 +\include{chapters/02-training}
272 +\include{chapters/03-regularization}
273 +
274 +\part{Core Architectures}
275 +\include{chapters/04-cnn}
276 +\include{chapters/05-rnn}
277 +\include{chapters/06-attention}
278 +\include{chapters/07-modern-transformers}
279 +
280 +\part{Graphs, Energy and Memory}
281 +\include{chapters/08-gnn}
282 +\include{chapters/09-energy}
283 +
284 +\part{Generative Models}
285 +\include{chapters/10-generative}
286 +\include{chapters/11-advanced-generative}
287 +
288 +\part{Specialized and Emerging Architectures}
289 +\include{chapters/12-bio-inspired}
290 +\include{chapters/13-emerging}
291 +
292 +\backmatter
293 +
294 +% ----------------------------------------------------------------------------
295 +% BIBLIOGRAPHY
296 +% ----------------------------------------------------------------------------
297 +\begin{thebibliography}{99}
298 +\addcontentsline{toc}{chapter}{Bibliography}
299 +
300 +\bibitem{rosenblatt1958}
301 +F.~Rosenblatt.
302 +\newblock The perceptron: A probabilistic model for information storage and
303 + organization in the brain.
304 +\newblock \emph{Psychological Review}, 65(6):386--408, 1958.
305 +
306 +\bibitem{rumelhart1986}
307 +D.~E. Rumelhart, G.~E. Hinton, and R.~J. Williams.
308 +\newblock Learning representations by back-propagating errors.
309 +\newblock \emph{Nature}, 323:533--536, 1986.
310 +
311 +\bibitem{lecun1998}
312 +Y.~LeCun, L.~Bottou, Y.~Bengio, and P.~Haffner.
313 +\newblock Gradient-based learning applied to document recognition.
314 +\newblock \emph{Proceedings of the IEEE}, 86(11):2278--2324, 1998.
315 +
316 +\bibitem{hochreiter1997}
317 +S.~Hochreiter and J.~Schmidhuber.
318 +\newblock Long short-term memory.
319 +\newblock \emph{Neural Computation}, 9(8):1735--1780, 1997.
320 +
321 +\bibitem{cho2014}
322 +K.~Cho, B.~van Merriënboer, C.~Gulcehre, D.~Bahdanau, F.~Bougares,
323 + H.~Schwenk, and Y.~Bengio.
324 +\newblock Learning phrase representations using {RNN} encoder--decoder for
325 + statistical machine translation.
326 +\newblock In \emph{Proceedings of EMNLP}, pages 1724--1734, 2014.
327 +
328 +\bibitem{srivastava2014}
329 +N.~Srivastava, G.~Hinton, A.~Krizhevsky, I.~Sutskever, and
330 + R.~Salakhutdinov.
331 +\newblock Dropout: A simple way to prevent neural networks from overfitting.
332 +\newblock \emph{Journal of Machine Learning Research}, 15:1929--1958, 2014.
333 +
334 +\bibitem{ioffe2015}
335 +S.~Ioffe and C.~Szegedy.
336 +\newblock Batch normalization: Accelerating deep network training by
337 + reducing internal covariate shift.
338 +\newblock In \emph{Proceedings of ICML}, pages 448--456, 2015.
339 +
340 +\bibitem{kingma2015adam}
341 +D.~P. Kingma and J.~Ba.
342 +\newblock Adam: A method for stochastic optimization.
343 +\newblock In \emph{Proceedings of ICLR}, 2015.
344 +
345 +\bibitem{he2016}
346 +K.~He, X.~Zhang, S.~Ren, and J.~Sun.
347 +\newblock Deep residual learning for image recognition.
348 +\newblock In \emph{Proceedings of CVPR}, pages 770--778, 2016.
349 +
350 +\bibitem{vaswani2017}
351 +A.~Vaswani, N.~Shazeer, N.~Parmar, J.~Uszkoreit, L.~Jones, A.~N. Gomez,
352 + Ł.~Kaiser, and I.~Polosukhin.
353 +\newblock Attention is all you need.
354 +\newblock In \emph{Advances in Neural Information Processing Systems~30},
355 + pages 5998--6008, 2017.
356 +
357 +\bibitem{kingma2014vae}
358 +D.~P. Kingma and M.~Welling.
359 +\newblock Auto-encoding variational {B}ayes.
360 +\newblock In \emph{Proceedings of ICLR}, 2014.
361 +
362 +\bibitem{goodfellow2014gan}
363 +I.~Goodfellow, J.~Pouget-Abadie, M.~Mirza, B.~Xu, D.~Warde-Farley,
364 + S.~Ozair, A.~Courville, and Y.~Bengio.
365 +\newblock Generative adversarial nets.
366 +\newblock In \emph{Advances in Neural Information Processing Systems~27},
367 + pages 2672--2680, 2014.
368 +
369 +\bibitem{goodfellow2016book}
370 +I.~Goodfellow, Y.~Bengio, and A.~Courville.
371 +\newblock \emph{Deep Learning}.
372 +\newblock MIT Press, 2016.
373 +
374 +\bibitem{hopfield1982}
375 +J.~J. Hopfield.
376 +\newblock Neural networks and physical systems with emergent collective
377 + computational abilities.
378 +\newblock \emph{Proceedings of the National Academy of Sciences},
379 + 79(8):2554--2558, 1982.
380 +
381 +\bibitem{ackley1985}
382 +D.~H. Ackley, G.~E. Hinton, and T.~J. Sejnowski.
383 +\newblock A learning algorithm for {B}oltzmann machines.
384 +\newblock \emph{Cognitive Science}, 9(1):147--169, 1985.
385 +
386 +\bibitem{kohonen1990}
387 +T.~Kohonen.
388 +\newblock The self-organizing map.
389 +\newblock \emph{Proceedings of the IEEE}, 78(9):1464--1480, 1990.
390 +
391 +\bibitem{maass1997}
392 +W.~Maass.
393 +\newblock Networks of spiking neurons: The third generation of neural
394 + network models.
395 +\newblock \emph{Neural Networks}, 10(9):1659--1671, 1997.
396 +
397 +\bibitem{jaeger2001}
398 +H.~Jaeger.
399 +\newblock The ``echo state'' approach to analysing and training recurrent
400 + neural networks.
401 +\newblock GMD Report 148, German National Research Center for Information
402 + Technology, 2001.
403 +
404 +\bibitem{hinton2002}
405 +G.~E. Hinton.
406 +\newblock Training products of experts by minimizing contrastive
407 + divergence.
408 +\newblock \emph{Neural Computation}, 14(8):1771--1800, 2002.
409 +
410 +\bibitem{hinton2006}
411 +G.~E. Hinton, S.~Osindero, and Y.-W. Teh.
412 +\newblock A fast learning algorithm for deep belief nets.
413 +\newblock \emph{Neural Computation}, 18(7):1527--1554, 2006.
414 +
415 +\bibitem{graves2014}
416 +A.~Graves, G.~Wayne, and I.~Danihelka.
417 +\newblock Neural {T}uring machines.
418 +\newblock arXiv:1410.5401, 2014.
419 +
420 +\bibitem{rezende2015}
421 +D.~J. Rezende and S.~Mohamed.
422 +\newblock Variational inference with normalizing flows.
423 +\newblock In \emph{Proceedings of ICML}, pages 1530--1538, 2015.
424 +
425 +\bibitem{oord2016wavenet}
426 +A.~van~den Oord, S.~Dieleman, H.~Zen, K.~Simonyan, O.~Vinyals, A.~Graves,
427 + N.~Kalchbrenner, A.~Senior, and K.~Kavukcuoglu.
428 +\newblock {WaveNet}: A generative model for raw audio.
429 +\newblock arXiv:1609.03499, 2016.
430 +
431 +\bibitem{dinh2017}
432 +L.~Dinh, J.~Sohl-Dickstein, and S.~Bengio.
433 +\newblock Density estimation using {Real NVP}.
434 +\newblock In \emph{Proceedings of ICLR}, 2017.
435 +
436 +\bibitem{kipf2017}
437 +T.~N. Kipf and M.~Welling.
438 +\newblock Semi-supervised classification with graph convolutional networks.
439 +\newblock In \emph{Proceedings of ICLR}, 2017.
440 +
441 +\bibitem{hamilton2017}
442 +W.~L. Hamilton, R.~Ying, and J.~Leskovec.
443 +\newblock Inductive representation learning on large graphs.
444 +\newblock In \emph{Advances in Neural Information Processing Systems~30},
445 + pages 1024--1034, 2017.
446 +
447 +\bibitem{shazeer2017}
448 +N.~Shazeer, A.~Mirhoseini, K.~Maziarz, A.~Davis, Q.~Le, G.~Hinton, and
449 + J.~Dean.
450 +\newblock Outrageously large neural networks: The sparsely-gated
451 + mixture-of-experts layer.
452 +\newblock In \emph{Proceedings of ICLR}, 2017.
453 +
454 +\bibitem{sabour2017}
455 +S.~Sabour, N.~Frosst, and G.~E. Hinton.
456 +\newblock Dynamic routing between capsules.
457 +\newblock In \emph{Advances in Neural Information Processing Systems~30},
458 + pages 3856--3866, 2017.
459 +
460 +\bibitem{velickovic2018}
461 +P.~Veličković, G.~Cucurull, A.~Casanova, A.~Romero, P.~Liò, and
462 + Y.~Bengio.
463 +\newblock Graph attention networks.
464 +\newblock In \emph{Proceedings of ICLR}, 2018.
465 +
466 +\bibitem{chen2018}
467 +R.~T.~Q. Chen, Y.~Rubanova, J.~Bettencourt, and D.~Duvenaud.
468 +\newblock Neural ordinary differential equations.
469 +\newblock In \emph{Advances in Neural Information Processing Systems~31},
470 + pages 6571--6583, 2018.
471 +
472 +\bibitem{xu2019}
473 +K.~Xu, W.~Hu, J.~Leskovec, and S.~Jegelka.
474 +\newblock How powerful are graph neural networks?
475 +\newblock In \emph{Proceedings of ICLR}, 2019.
476 +
477 +\bibitem{ho2020}
478 +J.~Ho, A.~Jain, and P.~Abbeel.
479 +\newblock Denoising diffusion probabilistic models.
480 +\newblock In \emph{Advances in Neural Information Processing Systems~33},
481 + pages 6840--6851, 2020.
482 +
483 +\bibitem{ramsauer2021}
484 +H.~Ramsauer, B.~Schäfl, J.~Lehner, P.~Seidl, M.~Widrich, T.~Adler,
485 + L.~Gruber, M.~Holzleitner, M.~Pavlović, G.~K. Sandve, V.~Greiff,
486 + D.~Kreil, M.~Kopp, G.~Klambauer, J.~Brandstetter, and S.~Hochreiter.
487 +\newblock Hopfield networks is all you need.
488 +\newblock In \emph{Proceedings of ICLR}, 2021.
489 +
490 +\bibitem{dosovitskiy2021}
491 +A.~Dosovitskiy, L.~Beyer, A.~Kolesnikov, D.~Weissenborn, X.~Zhai,
492 + T.~Unterthiner, M.~Dehghani, M.~Minderer, G.~Heigold, S.~Gelly,
493 + J.~Uszkoreit, and N.~Houlsby.
494 +\newblock An image is worth $16\times16$ words: Transformers for image
495 + recognition at scale.
496 +\newblock In \emph{Proceedings of ICLR}, 2021.
497 +
498 +\bibitem{gu2023}
499 +A.~Gu and T.~Dao.
500 +\newblock Mamba: Linear-time sequence modeling with selective state spaces.
501 +\newblock arXiv:2312.00752, 2023.
502 +
503 +\bibitem{liu2024kan}
504 +Z.~Liu, Y.~Wang, S.~Vaidya, F.~Ruehle, J.~Halverson, M.~Soljačić,
505 + T.~Y. Hou, and M.~Tegmark.
506 +\newblock {KAN}: Kolmogorov--Arnold networks.
507 +\newblock arXiv:2404.19756, 2024.
508 +
509 +\end{thebibliography}
510 +
511 +\end{document}
512