Artificial Neural Networks — Methods, Equations and Graphical Representations
Complete 119-page LaTeX book: 13 chapters in 5 parts, 256 numbered equations, 42 native TikZ/pgfplots figures, 26 estimation algorithms, 36-entry bibliography. Compiled PDF included. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
18 changed files +8,956 −0
added
.gitignore
+17 −0
@@ -0,0 +1,17 @@ | ||
| 1 | +# LaTeX build artifacts | |
| 2 | +*.aux | |
| 3 | +*.log | |
| 4 | +*.out | |
| 5 | +*.toc | |
| 6 | +*.lof | |
| 7 | +*.lot | |
| 8 | +*.loa | |
| 9 | +*.fls | |
| 10 | +*.fdb_latexmk | |
| 11 | +*.synctex.gz | |
| 12 | +*.bbl | |
| 13 | +*.blg | |
| 14 | +build-* | |
| 15 | + | |
| 16 | +# OS | |
| 17 | +.DS_Store | |
added
CLAUDE.md
+194 −0
@@ -0,0 +1,194 @@ | ||
| 1 | +# CLAUDE.md — Mission : construire un LIVRE LaTeX légendaire sur les réseaux de neurones | |
| 2 | + | |
| 3 | +> **Amendements utilisateur (2026-08-05)** : format **livre** (parties, | |
| 4 | +> préface, couverture) plutôt que thèse ; langue **anglaise** ; fichiers de | |
| 5 | +> chapitres renommés en anglais ; environnements théorème en anglais | |
| 6 | +> (`theorem`, `property`, `definition`, `remark`). | |
| 7 | +> | |
| 8 | +> **Extension majeure (2026-08-05)** : couverture de TOUS les grands types | |
| 9 | +> de réseaux de neurones, chacun avec équations + **algorithme | |
| 10 | +> d'estimation** (pseudocode) + figure TikZ, en profondeur maximale. | |
| 11 | +> Structure finale : **13 chapitres en 5 parties** — | |
| 12 | +> I. Foundations and Learning (01-foundations, 02-training, | |
| 13 | +> 03-regularization) ; II. Core Architectures (04-cnn, 05-rnn, | |
| 14 | +> 06-attention, 07-modern-transformers : ViT/MoE/efficient | |
| 15 | +> attention/Mamba) ; III. Graphs, Energy and Memory (08-gnn : MPNN/GCN/ | |
| 16 | +> GraphSAGE/GAT/GIN, 09-energy : Hopfield/Boltzmann/RBM/DBN) ; | |
| 17 | +> IV. Generative Models (10-generative : AE/VAE/GAN, 11-advanced-generative : | |
| 18 | +> flows/diffusion/autorégressifs) ; V. Specialized and Emerging | |
| 19 | +> (12-bio-inspired : SNN/SOM/reservoir, 13-emerging : capsules/Neural | |
| 20 | +> ODE/NTM/KAN). Bibliographie étendue à 36 entrées. Cible : ~120 pages, | |
| 21 | +> ≥30 figures, ≥150 équations, ≥15 algorithmes. | |
| 22 | + | |
| 23 | +> Ce fichier est une **spécification exécutable**. Claude : lis-le en entier, | |
| 24 | +> puis construis le projet complet décrit ci-dessous dans ce dossier, sans | |
| 25 | +> demander de confirmation intermédiaire. À la fin, compile et vérifie. | |
| 26 | + | |
| 27 | +--- | |
| 28 | + | |
| 29 | +## 1. Objectif | |
| 30 | + | |
| 31 | +Produire un document LaTeX **robuste, complet et élégant** de type thèse, | |
| 32 | +couvrant en détail toutes les grandes méthodes des réseaux de neurones. | |
| 33 | +Exigence absolue : **chaque méthode = équations rigoureuses + représentation | |
| 34 | +graphique TikZ**. Aucune figure externe (pas d'images importées) — tout en | |
| 35 | +TikZ/pgfplots natif. | |
| 36 | + | |
| 37 | +## 2. Métadonnées | |
| 38 | + | |
| 39 | +| Champ | Valeur | | |
| 40 | +|---|---| | |
| 41 | +| Titre | Artificial Neural Networks — Methods, Equations and Graphical Representations | | |
| 42 | +| Auteur | **Simon-Pierre Boucher** | | |
| 43 | +| Contact | **contact@spboucher.ai** (lien `mailto:` cliquable sur la page de titre) | | |
| 44 | +| Langue | Anglais (babel `english`) | | |
| 45 | +| Classe | `book`, 11pt, A4 | | |
| 46 | + | |
| 47 | +## 3. Structure du projet | |
| 48 | + | |
| 49 | +``` | |
| 50 | +these/ | |
| 51 | +├── CLAUDE.md # ce fichier | |
| 52 | +├── main.tex # préambule, page de titre, résumé, TOC, includes, bibliographie | |
| 53 | +└── chapters/ | |
| 54 | + ├── 01-fondations.tex | |
| 55 | + ├── 02-entrainement.tex | |
| 56 | + ├── 03-regularisation.tex | |
| 57 | + ├── 04-cnn.tex | |
| 58 | + ├── 05-rnn.tex | |
| 59 | + ├── 06-attention.tex | |
| 60 | + └── 07-generatifs.tex | |
| 61 | +``` | |
| 62 | + | |
| 63 | +## 4. Contenu exigé par chapitre | |
| 64 | + | |
| 65 | +### Chapitre 1 — Fondations | |
| 66 | +- Neurone artificiel : `z = w^T x + b`, `a = φ(z)` ; **figure TikZ** du neurone | |
| 67 | + (entrées, poids, somme Σ, activation φ). | |
| 68 | +- Perceptron de Rosenblatt : règle de mise à jour, théorème de convergence. | |
| 69 | +- Fonctions d'activation : sigmoïde, tanh, ReLU, Leaky ReLU, GELU — équations | |
| 70 | + **et dérivées** ; **courbes pgfplots** superposées. | |
| 71 | +- Softmax (équation). | |
| 72 | +- MLP : équations couche par couche `z^(ℓ) = W^(ℓ) a^(ℓ−1) + b^(ℓ)` ; | |
| 73 | + **figure TikZ** d'un réseau entièrement connecté à 2 couches cachées. | |
| 74 | +- Théorème d'approximation universelle (énoncé en environnement `propriete`). | |
| 75 | + | |
| 76 | +### Chapitre 2 — Entraînement | |
| 77 | +- Risque empirique, MSE, entropie croisée ; gradient softmax+CE = `ŷ − y`. | |
| 78 | +- **Rétropropagation** : les trois équations (δ^(L), récurrence δ^(ℓ), | |
| 79 | + gradients W et b) en environnement théorème ; **figure TikZ** du graphe de | |
| 80 | + calcul avec passe avant (flèches pleines) et passe arrière (flèches | |
| 81 | + pointillées) ; **algorithme** en pseudocode (`algorithm`/`algpseudocode`). | |
| 82 | +- Optimiseurs : SGD, Momentum, Nesterov, AdaGrad, RMSProp, **Adam complet** | |
| 83 | + (avec correction de biais) ; **figure pgfplots** comparant les trajectoires | |
| 84 | + SGD vs Momentum sur des contours de perte anisotropes. | |
| 85 | +- Ordonnancement du taux d'apprentissage (paliers, cosinus, warmup). | |
| 86 | +- Initialisation Xavier/Glorot et He/Kaiming (formules). | |
| 87 | + | |
| 88 | +### Chapitre 3 — Régularisation | |
| 89 | +- Décomposition biais–variance (équation). | |
| 90 | +- L2 (weight decay, forme de la mise à jour) et L1. | |
| 91 | +- **Dropout** : équation inverted dropout ; **figure TikZ** comparant réseau | |
| 92 | + complet vs réseau avec neurones désactivés (croix, pointillés). | |
| 93 | +- **Batch Normalization** : les 4 équations (μ, σ², ẑ, γẑ+β) ; comportement | |
| 94 | + à l'inférence. **Layer Normalization** (équation). | |
| 95 | +- Arrêt précoce (**courbe pgfplots** train vs validation), augmentation de | |
| 96 | + données, label smoothing, gradient clipping. | |
| 97 | + | |
| 98 | +### Chapitre 4 — CNN | |
| 99 | +- Convolution : équation avec stride et padding ; formule de dimension de | |
| 100 | + sortie ; **figure TikZ en grilles** : entrée 5×5, noyau 3×3 (valeurs | |
| 101 | + affichées), sortie 3×3, fenêtre active surlignée. | |
| 102 | +- Pooling max/moyenne (équations). | |
| 103 | +- **Figure TikZ** d'une architecture CNN complète (blocs conv/pool de | |
| 104 | + hauteurs décroissantes → aplatissement → dense → softmax, avec accolades | |
| 105 | + « extraction de caractéristiques » / « classification »). | |
| 106 | +- **ResNet** : équation `y = F(x) + x`, gradient à travers l'identité ; | |
| 107 | + **figure TikZ** du bloc résiduel avec raccourci. | |
| 108 | +- Champ réceptif (formule récursive). | |
| 109 | + | |
| 110 | +### Chapitre 5 — RNN / LSTM / GRU | |
| 111 | +- RNN simple : équations ; **figure TikZ** forme repliée = forme dépliée. | |
| 112 | +- BPTT : produit de jacobiennes, gradients évanescents/explosifs. | |
| 113 | +- **LSTM** : les 6 équations (portes f, i, o, candidat, cellule, sortie) ; | |
| 114 | + **figure TikZ détaillée de la cellule** — ligne d'état c_t horizontale en | |
| 115 | + haut, portes σ/tanh, opérations × et + explicites. Vérifier visuellement | |
| 116 | + qu'aucun nœud ne se chevauche. | |
| 117 | +- **GRU** : les 4 équations. | |
| 118 | +- Bidirectionnel, encodeur–décodeur seq2seq (factorisation autorégressive). | |
| 119 | + | |
| 120 | +### Chapitre 6 — Attention / Transformers | |
| 121 | +- Scaled dot-product attention (équation matricielle) + justification du | |
| 122 | + facteur 1/√d_k (argument de variance) ; **figure TikZ** du flux | |
| 123 | + Q,K,V → QKᵀ → ÷√d_k → softmax → ×V. | |
| 124 | +- Multi-têtes (équations head_i et concat·W^O). | |
| 125 | +- Encodage positionnel sinusoïdal (les deux équations). | |
| 126 | +- Bloc Transformer : Add&LayerNorm, FFN ; masque causal ; **grande figure | |
| 127 | + TikZ** de l'architecture encodeur–décodeur complète (deux colonnes, | |
| 128 | + connexions résiduelles, flèche K,V encodeur → attention croisée, ×N). | |
| 129 | +- Complexité O(n²d) ; familles BERT / GPT / T5. | |
| 130 | + | |
| 131 | +### Chapitre 7 — Modèles génératifs | |
| 132 | +- Auto-encodeur : équations + **figure TikZ** en sablier (goulot latent). | |
| 133 | +- **VAE** : ELBO (avec accolades « reconstruction » / « régularisation »), | |
| 134 | + KL en forme close pour gaussiennes diagonales, **reparamétrisation** ; | |
| 135 | + **figure TikZ** du flux x → encodeur → (μ, σ) → z = μ+σ⊙ε → décodeur. | |
| 136 | +- **GAN** : jeu minimax, discriminateur optimal D*, lien avec la divergence | |
| 137 | + de Jensen–Shannon, perte non saturante, WGAN ; **figure TikZ** | |
| 138 | + générateur/discriminateur avec gradient adverse en pointillés. | |
| 139 | +- **Tableau comparatif** (booktabs) VAE / GAN / autorégressif. | |
| 140 | +- Ouverture sur les modèles de diffusion (équation du bruitage). | |
| 141 | + | |
| 142 | +## 5. Conventions techniques (obligatoires) | |
| 143 | + | |
| 144 | +- **Préambule** : `inputenc utf8`, `fontenc T1`, `babel french`, `lmodern`, | |
| 145 | + `microtype`, `amsmath amssymb amsthm mathtools bm`, `tikz` (libraries : | |
| 146 | + `positioning, arrows.meta, calc, shapes.geometric, fit, backgrounds, | |
| 147 | + decorations.pathreplacing`), `pgfplots` (`compat=1.17`), `algorithm` + | |
| 148 | + `algpseudocode`, `geometry`, `fancyhdr`, `booktabs`, `hyperref` (liens | |
| 149 | + colorés). | |
| 150 | +- **Macros** : `\vect{x}` (vecteur gras via `\bm`), `\mat{W}`, `\Loss`, | |
| 151 | + `\E`, `\R`, `\softmax`, `\argmin`, `\argmax`. Notation cohérente partout ; | |
| 152 | + produit de Hadamard `\odot`. | |
| 153 | +- **Styles TikZ globaux** dans `main.tex` : `neuron`, `ninput` (bleu), | |
| 154 | + `nhidden` (orange), `noutput` (rouge), `bloc`, `fleche` (Stealth), `op` ; | |
| 155 | + palette nommée : `cinput` bleu, `chidden` orange, `coutput` rouge, | |
| 156 | + `cgate` vert, `cmem` violet. **Toutes** les figures réutilisent ces styles. | |
| 157 | +- **Environnements théorème** : `definition`, `remarque`, `propriete` | |
| 158 | + numérotés par chapitre. | |
| 159 | +- Toute équation référencée porte un `\label{eq:...}` et est citée par | |
| 160 | + `\eqref`. Toute figure : `figure` + `\caption` + `\label{fig:...}`. | |
| 161 | +- **Page de titre** : titre, sous-titre, mini-réseau TikZ décoratif, auteur | |
| 162 | + en italique, e-mail cliquable, date. | |
| 163 | +- **Résumé** + mots-clés avant la table des matières. | |
| 164 | +- **Bibliographie** `thebibliography` inline dans `main.tex` (pas de bibtex) : | |
| 165 | + Rosenblatt 1958, Rumelhart 1986, LeCun 1998, Hochreiter 1997, Cho 2014, | |
| 166 | + Srivastava 2014, Ioffe 2015, Kingma & Ba 2015, He 2016, Vaswani 2017, | |
| 167 | + Kingma & Welling 2014, Goodfellow 2014, Goodfellow et al. (livre) 2016 — | |
| 168 | + toutes citées au moins une fois dans le texte. | |
| 169 | + | |
| 170 | +## 6. Construction et vérification (workflow imposé) | |
| 171 | + | |
| 172 | +1. Vérifier les dépendances TeX (TeX Live complet requis : pgf/pgfplots, | |
| 173 | + babel-french, algorithms). | |
| 174 | +2. Créer `main.tex` puis chaque chapitre. | |
| 175 | +3. Compiler : `pdflatex -interaction=nonstopmode main.tex` **deux fois** | |
| 176 | + (TOC + références croisées). | |
| 177 | +4. **Zéro tolérance** : aucune erreur `!`, aucune `Undefined reference`, | |
| 178 | + aucune citation non résolue. Corriger et recompiler jusqu'à propreté. | |
| 179 | +5. **Contrôle visuel obligatoire** : rasteriser les pages contenant les | |
| 180 | + figures complexes (`pdftoppm -png`) et inspecter au minimum : page de | |
| 181 | + titre, cellule LSTM, architecture Transformer, convolution en grilles. | |
| 182 | + Corriger tout chevauchement de nœuds ou flèche mal routée, puis | |
| 183 | + recompiler. | |
| 184 | +6. Nettoyer les artefacts (`.aux`, `.log`, `.out`, `.toc`, PNG de contrôle). | |
| 185 | +7. Livrer : `main.pdf` + le dossier source complet (zip). | |
| 186 | + | |
| 187 | +## 7. Critères de « légendaire » | |
| 188 | + | |
| 189 | +- ~25 pages ou plus, densité soutenue mais lisible. | |
| 190 | +- Au moins **12 figures TikZ/pgfplots** et **50 équations numérotées**. | |
| 191 | +- Cohérence de notation absolue d'un chapitre à l'autre. | |
| 192 | +- Chaque affirmation non triviale adossée à une équation, une figure ou une | |
| 193 | + référence bibliographique. | |
| 194 | +- Le document doit pouvoir servir de support de cours autonome. | |
added
README.md
+103 −0
@@ -0,0 +1,103 @@ | ||
| 1 | +# Artificial Neural Networks — Methods, Equations and Graphical Representations | |
| 2 | + | |
| 3 | +**A complete, self-contained book on neural networks — every method presented with rigorous equations, an estimation algorithm in pseudocode, and a native TikZ figure.** | |
| 4 | + | |
| 5 | +*Simon-Pierre Boucher* · [contact@spboucher.ai](mailto:contact@spboucher.ai) | |
| 6 | + | |
| 7 | +--- | |
| 8 | + | |
| 9 | +## 📖 About | |
| 10 | + | |
| 11 | +This repository contains the full LaTeX source and compiled PDF of a 119-page book covering **all major families of artificial neural networks**, from Rosenblatt's perceptron (1958) to Kolmogorov–Arnold networks (2024). | |
| 12 | + | |
| 13 | +The book is built on one strict organizing principle: | |
| 14 | + | |
| 15 | +> **Every architecture = rigorous equations + estimation/training algorithm + faithful graphical representation.** | |
| 16 | + | |
| 17 | +Every single figure is drawn natively in **TikZ/pgfplots** — no imported images — so each diagram is exactly as precise as the equations it illustrates. Notation is unified across all chapters (bold lowercase vectors, bold uppercase matrices, Hadamard product `⊙`, shared color palette for inputs/hidden/output/gates/memory). | |
| 18 | + | |
| 19 | +## 📊 At a glance | |
| 20 | + | |
| 21 | +| Metric | Count | | |
| 22 | +|---|---:| | |
| 23 | +| Pages | 119 | | |
| 24 | +| Chapters | 13 (in 5 parts) | | |
| 25 | +| Numbered equations | 256 | | |
| 26 | +| Native TikZ/pgfplots figures | 42 | | |
| 27 | +| Estimation algorithms (pseudocode) | 26 | | |
| 28 | +| Comparison tables | 7 | | |
| 29 | +| Bibliography entries (all cited) | 36 | | |
| 30 | + | |
| 31 | +## 🗂 Contents | |
| 32 | + | |
| 33 | +### Part I — Foundations and Learning | |
| 34 | +1. **Foundations of Neural Networks** — artificial neuron, perceptron (Novikoff convergence theorem + learning algorithm), activation functions with derivatives, softmax, MLP, universal approximation theorem, XOR geometry, RBF contrast. | |
| 35 | +2. **Training Neural Networks** — empirical risk, MSE/cross-entropy, the softmax+CE gradient, the three backpropagation equations (with computational-graph figure and full pseudocode), SGD/Momentum/Nesterov/AdaGrad/RMSProp/**Adam** with bias correction, learning-rate schedules, Xavier/He initialization. | |
| 36 | +3. **Regularization** — bias–variance decomposition, L2/L1, inverted dropout, Batch & Layer Normalization, early stopping, label smoothing, gradient clipping. | |
| 37 | + | |
| 38 | +### Part II — Core Architectures | |
| 39 | +4. **Convolutional Neural Networks** — convolution with stride/padding (worked 5×5 ⊛ 3×3 grid figure), pooling, canonical architecture, **ResNet** residual block and its gradient, receptive fields, conv forward-pass algorithm. | |
| 40 | +5. **Recurrent Networks: RNN, LSTM and GRU** — folded/unfolded views, BPTT (Jacobian products, vanishing/exploding gradients, algorithm), the six **LSTM** equations with a detailed cell diagram, GRU, bidirectional RNNs, seq2seq. | |
| 41 | +6. **Attention and Transformers** — scaled dot-product attention with the 1/√d_k variance argument, multi-head attention, sinusoidal positional encoding, Add&Norm/FFN/causal masking, full **encoder–decoder architecture figure**, greedy decoding algorithm, BERT/GPT/T5. | |
| 42 | +7. **Modern Transformer Variants** — **Vision Transformer** (patches, [CLS], training algorithm), **Mixture of Experts** (noisy top-k routing, load balancing), efficient/linear attention, **state-space models and Mamba** (ZOH discretization, selective scan), scaling laws. | |
| 43 | + | |
| 44 | +### Part III — Graphs, Energy and Memory | |
| 45 | +8. **Graph Neural Networks** — message-passing framework, **GCN** (spectral derivation), **GraphSAGE** (minibatch estimation algorithm), **GAT**, **GIN** with the 1-WL expressiveness theorem, task heads. | |
| 46 | +9. **Energy-Based and Associative-Memory Networks** — **Hopfield** networks (energy descent theorem, capacity, recall algorithm), modern Hopfield ≈ attention, **Boltzmann machines**, **RBM** with **contrastive divergence (CD-k)**, deep belief networks with greedy layer-wise pretraining. | |
| 47 | + | |
| 48 | +### Part IV — Generative Models | |
| 49 | +10. **Autoencoders, VAEs and GANs** — hourglass autoencoder, **VAE** (ELBO with reconstruction/regularization decomposition, closed-form Gaussian KL, reparameterization trick, training algorithm), **GAN** (minimax game, optimal discriminator, Jensen–Shannon link, non-saturating loss, WGAN, alternating training algorithm), VAE/GAN/autoregressive comparison table. | |
| 50 | +11. **Normalizing Flows, Diffusion and Autoregressive Models** — change of variables, RealNVP coupling layers, **DDPM** (forward/reverse processes, variational bound, ε-parameterization, training *and* sampling algorithms), DDIM, classifier-free guidance, **WaveNet** dilated causal convolutions, energy-based models. | |
| 51 | + | |
| 52 | +### Part V — Specialized and Emerging Architectures | |
| 53 | +12. **Biologically Inspired Networks** — **spiking neurons** (LIF dynamics, f–I curve, STDP window, surrogate gradients), **self-organizing maps** (BMU, neighborhood kernel, training loop), **reservoir computing / echo state networks** (echo-state property, ridge readout). | |
| 54 | +13. **Emerging Architectures** — **capsule networks** (dynamic routing-by-agreement), **Neural ODEs** (adjoint method), **Neural Turing Machines** (full addressing pipeline), **Kolmogorov–Arnold networks** (superposition theorem, learnable edge functions). | |
| 55 | + | |
| 56 | +## 🔧 Building from source | |
| 57 | + | |
| 58 | +Requires a full TeX Live installation (pgf/pgfplots, `babel-english`, `algorithm`/`algpseudocode`, `booktabs`, `microtype`, `hyperref`). | |
| 59 | + | |
| 60 | +```bash | |
| 61 | +cd these # repository root | |
| 62 | +pdflatex -interaction=nonstopmode main.tex | |
| 63 | +pdflatex -interaction=nonstopmode main.tex # resolve TOC + cross-references | |
| 64 | +pdflatex -interaction=nonstopmode main.tex # settle part/page numbers | |
| 65 | +``` | |
| 66 | + | |
| 67 | +The build is clean by construction: **zero errors, zero undefined references, zero unresolved citations**. | |
| 68 | + | |
| 69 | +## 📁 Repository layout | |
| 70 | + | |
| 71 | +``` | |
| 72 | +. | |
| 73 | +├── README.md | |
| 74 | +├── CLAUDE.md # executable build specification of the book | |
| 75 | +├── main.pdf # compiled book (119 pages) | |
| 76 | +├── main.tex # preamble, cover, preface, abstract, TOC, bibliography | |
| 77 | +└── chapters/ | |
| 78 | + ├── 01-foundations.tex | |
| 79 | + ├── 02-training.tex | |
| 80 | + ├── 03-regularization.tex | |
| 81 | + ├── 04-cnn.tex | |
| 82 | + ├── 05-rnn.tex | |
| 83 | + ├── 06-attention.tex | |
| 84 | + ├── 07-modern-transformers.tex | |
| 85 | + ├── 08-gnn.tex | |
| 86 | + ├── 09-energy.tex | |
| 87 | + ├── 10-generative.tex | |
| 88 | + ├── 11-advanced-generative.tex | |
| 89 | + ├── 12-bio-inspired.tex | |
| 90 | + └── 13-emerging.tex | |
| 91 | +``` | |
| 92 | + | |
| 93 | +All global TikZ styles, the named color palette, notation macros and theorem environments are defined once in `main.tex` and reused by every chapter — guaranteeing visual and mathematical consistency across the whole book. | |
| 94 | + | |
| 95 | +## ✍️ Author | |
| 96 | + | |
| 97 | +**Simon-Pierre Boucher** | |
| 98 | +📧 [contact@spboucher.ai](mailto:contact@spboucher.ai) | |
| 99 | + | |
| 100 | +## 📜 License | |
| 101 | + | |
| 102 | +© 2026 Simon-Pierre Boucher. All rights reserved. | |
| 103 | +The source is published for reading and personal study; please contact the author for any other use. | |
added
chapters/01-foundations.tex
+706 −0
@@ -0,0 +1,706 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% Artificial Neural Networks — Methods, Equations and Graphical | |
| 3 | +% Representations | |
| 4 | +% Author : Simon-Pierre Boucher — contact@spboucher.ai | |
| 5 | +% Chapter 1 : Foundations of Neural Networks (chapters/01-foundations.tex) | |
| 6 | +% ============================================================================ | |
| 7 | + | |
| 8 | +\chapter{Foundations of Neural Networks}\label{chap:foundations} | |
| 9 | + | |
| 10 | +Artificial neural networks rest on a small number of ideas of remarkable | |
| 11 | +economy: a neuron is an affine map followed by a nonlinearity, a network is | |
| 12 | +a composition of such maps, and learning is the adjustment of the affine | |
| 13 | +parameters from examples. This chapter develops these ideas in their | |
| 14 | +historical and mathematical order. We begin with the artificial neuron as a | |
| 15 | +radical abstraction of its biological counterpart, formalize the perceptron | |
| 16 | +of Rosenblatt~\cite{rosenblatt1958} together with its learning rule and | |
| 17 | +convergence guarantee, examine the activation functions that give networks | |
| 18 | +their expressive power, and assemble these components into the multilayer | |
| 19 | +perceptron (MLP). The chapter closes with the universal approximation | |
| 20 | +theorem, which explains \emph{why} such compositions can, in principle, | |
| 21 | +represent essentially any continuous function. | |
| 22 | + | |
| 23 | +Throughout the book, vectors are bold lowercase ($\vect{x} \in \R^n$), | |
| 24 | +matrices bold uppercase ($\mat{W} \in \R^{m \times n}$), scalars italic, | |
| 25 | +and the Hadamard (elementwise) product is written $\odot$. The loss is | |
| 26 | +$\Loss$ and expectation is $\E$. | |
| 27 | + | |
| 28 | +% ---------------------------------------------------------------------------- | |
| 29 | +\section{From the Biological Neuron to the Artificial Neuron} | |
| 30 | +% ---------------------------------------------------------------------------- | |
| 31 | + | |
| 32 | +A biological neuron receives electrochemical signals through its | |
| 33 | +\emph{dendrites}, integrates them in the cell body, and — if the aggregated | |
| 34 | +membrane depolarization crosses a threshold — emits an action potential | |
| 35 | +that propagates along the axon to synapses connecting to downstream | |
| 36 | +neurons. Two properties of this system survive in the mathematical | |
| 37 | +abstraction: synaptic transmission is \emph{weighted} (a synapse may be | |
| 38 | +excitatory or inhibitory, strong or weak), and firing is \emph{nonlinear | |
| 39 | +and threshold-based}. Everything else — spike timing, refractory periods, | |
| 40 | +dendritic compartmentalization, neuromodulation — is discarded. The | |
| 41 | +artificial neuron is therefore best understood as a computational primitive | |
| 42 | +\emph{inspired by}, not a model of, neurophysiology. | |
| 43 | + | |
| 44 | +The founding formalization is due to McCulloch and Pitts (1943). Their | |
| 45 | +neuron takes Boolean inputs $x_i \in \{0,1\}$ and produces a Boolean output | |
| 46 | +through a threshold (Heaviside) function: | |
| 47 | +\begin{equation} | |
| 48 | + y \;=\; \Theta\!\left(\sum_{i=1}^{n} w_i x_i - \theta\right), | |
| 49 | + \qquad | |
| 50 | + \Theta(u) \;=\; | |
| 51 | + \begin{cases} | |
| 52 | + 1 & \text{if } u \geq 0,\\ | |
| 53 | + 0 & \text{if } u < 0. | |
| 54 | + \end{cases} | |
| 55 | + \label{eq:found-mp} | |
| 56 | +\end{equation} | |
| 57 | +Networks of such units can implement any Boolean function — AND is obtained | |
| 58 | +with $\theta = n$, OR with $\theta = 1$, NOT via inhibition — and, when | |
| 59 | +cycles are allowed, any finite-state automaton. The decisive limitation of | |
| 60 | +the McCulloch--Pitts unit is that it does not \emph{learn}: the weights | |
| 61 | +$w_i$ and the threshold $\theta$ must be set by the designer. The history | |
| 62 | +of neural networks is, in large part, the history of removing this | |
| 63 | +limitation. | |
| 64 | + | |
| 65 | +% ---------------------------------------------------------------------------- | |
| 66 | +\section{The Artificial Neuron} | |
| 67 | +% ---------------------------------------------------------------------------- | |
| 68 | + | |
| 69 | +\begin{definition}[Artificial neuron]\label{def:found-neuron} | |
| 70 | +An \emph{artificial neuron} with weights $\vect{w} \in \R^n$, bias | |
| 71 | +$b \in \R$ and activation function | |
| 72 | +$\varphi : \R \to \R$ is the map that assigns to an input | |
| 73 | +$\vect{x} \in \R^n$ the \emph{pre-activation} $z$ and the | |
| 74 | +\emph{activation} $a$ defined by | |
| 75 | +\begin{equation} | |
| 76 | + z \;=\; \vect{w}\transp \vect{x} + b \;=\; \sum_{i=1}^{n} w_i x_i + b, | |
| 77 | + \qquad | |
| 78 | + a \;=\; \varphi(z). | |
| 79 | + \label{eq:found-neuron} | |
| 80 | +\end{equation} | |
| 81 | +\end{definition} | |
| 82 | + | |
| 83 | +The computation~\eqref{eq:found-neuron} decomposes into three stages, | |
| 84 | +represented graphically in Figure~\ref{fig:found-neuron}: a weighted | |
| 85 | +aggregation of the inputs, the addition of a bias, and a pointwise | |
| 86 | +nonlinearity. The bias can be absorbed into the weights by augmenting the | |
| 87 | +input, $\vect{x} \leftarrow (\vect{x}, 1)$ and | |
| 88 | +$\vect{w} \leftarrow (\vect{w}, b)$, a convention we use freely whenever it | |
| 89 | +lightens notation. | |
| 90 | + | |
| 91 | +\begin{figure}[htbp] | |
| 92 | + \centering | |
| 93 | + \begin{tikzpicture} | |
| 94 | + % --- inputs ------------------------------------------------------------- | |
| 95 | + \node[ninput] (x1) at (0, 2.2) {$x_1$}; | |
| 96 | + \node[ninput] (x2) at (0, 1.1) {$x_2$}; | |
| 97 | + \node[ninput] (x3) at (0, 0.0) {$x_3$}; | |
| 98 | + \node (xd) at (0,-0.85) {$\vdots$}; | |
| 99 | + \node[ninput] (xn) at (0,-1.7) {$x_n$}; | |
| 100 | + % --- bias --------------------------------------------------------------- | |
| 101 | + \node[neuron, fill=black!8] (bias) at (3.2, 2.3) {$1$}; | |
| 102 | + % --- sum and activation --------------------------------------------------- | |
| 103 | + \node[op, minimum size=9mm] (sum) at (3.2, 0.25) {$\Sigma$}; | |
| 104 | + \node[nhidden, minimum size=9mm] (phi) at (5.5, 0.25) {$\varphi$}; | |
| 105 | + % --- output --------------------------------------------------------------- | |
| 106 | + \node (out) at (7.9, 0.25) {$a = \varphi(z)$}; | |
| 107 | + % --- edges with weight labels --------------------------------------------- | |
| 108 | + \draw[fleche] (x1) -- node[etiquette, above, pos=0.40] {$w_1$} (sum); | |
| 109 | + \draw[fleche] (x2) -- node[etiquette, above, pos=0.45] {$w_2$} (sum); | |
| 110 | + \draw[fleche] (x3) -- node[etiquette, above, pos=0.50] {$w_3$} (sum); | |
| 111 | + \draw[fleche] (xn) -- node[etiquette, below, pos=0.45] {$w_n$} (sum); | |
| 112 | + \draw[fleche] (bias) -- node[etiquette, right, pos=0.45] {$b$} (sum); | |
| 113 | + \draw[fleche] (sum) -- node[etiquette, above] {$z$} (phi); | |
| 114 | + \draw[fleche] (phi) -- (out); | |
| 115 | + \end{tikzpicture} | |
| 116 | + \caption{The artificial neuron of | |
| 117 | + Definition~\ref{def:found-neuron}. Inputs $x_1, \dots, x_n$ are | |
| 118 | + weighted by $w_1, \dots, w_n$ and aggregated by the summation node | |
| 119 | + $\Sigma$ together with the bias $b$; the pre-activation | |
| 120 | + $z = \vect{w}\transp\vect{x} + b$ then passes through the activation | |
| 121 | + function $\varphi$ to produce the output $a = \varphi(z)$.} | |
| 122 | + \label{fig:found-neuron} | |
| 123 | +\end{figure} | |
| 124 | + | |
| 125 | +\begin{remark}[Geometry of a single neuron] | |
| 126 | +The level set $\{\vect{x} : \vect{w}\transp\vect{x} + b = 0\}$ is a | |
| 127 | +hyperplane with normal vector $\vect{w}$, at signed distance | |
| 128 | +$-b/\lVert\vect{w}\rVert$ from the origin. A single neuron with a monotone | |
| 129 | +activation therefore responds along one direction of input space only: it | |
| 130 | +is a \emph{linear feature detector} followed by a nonlinear read-out. | |
| 131 | +Everything a deep network does can be traced back to compositions of these | |
| 132 | +elementary half-space responses. | |
| 133 | +\end{remark} | |
| 134 | + | |
| 135 | +% ---------------------------------------------------------------------------- | |
| 136 | +\section{The Rosenblatt Perceptron} | |
| 137 | +% ---------------------------------------------------------------------------- | |
| 138 | + | |
| 139 | +Rosenblatt's perceptron~\cite{rosenblatt1958} is historically the first | |
| 140 | +neuron that \emph{learns}: its weights are adjusted from labelled examples | |
| 141 | +rather than hand-designed. For binary classification with labels | |
| 142 | +$y \in \{-1, +1\}$, the perceptron predicts with the sign of the | |
| 143 | +pre-activation, | |
| 144 | +\begin{equation} | |
| 145 | + \hat{y} \;=\; \operatorname{sign}\!\left(\vect{w}\transp\vect{x} + b\right) | |
| 146 | + \;=\; | |
| 147 | + \begin{cases} | |
| 148 | + +1 & \text{if } \vect{w}\transp\vect{x} + b \geq 0,\\ | |
| 149 | + -1 & \text{otherwise,} | |
| 150 | + \end{cases} | |
| 151 | + \label{eq:found-perceptron-decision} | |
| 152 | +\end{equation} | |
| 153 | +so that the decision boundary is the hyperplane | |
| 154 | +$\vect{w}\transp\vect{x} + b = 0$. Learning proceeds \emph{online}: the | |
| 155 | +examples $(\vect{x}^{(k)}, y^{(k)})$ are presented one at a time, and each | |
| 156 | +misclassified example triggers the update | |
| 157 | +\begin{equation} | |
| 158 | + \vect{w} \;\leftarrow\; \vect{w} + \eta\, y^{(k)} \vect{x}^{(k)}, | |
| 159 | + \qquad | |
| 160 | + b \;\leftarrow\; b + \eta\, y^{(k)}, | |
| 161 | + \label{eq:found-perceptron-update} | |
| 162 | +\end{equation} | |
| 163 | +with learning rate $\eta > 0$; correctly classified examples produce no | |
| 164 | +update. Rule~\eqref{eq:found-perceptron-update} is intuitive — it rotates | |
| 165 | +the hyperplane toward each mistake — and it requires no differentiability. | |
| 166 | +Indeed, the step function of~\eqref{eq:found-perceptron-decision} has zero | |
| 167 | +derivative almost everywhere, so the perceptron rule is \emph{not} | |
| 168 | +gradient descent on the misclassification error. It is, however, | |
| 169 | +exactly stochastic subgradient descent on the \emph{perceptron criterion} | |
| 170 | +\begin{equation} | |
| 171 | + \Loss(\vect{w}, b) | |
| 172 | + \;=\; | |
| 173 | + \max\!\left(0,\; -y\left(\vect{w}\transp\vect{x} + b\right)\right), | |
| 174 | + \label{eq:found-perceptron-criterion} | |
| 175 | +\end{equation} | |
| 176 | +which vanishes on correct classifications and grows linearly with the | |
| 177 | +margin of error otherwise — an early ancestor of the hinge loss. The | |
| 178 | +complete estimation procedure, which simply cycles through the data | |
| 179 | +applying~\eqref{eq:found-perceptron-update} until an error-free pass, | |
| 180 | +is summarized in Algorithm~\ref{alg:found-perceptron}. | |
| 181 | + | |
| 182 | +\begin{algorithm}[htbp] | |
| 183 | + \caption{Perceptron learning (Rosenblatt)} | |
| 184 | + \label{alg:found-perceptron} | |
| 185 | + \begin{algorithmic}[1] | |
| 186 | + \Require training set $\{(\vect{x}^{(k)}, y^{(k)})\}_{k=1}^{m}$ with | |
| 187 | + $y^{(k)} \in \{-1,+1\}$, learning rate $\eta > 0$ | |
| 188 | + \State $\vect{w} \gets \vect{0}$, \quad $b \gets 0$ | |
| 189 | + \Repeat | |
| 190 | + \State $\mathit{errors} \gets 0$ | |
| 191 | + \For{$k = 1, \dots, m$} | |
| 192 | + \If{$y^{(k)}\bigl(\vect{w}\transp\vect{x}^{(k)} + b\bigr) \leq 0$} | |
| 193 | + \Comment{misclassified example} | |
| 194 | + \State $\vect{w} \gets \vect{w} + \eta\, y^{(k)}\vect{x}^{(k)}$ | |
| 195 | + \State $b \gets b + \eta\, y^{(k)}$ | |
| 196 | + \State $\mathit{errors} \gets \mathit{errors} + 1$ | |
| 197 | + \EndIf | |
| 198 | + \EndFor | |
| 199 | + \Until{$\mathit{errors} = 0$} | |
| 200 | + \State \Return $\vect{w},\, b$ | |
| 201 | + \end{algorithmic} | |
| 202 | +\end{algorithm} | |
| 203 | + | |
| 204 | +The perceptron admits one of the cleanest guarantees in machine learning, | |
| 205 | +formalized by Novikoff (1962). | |
| 206 | + | |
| 207 | +\begin{theorem}[Perceptron convergence]\label{thm:found-convergence} | |
| 208 | +Suppose the training set | |
| 209 | +$\{(\vect{x}^{(k)}, y^{(k)})\}_{k=1}^{m}$ is linearly separable with | |
| 210 | +margin $\gamma > 0$: there exists a unit vector $\vect{w}^{\star}$, | |
| 211 | +$\lVert\vect{w}^{\star}\rVert = 1$, such that | |
| 212 | +$y^{(k)}\,\vect{w}^{\star\top}\vect{x}^{(k)} \geq \gamma$ for all $k$. | |
| 213 | +Let $R = \max_k \lVert\vect{x}^{(k)}\rVert$. Then the perceptron | |
| 214 | +algorithm~\eqref{eq:found-perceptron-update}, started from | |
| 215 | +$\vect{w} = \vect{0}$, makes at most | |
| 216 | +\begin{equation} | |
| 217 | + T \;\leq\; \left(\frac{R}{\gamma}\right)^{2} | |
| 218 | + \label{eq:found-novikoff} | |
| 219 | +\end{equation} | |
| 220 | +updates before finding a separating hyperplane. | |
| 221 | +\end{theorem} | |
| 222 | + | |
| 223 | +\begin{remark} | |
| 224 | +The bound~\eqref{eq:found-novikoff} is independent of the input dimension | |
| 225 | +$n$ and of the number of examples $m$: only the \emph{normalized margin} | |
| 226 | +$\gamma / R$ matters. This is an early instance of a recurring theme — | |
| 227 | +the difficulty of a learning problem is governed by geometric quantities, | |
| 228 | +not by raw dimensionality. | |
| 229 | +\end{remark} | |
| 230 | + | |
| 231 | +Shortly after the perceptron, Widrow and Hoff (1960) introduced ADALINE, | |
| 232 | +which differs in one crucial respect: the error is computed on the | |
| 233 | +\emph{linear} pre-activation $z = \vect{w}\transp\vect{x} + b$ rather than | |
| 234 | +on the thresholded output. Minimizing the squared error | |
| 235 | +$\tfrac{1}{2}(d - z)^2$ against a target $d$ gives the \emph{delta rule} | |
| 236 | +(or LMS, least-mean-squares, rule) | |
| 237 | +\begin{equation} | |
| 238 | + \vect{w} \;\leftarrow\; \vect{w} + \eta\,(d - z)\,\vect{x}, | |
| 239 | + \label{eq:found-adaline} | |
| 240 | +\end{equation} | |
| 241 | +which, unlike~\eqref{eq:found-perceptron-update}, \emph{is} genuine | |
| 242 | +stochastic gradient descent on a differentiable objective. The delta rule | |
| 243 | +keeps improving even after the data are separated, driving the weights | |
| 244 | +toward the minimum mean-squared-error solution, and it does not diverge on | |
| 245 | +non-separable data. Gradient-based learning of exactly this kind — extended | |
| 246 | +to many layers — is the subject of the next chapter. | |
| 247 | + | |
| 248 | +% ---------------------------------------------------------------------------- | |
| 249 | +\subsection{The Limits of Linear Separation: XOR} | |
| 250 | +% ---------------------------------------------------------------------------- | |
| 251 | + | |
| 252 | +The perceptron can only realize decision boundaries that are hyperplanes. | |
| 253 | +Minsky and Papert (1969) analyzed this restriction with mathematical | |
| 254 | +precision, and their canonical counterexample is the exclusive-or function | |
| 255 | +XOR, shown in Figure~\ref{fig:found-xor}: the positive examples | |
| 256 | +$(0,1)$ and $(1,0)$ sit on one diagonal of the unit square, the negative | |
| 257 | +examples $(0,0)$ and $(1,1)$ on the other, and no straight line separates | |
| 258 | +two diagonals. The proof is elementary. If a separating line | |
| 259 | +$w_1 x_1 + w_2 x_2 + b = 0$ existed, then $(0,0) \mapsto 0$ forces | |
| 260 | +$b < 0$, while the positive cases force $w_1 + b \geq 0$ and | |
| 261 | +$w_2 + b \geq 0$; adding these, $w_1 + w_2 + 2b \geq 0$, hence | |
| 262 | +$w_1 + w_2 + b \geq -b > 0$, which would classify $(1,1)$ as positive — a | |
| 263 | +contradiction. | |
| 264 | + | |
| 265 | +\begin{figure}[htbp] | |
| 266 | + \centering | |
| 267 | + \begin{tikzpicture}[scale=2.6] | |
| 268 | + % axes | |
| 269 | + \draw[fleche, black!60] (-0.30, 0) -- (1.55, 0) | |
| 270 | + node[below, font=\small] {$x_1$}; | |
| 271 | + \draw[fleche, black!60] (0, -0.30) -- (0, 1.55) | |
| 272 | + node[left, font=\small] {$x_2$}; | |
| 273 | + % the two hidden-unit hyperplanes | |
| 274 | + \draw[cgate, dashed, thick] (-0.15, 0.65) -- (0.65, -0.15) | |
| 275 | + node[etiquette, below, text=cgate!60!black] | |
| 276 | + {$x_1 + x_2 = \tfrac{1}{2}$}; | |
| 277 | + \draw[cgate, dashed, thick] (0.38, 1.18) -- (1.18, 0.38) | |
| 278 | + node[etiquette, below right, text=cgate!60!black] | |
| 279 | + {$x_1 + x_2 = \tfrac{3}{2}$}; | |
| 280 | + % class 0 (blue) and class 1 (red) points | |
| 281 | + \node[circle, fill=cinput, draw=cinput!60!black, inner sep=2.4pt, | |
| 282 | + label={[etiquette]below left:$(0,0)$}] at (0,0) {}; | |
| 283 | + \node[circle, fill=cinput, draw=cinput!60!black, inner sep=2.4pt, | |
| 284 | + label={[etiquette]above right:$(1,1)$}] at (1,1) {}; | |
| 285 | + \node[circle, fill=coutput, draw=coutput!60!black, inner sep=2.4pt, | |
| 286 | + label={[etiquette]above left:$(0,1)$}] at (0,1) {}; | |
| 287 | + \node[circle, fill=coutput, draw=coutput!60!black, inner sep=2.4pt, | |
| 288 | + label={[etiquette]below right:$(1,0)$}] at (1,0) {}; | |
| 289 | + \end{tikzpicture} | |
| 290 | + \caption{The XOR problem. Negative examples (blue) occupy one diagonal | |
| 291 | + of the unit square, positive examples (red) the other; no single line | |
| 292 | + separates the two classes. The two dashed lines show the resolution by | |
| 293 | + a hidden layer: two threshold units implementing | |
| 294 | + $x_1 + x_2 \geq \tfrac{1}{2}$ (OR) and $x_1 + x_2 \leq \tfrac{3}{2}$ | |
| 295 | + (NAND) carve out the strip between the lines, and an AND unit on top | |
| 296 | + of them computes XOR exactly.} | |
| 297 | + \label{fig:found-xor} | |
| 298 | +\end{figure} | |
| 299 | + | |
| 300 | +Crucially, XOR \emph{is} solvable with one hidden layer: | |
| 301 | +$\mathrm{XOR}(x_1, x_2) = \mathrm{OR}(x_1, x_2) \wedge | |
| 302 | +\neg\,\mathrm{AND}(x_1, x_2)$, i.e.\ two threshold units feeding a third. | |
| 303 | +Minsky and Papert's pessimism about such multilayer extensions — for which | |
| 304 | +no training algorithm was then known — is commonly cited as a trigger of | |
| 305 | +the first connectionist winter, which lasted until the mid-1980s. The | |
| 306 | +missing algorithm, backpropagation, is derived in the next chapter; the | |
| 307 | +representational question of what multilayer networks \emph{can} express | |
| 308 | +is answered at the end of this one. | |
| 309 | + | |
| 310 | +% ---------------------------------------------------------------------------- | |
| 311 | +\section{Activation Functions} | |
| 312 | +% ---------------------------------------------------------------------------- | |
| 313 | + | |
| 314 | +The nonlinearity $\varphi$ is not a detail: without it, a network of any | |
| 315 | +depth collapses. If every layer applied only an affine map, their | |
| 316 | +composition would again be affine, and the deepest network would be | |
| 317 | +exactly as expressive as a single linear layer. The choice of $\varphi$ | |
| 318 | +also governs how gradients flow backward through the network — as we will | |
| 319 | +see in the next chapter, training multiplies derivatives $\varphi'$ across | |
| 320 | +layers, so activations that \emph{saturate} (have near-zero derivative | |
| 321 | +over most of their domain) starve deep networks of learning signal. We now | |
| 322 | +review the five activations that dominate practice, giving for each the | |
| 323 | +definition and the derivative; Figure~\ref{fig:found-activations} plots | |
| 324 | +them side by side. | |
| 325 | + | |
| 326 | +\paragraph{Sigmoid.} The logistic sigmoid maps $\R$ onto $(0,1)$ and was | |
| 327 | +historically the default choice: | |
| 328 | +\begin{equation} | |
| 329 | + \sigma(z) \;=\; \frac{1}{1 + e^{-z}}, | |
| 330 | + \qquad | |
| 331 | + \sigma'(z) \;=\; \sigma(z)\bigl(1 - \sigma(z)\bigr). | |
| 332 | + \label{eq:found-sigmoid} | |
| 333 | +\end{equation} | |
| 334 | +Its derivative is bounded by $\sigma'(0) = \tfrac{1}{4}$ and decays | |
| 335 | +exponentially for $\lvert z \rvert \gtrsim 5$: the function | |
| 336 | +\emph{saturates}, and its outputs are not zero-centred, which induces | |
| 337 | +correlated gradient signs within a layer. It survives today mainly in | |
| 338 | +gates and binary output units. | |
| 339 | + | |
| 340 | +\paragraph{Hyperbolic tangent.} A rescaled sigmoid, | |
| 341 | +$\tanh(z) = 2\sigma(2z) - 1$, with range $(-1, 1)$: | |
| 342 | +\begin{equation} | |
| 343 | + \tanh(z) \;=\; \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}, | |
| 344 | + \qquad | |
| 345 | + \frac{d}{dz}\tanh(z) \;=\; 1 - \tanh^{2}(z). | |
| 346 | + \label{eq:found-tanh} | |
| 347 | +\end{equation} | |
| 348 | +Zero-centred and with maximal derivative $1$ at the origin, it is better | |
| 349 | +conditioned than the sigmoid for hidden layers, though still saturating; | |
| 350 | +it remains standard inside recurrent cells. | |
| 351 | + | |
| 352 | +\paragraph{ReLU.} The rectified linear unit, repopularized around | |
| 353 | +2010--2012, is the piecewise-linear map | |
| 354 | +\begin{equation} | |
| 355 | + \mathrm{ReLU}(z) \;=\; \max(0, z), | |
| 356 | + \qquad | |
| 357 | + \mathrm{ReLU}'(z) \;=\; | |
| 358 | + \begin{cases} | |
| 359 | + 1 & z > 0,\\ | |
| 360 | + 0 & z < 0, | |
| 361 | + \end{cases} | |
| 362 | + \label{eq:found-relu} | |
| 363 | +\end{equation} | |
| 364 | +with the convention $\mathrm{ReLU}'(0) = 0$ at the (measure-zero) | |
| 365 | +non-differentiable point. For positive inputs the gradient is exactly $1$, | |
| 366 | +so ReLU does not saturate on its active half — the property that unlocked | |
| 367 | +the training of genuinely deep networks. Its failure mode is the | |
| 368 | +\emph{dying ReLU}: a unit pushed into the negative regime for every input | |
| 369 | +receives zero gradient forever and never recovers. | |
| 370 | + | |
| 371 | +\paragraph{Leaky ReLU.} A minimal repair of the dying-unit problem replaces | |
| 372 | +the zero slope with a small $\alpha > 0$ (typically $\alpha = 0.01$): | |
| 373 | +\begin{equation} | |
| 374 | + \mathrm{LReLU}(z) \;=\; \max(\alpha z,\, z) | |
| 375 | + \;=\; | |
| 376 | + \begin{cases} | |
| 377 | + z & z \geq 0,\\ | |
| 378 | + \alpha z & z < 0, | |
| 379 | + \end{cases} | |
| 380 | + \qquad | |
| 381 | + \mathrm{LReLU}'(z) \;=\; | |
| 382 | + \begin{cases} | |
| 383 | + 1 & z > 0,\\ | |
| 384 | + \alpha & z < 0. | |
| 385 | + \end{cases} | |
| 386 | + \label{eq:found-leaky} | |
| 387 | +\end{equation} | |
| 388 | +The parametric variant (PReLU) learns $\alpha$ per channel at negligible | |
| 389 | +cost. | |
| 390 | + | |
| 391 | +\paragraph{GELU.} The Gaussian error linear unit weights its input by the | |
| 392 | +probability that a standard Gaussian falls below it. With $\Phi$ and | |
| 393 | +$\phi$ the standard normal cumulative distribution function and density, | |
| 394 | +\begin{equation} | |
| 395 | + \mathrm{GELU}(z) \;=\; z\,\Phi(z) | |
| 396 | + \;=\; \frac{z}{2}\left[1 + \operatorname{erf}\!\left( | |
| 397 | + \frac{z}{\sqrt{2}}\right)\right], | |
| 398 | + \qquad | |
| 399 | + \frac{d}{dz}\mathrm{GELU}(z) \;=\; \Phi(z) + z\,\phi(z). | |
| 400 | + \label{eq:found-gelu} | |
| 401 | +\end{equation} | |
| 402 | +GELU can be read as a deterministic version of stochastic gating: instead | |
| 403 | +of dropping a unit with probability $1 - \Phi(z)$, it scales the unit by | |
| 404 | +the expected mask. It is smooth, non-monotonic (a shallow negative dip | |
| 405 | +near $z \approx -0.75$), and is the default activation in Transformer | |
| 406 | +architectures. Implementations commonly use the tanh approximation | |
| 407 | +\begin{equation} | |
| 408 | + \mathrm{GELU}(z) \;\approx\; \tfrac{1}{2}\, z \left(1 + \tanh\!\left[ | |
| 409 | + \sqrt{\tfrac{2}{\pi}}\left(z + 0.044715\, z^{3}\right) | |
| 410 | + \right]\right). | |
| 411 | + \label{eq:found-gelu-approx} | |
| 412 | +\end{equation} | |
| 413 | + | |
| 414 | +\begin{figure}[htbp] | |
| 415 | + \centering | |
| 416 | + \begin{tikzpicture} | |
| 417 | + \begin{axis}[ | |
| 418 | + width=0.50\textwidth, height=6.2cm, | |
| 419 | + title={\small (a) Activation functions}, | |
| 420 | + xlabel={$z$}, ylabel={$\varphi(z)$}, | |
| 421 | + xmin=-4, xmax=4, ymin=-1.6, ymax=4, | |
| 422 | + grid=major, grid style={black!12}, | |
| 423 | + legend pos=north west, | |
| 424 | + legend style={font=\scriptsize, fill=white, fill opacity=0.85, | |
| 425 | + text opacity=1, draw=black!30}, | |
| 426 | + every axis plot/.append style={thick}, | |
| 427 | + samples=200, domain=-4:4, | |
| 428 | + ] | |
| 429 | + \addplot[cinput] {1/(1+exp(-x))}; | |
| 430 | + \addlegendentry{sigmoid} | |
| 431 | + \addplot[chidden] {tanh(x)}; | |
| 432 | + \addlegendentry{$\tanh$} | |
| 433 | + \addplot[coutput] {max(0,x)}; | |
| 434 | + \addlegendentry{ReLU} | |
| 435 | + \addplot[cgate, dashed] {max(0,x) + 0.1*min(0,x)}; | |
| 436 | + \addlegendentry{Leaky ReLU} | |
| 437 | + \addplot[cmem] | |
| 438 | + {0.5*x*(1 + tanh(0.7978845608*(x + 0.044715*x^3)))}; | |
| 439 | + \addlegendentry{GELU} | |
| 440 | + \end{axis} | |
| 441 | + \end{tikzpicture}\hfill | |
| 442 | + \begin{tikzpicture} | |
| 443 | + \begin{axis}[ | |
| 444 | + width=0.50\textwidth, height=6.2cm, | |
| 445 | + title={\small (b) Derivatives}, | |
| 446 | + xlabel={$z$}, ylabel={$\varphi'(z)$}, | |
| 447 | + xmin=-4, xmax=4, ymin=-0.25, ymax=1.35, | |
| 448 | + grid=major, grid style={black!12}, | |
| 449 | + legend pos=north west, | |
| 450 | + legend style={font=\scriptsize, fill=white, fill opacity=0.85, | |
| 451 | + text opacity=1, draw=black!30}, | |
| 452 | + every axis plot/.append style={thick}, | |
| 453 | + samples=200, domain=-4:4, | |
| 454 | + ] | |
| 455 | + \addplot[cinput] {exp(-x)/((1+exp(-x))^2)}; | |
| 456 | + \addlegendentry{sigmoid$'$} | |
| 457 | + \addplot[chidden] {1 - tanh(x)^2}; | |
| 458 | + \addlegendentry{$\tanh'$} | |
| 459 | + \addplot[coutput] coordinates {(-4,0) (0,0)}; | |
| 460 | + \addlegendentry{ReLU$'$} | |
| 461 | + \addplot[coutput, forget plot] coordinates {(0,1) (4,1)}; | |
| 462 | + \addplot[cgate, dashed] coordinates {(-4,0.1) (0,0.1)}; | |
| 463 | + \addlegendentry{Leaky ReLU$'$} | |
| 464 | + \addplot[cgate, dashed, forget plot] coordinates {(0,1) (4,1)}; | |
| 465 | + \addplot[cmem] | |
| 466 | + {0.5*(1 + tanh(0.7978845608*(x + 0.044715*x^3))) | |
| 467 | + + x*0.3989422804*exp(-x^2/2)}; | |
| 468 | + \addlegendentry{GELU$'$} | |
| 469 | + \end{axis} | |
| 470 | + \end{tikzpicture} | |
| 471 | + \caption{The five standard activation functions | |
| 472 | + \eqref{eq:found-sigmoid}--\eqref{eq:found-gelu} (left) and their | |
| 473 | + derivatives (right). Leaky ReLU is drawn with $\alpha = 0.1$ for | |
| 474 | + visibility (in practice $\alpha = 0.01$ is typical). Note the | |
| 475 | + saturation of sigmoid and $\tanh$ — their derivatives vanish for | |
| 476 | + $\lvert z\rvert \gtrsim 4$ — against the constant unit slope of the | |
| 477 | + ReLU family on the positive half-line, and the smooth non-monotonic | |
| 478 | + profile of GELU.} | |
| 479 | + \label{fig:found-activations} | |
| 480 | +\end{figure} | |
| 481 | + | |
| 482 | +% ---------------------------------------------------------------------------- | |
| 483 | +\subsection{The Softmax Function} | |
| 484 | +% ---------------------------------------------------------------------------- | |
| 485 | + | |
| 486 | +Multiclass classification requires a vector-valued output layer that maps | |
| 487 | +$K$ real \emph{logits} to a probability distribution over $K$ classes. | |
| 488 | +This is the role of the softmax: | |
| 489 | +\begin{equation} | |
| 490 | + \softmax(\vect{z})_i \;=\; \frac{e^{z_i}}{\sum_{j=1}^{K} e^{z_j}}, | |
| 491 | + \qquad i = 1, \dots, K, | |
| 492 | + \label{eq:found-softmax} | |
| 493 | +\end{equation} | |
| 494 | +whose outputs are positive and sum to one. Its Jacobian has the compact | |
| 495 | +form | |
| 496 | +\begin{equation} | |
| 497 | + \frac{\partial\, \softmax(\vect{z})_i}{\partial z_j} | |
| 498 | + \;=\; | |
| 499 | + \softmax(\vect{z})_i \left(\delta_{ij} - \softmax(\vect{z})_j\right), | |
| 500 | + \label{eq:found-softmax-jacobian} | |
| 501 | +\end{equation} | |
| 502 | +with $\delta_{ij}$ the Kronecker delta — an expression that will combine | |
| 503 | +particularly gracefully with the cross-entropy loss in the next chapter. | |
| 504 | + | |
| 505 | +\begin{remark}[Shift invariance and temperature] | |
| 506 | +Softmax is invariant to adding a constant to every logit, | |
| 507 | +$\softmax(\vect{z} + c\vect{1}) = \softmax(\vect{z})$; numerically stable | |
| 508 | +implementations exploit this by subtracting $\max_j z_j$ before | |
| 509 | +exponentiating. Dividing the logits by a \emph{temperature} $T > 0$ | |
| 510 | +interpolates between a hard maximum ($T \to 0$) and the uniform | |
| 511 | +distribution ($T \to \infty$). | |
| 512 | +\end{remark} | |
| 513 | + | |
| 514 | +% ---------------------------------------------------------------------------- | |
| 515 | +\section{The Multilayer Perceptron} | |
| 516 | +% ---------------------------------------------------------------------------- | |
| 517 | + | |
| 518 | +A single neuron detects one linear feature; the multilayer perceptron | |
| 519 | +composes entire \emph{layers} of them, each layer feeding the next. | |
| 520 | + | |
| 521 | +\begin{definition}[Multilayer perceptron]\label{def:found-mlp} | |
| 522 | +An MLP with $L$ layers of widths $n_1, \dots, n_L$ on inputs of dimension | |
| 523 | +$n_0$ is defined by weight matrices | |
| 524 | +$\mat{W}^{(\ell)} \in \R^{n_\ell \times n_{\ell-1}}$ and bias vectors | |
| 525 | +$\vect{b}^{(\ell)} \in \R^{n_\ell}$, $\ell = 1, \dots, L$. Setting | |
| 526 | +$\vect{a}^{(0)} = \vect{x}$, the \emph{forward propagation} computes, for | |
| 527 | +$\ell = 1, \dots, L$, | |
| 528 | +\begin{align} | |
| 529 | + \vect{z}^{(\ell)} &= \mat{W}^{(\ell)} \vect{a}^{(\ell-1)} | |
| 530 | + + \vect{b}^{(\ell)}, | |
| 531 | + \label{eq:found-mlp-z}\\ | |
| 532 | + \vect{a}^{(\ell)} &= \varphi^{(\ell)}\!\left(\vect{z}^{(\ell)}\right), | |
| 533 | + \label{eq:found-mlp-a} | |
| 534 | +\end{align} | |
| 535 | +where $\varphi^{(\ell)}$ acts componentwise, and the network output is | |
| 536 | +$\hat{\vect{y}} = \vect{a}^{(L)}$. | |
| 537 | +\end{definition} | |
| 538 | + | |
| 539 | +Componentwise, \eqref{eq:found-mlp-z} reads | |
| 540 | +\begin{equation} | |
| 541 | + z^{(\ell)}_j \;=\; \sum_{i=1}^{n_{\ell-1}} | |
| 542 | + W^{(\ell)}_{ji}\, a^{(\ell-1)}_i + b^{(\ell)}_j, | |
| 543 | + \label{eq:found-mlp-component} | |
| 544 | +\end{equation} | |
| 545 | +so that each unit of layer $\ell$ is exactly an artificial neuron in the | |
| 546 | +sense of Definition~\ref{def:found-neuron}, whose inputs are the | |
| 547 | +activations of the previous layer. The final activation | |
| 548 | +$\varphi^{(L)}$ is chosen to match the task: identity for regression, | |
| 549 | +sigmoid for binary classification, softmax~\eqref{eq:found-softmax} for | |
| 550 | +multiclass classification, while hidden layers use one of the | |
| 551 | +nonlinearities of Figure~\ref{fig:found-activations}. A network with two | |
| 552 | +hidden layers is drawn in Figure~\ref{fig:found-mlp}. | |
| 553 | + | |
| 554 | +\begin{figure}[htbp] | |
| 555 | + \centering | |
| 556 | + \begin{tikzpicture} | |
| 557 | + % --- input layer -------------------------------------------------------- | |
| 558 | + \foreach \i in {1,2,3} | |
| 559 | + \node[ninput] (x\i) at (0, 2.2-\i*1.1) {$x_{\i}$}; | |
| 560 | + % --- hidden layer 1 ----------------------------------------------------- | |
| 561 | + \foreach \j in {1,...,4} | |
| 562 | + \node[nhidden] (h1\j) at (2.7, 2.75-\j*1.1) {}; | |
| 563 | + % --- hidden layer 2 ----------------------------------------------------- | |
| 564 | + \foreach \j in {1,...,4} | |
| 565 | + \node[nhidden] (h2\j) at (5.4, 2.75-\j*1.1) {}; | |
| 566 | + % --- output layer ------------------------------------------------------- | |
| 567 | + \foreach \k in {1,2} | |
| 568 | + \node[noutput] (o\k) at (8.1, 1.1-\k*1.1) {}; | |
| 569 | + % --- connections ---------------------------------------------------------- | |
| 570 | + \foreach \i in {1,2,3} \foreach \j in {1,...,4} | |
| 571 | + \draw[black!35, semithick] (x\i) -- (h1\j); | |
| 572 | + \foreach \i in {1,...,4} \foreach \j in {1,...,4} | |
| 573 | + \draw[black!35, semithick] (h1\i) -- (h2\j); | |
| 574 | + \foreach \i in {1,...,4} \foreach \k in {1,2} | |
| 575 | + \draw[black!35, semithick] (h2\i) -- (o\k); | |
| 576 | + % --- outputs --------------------------------------------------------------- | |
| 577 | + \draw[fleche] (o1) -- ++(1.3,0) node[right] {$\hat{y}_1$}; | |
| 578 | + \draw[fleche] (o2) -- ++(1.3,0) node[right] {$\hat{y}_2$}; | |
| 579 | + % --- weight-matrix labels --------------------------------------------------- | |
| 580 | + \node[etiquette] at (1.35, 2.55) {$\mat{W}^{(1)}, \vect{b}^{(1)}$}; | |
| 581 | + \node[etiquette] at (4.05, 2.55) {$\mat{W}^{(2)}, \vect{b}^{(2)}$}; | |
| 582 | + \node[etiquette] at (6.75, 2.55) {$\mat{W}^{(3)}, \vect{b}^{(3)}$}; | |
| 583 | + % --- layer labels --------------------------------------------------------- | |
| 584 | + \node[etiquette, align=center] at (0, -2.6) | |
| 585 | + {input layer\\ $\vect{a}^{(0)} = \vect{x}$}; | |
| 586 | + \node[etiquette, align=center] at (2.7, -2.6) | |
| 587 | + {hidden layer 1\\ $\vect{a}^{(1)}$}; | |
| 588 | + \node[etiquette, align=center] at (5.4, -2.6) | |
| 589 | + {hidden layer 2\\ $\vect{a}^{(2)}$}; | |
| 590 | + \node[etiquette, align=center] at (8.1, -2.6) | |
| 591 | + {output layer\\ $\hat{\vect{y}} = \vect{a}^{(3)}$}; | |
| 592 | + \end{tikzpicture} | |
| 593 | + \caption{A fully connected network with two hidden layers | |
| 594 | + ($n_0 = 3$, $n_1 = n_2 = 4$, $n_3 = 2$). Each edge carries one entry | |
| 595 | + of a weight matrix $\mat{W}^{(\ell)}$; each column of neurons applies | |
| 596 | + the affine map~\eqref{eq:found-mlp-z} followed by the pointwise | |
| 597 | + nonlinearity~\eqref{eq:found-mlp-a}.} | |
| 598 | + \label{fig:found-mlp} | |
| 599 | +\end{figure} | |
| 600 | + | |
| 601 | +\begin{remark}[Batched computation] | |
| 602 | +For a mini-batch of $m$ examples stacked as rows of | |
| 603 | +$\mat{X} \in \R^{m \times n_0}$, forward propagation becomes | |
| 604 | +$\mat{Z}^{(\ell)} = \mat{A}^{(\ell-1)} \mat{W}^{(\ell)\top} | |
| 605 | ++ \vect{1}_m \vect{b}^{(\ell)\top}$ — a chain of dense matrix products. | |
| 606 | +This is why MLPs map so efficiently onto modern hardware: the entire | |
| 607 | +network is a sequence of GEMM primitives. | |
| 608 | +\end{remark} | |
| 609 | + | |
| 610 | +\begin{remark}[Why the nonlinearity is essential] | |
| 611 | +If $\varphi^{(\ell)} = \mathrm{id}$ for all $\ell$, then | |
| 612 | +$\hat{\vect{y}} = \mat{W}^{(L)} \cdots \mat{W}^{(1)} \vect{x} + \vect{c}$ | |
| 613 | +for some constant $\vect{c}$: the composition of affine maps is affine, | |
| 614 | +and depth buys nothing. All the expressive power of the MLP resides in the | |
| 615 | +interleaving of~\eqref{eq:found-mlp-z} with the | |
| 616 | +nonlinearity~\eqref{eq:found-mlp-a}. | |
| 617 | +\end{remark} | |
| 618 | + | |
| 619 | +% ---------------------------------------------------------------------------- | |
| 620 | +\subsection{Universal Approximation} | |
| 621 | +% ---------------------------------------------------------------------------- | |
| 622 | + | |
| 623 | +How expressive is the MLP? The classical answer, due independently to | |
| 624 | +Cybenko (1989) and to Hornik, Stinchcombe and White (1989), is that | |
| 625 | +\emph{one} hidden layer already suffices, provided it is wide enough. | |
| 626 | + | |
| 627 | +\begin{property}[Universal approximation]\label{prop:found-uat} | |
| 628 | +Let $\varphi$ be a continuous sigmoidal function | |
| 629 | +($\varphi(t) \to 1$ as $t \to +\infty$ and $\varphi(t) \to 0$ as | |
| 630 | +$t \to -\infty$). Then finite sums of the form | |
| 631 | +\begin{equation} | |
| 632 | + G(\vect{x}) \;=\; \sum_{j=1}^{N} \alpha_j\, | |
| 633 | + \varphi\!\left(\vect{w}_j\transp \vect{x} + \theta_j\right) | |
| 634 | + \label{eq:found-uat} | |
| 635 | +\end{equation} | |
| 636 | +are dense in $C([0,1]^{n})$ for the uniform norm: for every continuous | |
| 637 | +$f : [0,1]^n \to \R$ and every $\varepsilon > 0$, there exist $N$ and | |
| 638 | +parameters $\{\alpha_j, \vect{w}_j, \theta_j\}$ such that | |
| 639 | +$\lvert G(\vect{x}) - f(\vect{x})\rvert < \varepsilon$ for all | |
| 640 | +$\vect{x} \in [0,1]^n$. More generally (Leshno et al., 1993), a | |
| 641 | +one-hidden-layer network with a locally bounded, piecewise-continuous | |
| 642 | +activation is a universal approximator \emph{if and only if} the | |
| 643 | +activation is not a polynomial. | |
| 644 | +\end{property} | |
| 645 | + | |
| 646 | +The ``if and only if'' clause explains why ReLU — which is not sigmoidal — | |
| 647 | +is nevertheless universal, and why a purely linear network is not. Two | |
| 648 | +caveats temper the theorem's optimism, and both shape the rest of this | |
| 649 | +book~\cite{goodfellow2016book}. First, | |
| 650 | +Property~\ref{prop:found-uat} is an \emph{existence} result: it is silent | |
| 651 | +on how many hidden units are required (the width $N$ may grow | |
| 652 | +exponentially with the input dimension $n$) and on whether any learning | |
| 653 | +algorithm will \emph{find} the approximating weights. Second, it concerns | |
| 654 | +shallow networks only; depth-separation results show that certain | |
| 655 | +functions computable by a deep network with polynomially many units | |
| 656 | +require exponentially many units at bounded depth. Approximation theory | |
| 657 | +thus motivates depth, but it is the training machinery of the next | |
| 658 | +chapter — loss functions, backpropagation, and stochastic optimization — | |
| 659 | +that makes depth usable in practice. | |
| 660 | + | |
| 661 | +% ---------------------------------------------------------------------------- | |
| 662 | +\section{A Contrast: Radial Basis Function Networks} | |
| 663 | +% ---------------------------------------------------------------------------- | |
| 664 | + | |
| 665 | +The MLP is not the only way to combine simple units into a universal | |
| 666 | +approximator, and a brief look at its classical alternative sharpens our | |
| 667 | +understanding of what makes the MLP distinctive. A \emph{radial basis | |
| 668 | +function} (RBF) network, introduced by Broomhead and Lowe (1988) and | |
| 669 | +refined by Moody and Darken (1989), has exactly one hidden layer of $J$ | |
| 670 | +\emph{locally tuned} units, each defined by a centre | |
| 671 | +$\vect{\mu}_j \in \R^n$ and a width $\sigma_j > 0$, followed by a linear | |
| 672 | +output layer: | |
| 673 | +\begin{equation} | |
| 674 | + f(\vect{x}) \;=\; \sum_{j=1}^{J} w_j\, | |
| 675 | + \varphi_j(\vect{x}) + b, | |
| 676 | + \qquad | |
| 677 | + \varphi_j(\vect{x}) \;=\; | |
| 678 | + \exp\!\left( | |
| 679 | + -\frac{\lVert \vect{x} - \vect{\mu}_j \rVert^{2}}{2\sigma_j^{2}} | |
| 680 | + \right). | |
| 681 | + \label{eq:found-rbf} | |
| 682 | +\end{equation} | |
| 683 | +The contrast with Definition~\ref{def:found-neuron} is fundamental. An | |
| 684 | +MLP unit computes an \emph{inner product} $\vect{w}\transp\vect{x}$ and | |
| 685 | +responds along a hyperplane — a global, distributed representation. An | |
| 686 | +RBF unit computes a \emph{distance} $\lVert\vect{x} - \vect{\mu}_j\rVert$ | |
| 687 | +and responds only in a localized neighbourhood of its centre — a local, | |
| 688 | +spherical receptive field. Locality makes RBF networks fast to train: | |
| 689 | +once the centres are placed (by random subsampling or $k$-means | |
| 690 | +clustering), the model is \emph{linear} in the output weights, and with | |
| 691 | +the design matrix $\Phi_{ij} = \varphi_j(\vect{x}^{(i)})$ and targets | |
| 692 | +$\vect{y}$, the ridge-regularized least-squares solution is closed-form, | |
| 693 | +\begin{equation} | |
| 694 | + \vect{w} \;=\; | |
| 695 | + \left(\mat{\Phi}\transp \mat{\Phi} + \lambda \mat{I}\right)^{-1} | |
| 696 | + \mat{\Phi}\transp \vect{y}. | |
| 697 | + \label{eq:found-rbf-ls} | |
| 698 | +\end{equation} | |
| 699 | +The price of locality is the curse of dimensionality: covering a | |
| 700 | +high-dimensional input space with local bumps requires exponentially many | |
| 701 | +centres, whereas the global half-space features of the MLP can be shared | |
| 702 | +and composed. This trade-off — local interpolation versus global, | |
| 703 | +composable features — anticipates a pattern that recurs throughout the | |
| 704 | +book, and it is the composable option that deep learning has embraced. | |
| 705 | +Everything now hinges on one question: how are the weights of a | |
| 706 | +multilayer network actually learned? The next chapter answers it. | |
added
chapters/02-training.tex
+554 −0
@@ -0,0 +1,554 @@ | ||
| 1 | +% ============================================================================ | |
| 2 | +% Artificial Neural Networks — Methods, Equations and Graphical | |
| 3 | +% Representations | |
| 4 | +% Author : Simon-Pierre Boucher — contact@spboucher.ai | |
| 5 | +% Chapter 2 : Training Neural Networks (chapters/02-training.tex) | |
| 6 | +% ============================================================================ | |
| 7 | +\chapter{Training Neural Networks}\label{chap:training} | |
| 8 | + | |
| 9 | +The previous chapter established what a feed-forward network computes; this | |
| 10 | +chapter establishes how its parameters are found. Training is cast as the | |
| 11 | +minimization of an empirical risk, and virtually every modern network is | |
| 12 | +trained by the same recipe: compute the gradient of the loss with respect to | |
| 13 | +every parameter by \emph{backpropagation}~\cite{rumelhart1986}, then descend | |
| 14 | +along that gradient with a first-order update rule. We derive the | |
| 15 | +backpropagation equations in full, present the algorithm in pseudocode, and | |
| 16 | +then survey the family of optimizers — from plain stochastic gradient | |
| 17 | +descent to Adam~\cite{kingma2015adam} — together with the learning-rate | |
| 18 | +schedules and weight-initialization schemes that make deep training stable | |
| 19 | +in practice. | |
| 20 | + | |
| 21 | +% ============================================================================ | |
| 22 | +\section{The Learning Problem} | |
| 23 | + | |
| 24 | +\begin{definition}[Empirical risk minimization]\label{def:train-erm} | |
| 25 | +Let $f(\cdot\,;\vect{\theta})$ be a network with parameters $\vect{\theta}$, | |
| 26 | +let $\Loss(\hat{\vect{y}},\vect{y})$ be a per-example loss, and let | |
| 27 | +$\mathcal{D}$ be the data distribution. The \emph{(true) risk} and its | |
| 28 | +Monte-Carlo estimate on a training set | |
| 29 | +$\{(\vect{x}_i,\vect{y}_i)\}_{i=1}^{n}$, the \emph{empirical risk}, are | |
| 30 | +\begin{equation}\label{eq:train-risk} | |
| 31 | + R(\vect{\theta}) | |
| 32 | + = \E_{(\vect{x},\vect{y})\sim\mathcal{D}} | |
| 33 | + \bigl[\Loss\bigl(f(\vect{x};\vect{\theta}),\vect{y}\bigr)\bigr], | |
| 34 | + \qquad | |
| 35 | + \widehat{R}(\vect{\theta}) | |
| 36 | + = \frac{1}{n}\sum_{i=1}^{n} | |
| 37 | + \Loss\bigl(f(\vect{x}_i;\vect{\theta}),\vect{y}_i\bigr). | |
| 38 | +\end{equation} | |
| 39 | +Training solves | |
| 40 | +$\vect{\theta}^{\star} \in \argmin_{\vect{\theta}} \widehat{R}(\vect{\theta})$. | |
| 41 | +\end{definition} | |
| 42 | + | |
| 43 | +The choice of $\Loss$ encodes the task. We review the two workhorses — | |
| 44 | +squared error for regression, cross-entropy for classification — and the | |
| 45 | +single most useful gradient identity in deep learning. | |
| 46 | + | |
| 47 | +\subsection{Regression: mean squared error} | |
| 48 | + | |
| 49 | +For scalar targets $y_i \in \R$ and predictions $\hat{y}_i$, the | |
| 50 | +\emph{mean squared error} and its gradient are | |
| 51 | +\begin{equation}\label{eq:train-mse} | |
| 52 | + \Loss_{\mathrm{MSE}} | |
| 53 | + = \frac{1}{n}\sum_{i=1}^{n}\bigl(y_i - \hat{y}_i\bigr)^{2}, | |
| 54 | + \qquad | |
| 55 | + \frac{\partial \Loss_{\mathrm{MSE}}}{\partial \hat{y}_i} | |
| 56 | + = -\frac{2}{n}\,\bigl(y_i - \hat{y}_i\bigr). | |
| 57 | +\end{equation} | |
| 58 | +Minimizing~\eqref{eq:train-mse} is maximum-likelihood estimation under | |
| 59 | +Gaussian observation noise; the quadratic growth makes it sensitive to | |
| 60 | +outliers, which motivates robust alternatives such as the mean absolute | |
| 61 | +error $\frac{1}{n}\sum_i |y_i-\hat y_i|$ (Laplace likelihood, whose | |
| 62 | +minimizer is the conditional median) and the Huber loss, quadratic for | |
| 63 | +small residuals and linear in the tails. | |
| 64 | + | |
| 65 | +\subsection{Classification: cross-entropy} | |
| 66 | + | |
| 67 | +For binary labels $y\in\{0,1\}$ with $\hat{y}=\sigma(z)$ produced by a | |
| 68 | +sigmoid over the logit $z$, the \emph{binary cross-entropy} is | |
| 69 | +\begin{equation}\label{eq:train-bce} | |
| 70 | + \Loss_{\mathrm{BCE}} | |
| 71 | + = -\frac{1}{n}\sum_{i=1}^{n} | |
| 72 | + \Bigl[\,y_i\ln\hat{y}_i + (1-y_i)\ln\bigl(1-\hat{y}_i\bigr)\Bigr], | |
| 73 | + \qquad | |
| 74 | + \frac{\partial \Loss_{\mathrm{BCE}}}{\partial z} = \hat{y}-y . | |
| 75 | +\end{equation} | |
| 76 | +For $K$-way classification with one-hot target $\vect{y}$ and | |
| 77 | +$\hat{\vect{y}}=\softmax(\vect{z})$, the \emph{categorical cross-entropy} | |
| 78 | +is | |
| 79 | +\begin{equation}\label{eq:train-ce} | |
| 80 | + \Loss_{\mathrm{CE}} | |
| 81 | + = -\sum_{k=1}^{K} y_k \ln \hat{y}_k , | |
| 82 | +\end{equation} | |
| 83 | +which equals, up to the (constant) entropy of $\vect{y}$, the | |
| 84 | +Kullback--Leibler divergence $\KL(\vect{y}\,\|\,\hat{\vect{y}})$. | |
| 85 | + | |
| 86 | +\begin{property}[Softmax--cross-entropy gradient]\label{prop:train-ce-grad} | |
| 87 | +With $\hat{\vect{y}}=\softmax(\vect{z})$ and $\Loss_{\mathrm{CE}}$ as | |
| 88 | +in~\eqref{eq:train-ce}, | |
| 89 | +\begin{equation}\label{eq:train-ce-grad} | |
| 90 | + \frac{\partial \Loss_{\mathrm{CE}}}{\partial \vect{z}} | |
| 91 | + \;=\; \hat{\vect{y}} - \vect{y}. | |
| 92 | +\end{equation} | |
| 93 | +\end{property} | |
| 94 | + | |
| 95 | +\begin{proof}[Sketch] | |
| 96 | +The softmax Jacobian is | |
| 97 | +$\partial\hat{y}_i/\partial z_j = \hat{y}_i(\delta_{ij}-\hat{y}_j)$ with | |
| 98 | +$\delta_{ij}$ the Kronecker delta. Chaining it against | |
| 99 | +$\partial\Loss/\partial\hat{y}_i = -y_i/\hat{y}_i$ gives | |
| 100 | +$\partial\Loss/\partial z_j | |
| 101 | + = \sum_i (-y_i/\hat{y}_i)\,\hat{y}_i(\delta_{ij}-\hat{y}_j) | |
| 102 | + = -y_j + \hat{y}_j\sum_i y_i = \hat{y}_j - y_j$, | |
| 103 | +since $\sum_i y_i = 1$. | |
| 104 | +\end{proof} | |
| 105 | + | |
| 106 | +\begin{remark} | |
| 107 | +The cancellation in~\eqref{eq:train-ce-grad} is not a coincidence: it holds | |
| 108 | +for every matched pair of a canonical link and its exponential-family | |
| 109 | +negative log-likelihood (sigmoid with binary cross-entropy | |
| 110 | +in~\eqref{eq:train-bce}, identity with MSE). The practical consequence is | |
| 111 | +the absence of saturating $\sigma'$ factors at the output layer: a | |
| 112 | +confidently wrong prediction still receives a large gradient. | |
| 113 | +\end{remark} | |
| 114 | + | |
| 115 | +% ============================================================================ | |
| 116 | +\section{Backpropagation} | |
| 117 | + | |
| 118 | +Consider the $L$-layer network of Chapter~\ref{chap:foundations}, | |
| 119 | +\begin{equation}\label{eq:train-forward} | |
| 120 | + \vect{z}^{(\ell)} = \mat{W}^{(\ell)}\vect{a}^{(\ell-1)}+\vect{b}^{(\ell)}, | |
| 121 | + \qquad | |
| 122 | + \vect{a}^{(\ell)} = \varphi\bigl(\vect{z}^{(\ell)}\bigr), | |
| 123 | + \qquad \ell = 1,\dots,L, | |
| 124 | +\end{equation} | |
| 125 | +with $\vect{a}^{(0)}=\vect{x}$ and a scalar loss | |
| 126 | +$\Loss\bigl(\vect{a}^{(L)},\vect{y}\bigr)$. Backpropagation is reverse-mode | |
| 127 | +automatic differentiation applied to this composition: it computes | |
| 128 | +$\partial\Loss/\partial\vect{\theta}$ for \emph{all} parameters at the cost | |
| 129 | +of a constant number of forward passes. The central object is the | |
| 130 | +\emph{error signal} of layer $\ell$, | |
| 131 | +\begin{equation}\label{eq:train-delta-def} | |
| 132 | + \vect{\delta}^{(\ell)} | |
| 133 | + \;\equiv\; \frac{\partial \Loss}{\partial \vect{z}^{(\ell)}} | |
| 134 | + \;\in\; \R^{n_\ell}. | |
| 135 | +\end{equation} | |
| 136 | + | |
| 137 | +\begin{theorem}[Backpropagation equations~\cite{rumelhart1986}] | |
| 138 | +\label{thm:train-backprop} | |
| 139 | +For the network~\eqref{eq:train-forward}, the error signals and parameter | |
| 140 | +gradients satisfy | |
| 141 | +\begin{align} | |
| 142 | + \vect{\delta}^{(L)} | |
| 143 | + &= \nabla_{\vect{a}^{(L)}}\Loss | |
| 144 | + \,\odot\, \varphi'\bigl(\vect{z}^{(L)}\bigr), | |
| 145 | + \label{eq:train-bp-out}\\[2pt] | |
| 146 | + \vect{\delta}^{(\ell)} | |
| 147 | + &= \Bigl(\mat{W}^{(\ell+1)\transp}\,\vect{\delta}^{(\ell+1)}\Bigr) | |
| 148 | + \odot \varphi'\bigl(\vect{z}^{(\ell)}\bigr), | |
| 149 | + \quad \ell = L-1,\dots,1, | |
| 150 | + \label{eq:train-bp-rec}\\[2pt] | |
| 151 | + \frac{\partial \Loss}{\partial \mat{W}^{(\ell)}} | |
| 152 | + &= \vect{\delta}^{(\ell)}\,\vect{a}^{(\ell-1)\transp}, | |
| 153 | + \qquad | |
| 154 | + \frac{\partial \Loss}{\partial \vect{b}^{(\ell)}} | |
| 155 | + = \vect{\delta}^{(\ell)} . | |
| 156 | + \label{eq:train-bp-w} | |
| 157 | +\end{align} | |
| 158 | +\end{theorem} | |
| 159 | + | |
| 160 | +\begin{proof}[Sketch] | |
| 161 | +Equation~\eqref{eq:train-bp-out} is the chain rule through | |
| 162 | +$\vect{a}^{(L)}=\varphi(\vect{z}^{(L)})$. For~\eqref{eq:train-bp-rec}, | |
| 163 | +each $z^{(\ell+1)}_k$ depends on $z^{(\ell)}_j$ through | |
| 164 | +$W^{(\ell+1)}_{kj}\varphi'(z^{(\ell)}_j)$, so summing over downstream | |
| 165 | +paths, | |
| 166 | +\[ | |
| 167 | +\delta^{(\ell)}_j | |
| 168 | + = \sum_k \delta^{(\ell+1)}_k W^{(\ell+1)}_{kj}\, | |
| 169 | + \varphi'\bigl(z^{(\ell)}_j\bigr), | |
| 170 | +\] | |
| 171 | +which is~\eqref{eq:train-bp-rec} in matrix form. | |
| 172 | +For~\eqref{eq:train-bp-w}, note | |
| 173 | +$\partial z^{(\ell)}_j / \partial W^{(\ell)}_{ji} = a^{(\ell-1)}_i$ and | |
| 174 | +$\partial z^{(\ell)}_j / \partial b^{(\ell)}_j = 1$. | |
| 175 | +\end{proof} | |
| 176 | + | |
| 177 | +The name of the algorithm is visible in~\eqref{eq:train-bp-rec}: the error | |
| 178 | +is propagated \emph{backwards} through the transposes of the forward weight | |
| 179 | +matrices. With softmax output and cross-entropy loss, | |
| 180 | +Property~\ref{prop:train-ce-grad} replaces~\eqref{eq:train-bp-out} directly | |
| 181 | +by $\vect{\delta}^{(L)} = \hat{\vect{y}}-\vect{y}$. | |
| 182 | +Figure~\ref{fig:train-compgraph} shows the flow of both passes on the | |
| 183 | +computational graph, and Algorithm~\ref{alg:train-backprop} states the full | |
| 184 | +procedure for one mini-batch. | |
| 185 | + | |
| 186 | +\begin{figure}[htbp] | |
| 187 | + \centering | |
| 188 | + \begin{tikzpicture} | |
| 189 | + % ---- forward row ------------------------------------------------- | |
| 190 | + \node[blocinput, minimum width=1.15cm] (x) at (0,0) {$\vect{x}$}; | |
| 191 | + \node[blochidden, minimum width=1.15cm] (z1) at (2.35,0) {$\vect{z}^{(1)}$}; | |
| 192 | + \node[blochidden, minimum width=1.15cm] (a1) at (4.70,0) {$\vect{a}^{(1)}$}; | |
| 193 | + \node[blochidden, minimum width=1.15cm] (z2) at (7.05,0) {$\vect{z}^{(2)}$}; | |
| 194 | + \node[blocoutput, minimum width=1.15cm] (a2) at (9.40,0) {$\hat{\vect{y}}$}; | |
| 195 | + \node[bloc, minimum width=1.15cm] (Ls) at (11.75,0){$\Loss$}; | |
| 196 | + % ---- parameter and target nodes ------------------------------------ | |
| 197 | + \node[mem, minimum width=1.5cm] (W1) at (2.35,1.9) | |
| 198 | + {$\mat{W}^{(1)},\vect{b}^{(1)}$}; | |
| 199 | + \node[mem, minimum width=1.5cm] (W2) at (7.05,1.9) | |
| 200 | + {$\mat{W}^{(2)},\vect{b}^{(2)}$}; | |
| 201 | + \node[blocinput, minimum width=1.15cm] (y) at (11.75,1.9) {$\vect{y}$}; | |
| 202 | + % ---- forward arrows ------------------------------------------------ | |
| 203 | + \draw[fleche] (x) -- (z1); | |
| 204 | + \draw[fleche] (z1) -- node[above, etiquette] {$\varphi$} (a1); | |
| 205 | + \draw[fleche] (a1) -- (z2); | |
| 206 | + \draw[fleche] (z2) -- node[above, etiquette] {$\varphi$} (a2); | |
| 207 | + \draw[fleche] (a2) -- (Ls); | |
| 208 | + \draw[fleche] (W1) -- (z1); | |
| 209 | + \draw[fleche] (W2) -- (z2); | |
| 210 | + \draw[fleche] (y) -- (Ls); | |
| 211 | + % ---- backward (dashed) arrows, routed below ------------------------ | |
| 212 | + \draw[flechep, coutput] (Ls.south) to[bend left=35] | |
| 213 | + node[below, etiquette] {$\nabla_{\hat{\vect{y}}}\Loss$} (a2.south); | |
| 214 | + \draw[flechep, coutput] (a2.south) to[bend left=35] | |
| 215 | + node[below, etiquette] {$\vect{\delta}^{(2)}$} (z2.south); | |
| 216 | + \draw[flechep, coutput] (z2.south) to[bend left=35] | |
| 217 | + node[below, etiquette] {$\mat{W}^{(2)\transp}\vect{\delta}^{(2)}$} | |
| 218 | + (a1.south); | |
| 219 | + \draw[flechep, coutput] (a1.south) to[bend left=35] | |
| 220 | + node[below, etiquette] {$\vect{\delta}^{(1)}$} (z1.south); | |
| 221 | + % ---- gradients to parameters --------------------------------------- | |
| 222 | + \draw[flechep, coutput] (z1.north east) to[bend right=30] | |
| 223 | + node[right, etiquette, xshift=2pt] | |
| 224 | + {$\vect{\delta}^{(1)}\vect{a}^{(0)\transp}$} (W1.east); | |
| 225 | + \draw[flechep, coutput] (z2.north east) to[bend right=30] | |
| 226 | + node[right, etiquette, xshift=2pt] | |
| 227 | + {$\vect{\delta}^{(2)}\vect{a}^{(1)\transp}$} (W2.east); | |
| 228 | + \end{tikzpicture} | |
| 229 | + \caption{Computational graph of a two-layer network. Solid arrows: | |
| 230 | + forward pass~\eqref{eq:train-forward}. Dashed red arrows: backward | |
| 231 | + pass — the error signals $\vect{\delta}^{(\ell)}$ of | |
| 232 | + Theorem~\ref{thm:train-backprop} flow from the loss back through the | |
| 233 | + layers, branching off to the parameter | |
| 234 | + gradients~\eqref{eq:train-bp-w}.} | |
| 235 | + \label{fig:train-compgraph} | |
| 236 | +\end{figure} | |
| 237 | + | |
| 238 | +\begin{algorithm}[htbp] | |
| 239 | + \caption{Backpropagation with mini-batch gradient descent} | |
| 240 | + \label{alg:train-backprop} | |
| 241 | + \begin{algorithmic}[1] | |
| 242 | + \Require mini-batch $\{(\vect{x}_i,\vect{y}_i)\}_{i=1}^{m}$, parameters | |
| 243 | + $\{\mat{W}^{(\ell)},\vect{b}^{(\ell)}\}_{\ell=1}^{L}$, learning rate | |
| 244 | + $\eta$ | |
| 245 | + \For{$i = 1$ \textbf{to} $m$} \Comment{forward pass} | |
| 246 | + \State $\vect{a}^{(0)} \gets \vect{x}_i$ | |
| 247 | + \For{$\ell = 1$ \textbf{to} $L$} | |
| 248 | + \State $\vect{z}^{(\ell)} \gets | |
| 249 | + \mat{W}^{(\ell)}\vect{a}^{(\ell-1)} + \vect{b}^{(\ell)}$; | |
| 250 | + \quad $\vect{a}^{(\ell)} \gets \varphi(\vect{z}^{(\ell)})$ | |
| 251 | + \EndFor | |
| 252 | + \State $\vect{\delta}^{(L)} \gets | |
| 253 | + \nabla_{\vect{a}^{(L)}}\Loss \odot \varphi'(\vect{z}^{(L)})$ | |
| 254 | + \Comment{backward pass; $\hat{\vect{y}}-\vect{y}$ for softmax+CE} | |
| 255 | + \For{$\ell = L-1$ \textbf{down to} $1$} | |
| 256 | + \State $\vect{\delta}^{(\ell)} \gets | |
| 257 | + \bigl(\mat{W}^{(\ell+1)\transp}\vect{\delta}^{(\ell+1)}\bigr) | |
| 258 | + \odot \varphi'(\vect{z}^{(\ell)})$ | |
| 259 | + \EndFor | |
| 260 | + \State accumulate | |
| 261 | + $\Delta\mat{W}^{(\ell)} \mathrel{+}= | |
| 262 | + \vect{\delta}^{(\ell)}\vect{a}^{(\ell-1)\transp}$, | |
| 263 | + \; $\Delta\vect{b}^{(\ell)} \mathrel{+}= \vect{\delta}^{(\ell)}$ | |
| 264 | + \textbf{for all} $\ell$ | |
| 265 | + \EndFor | |
| 266 | + \For{$\ell = 1$ \textbf{to} $L$} \Comment{gradient step} | |
| 267 | + \State $\mat{W}^{(\ell)} \gets | |
| 268 | + \mat{W}^{(\ell)} - \dfrac{\eta}{m}\,\Delta\mat{W}^{(\ell)}$; | |
| 269 | + \quad | |
| 270 | + $\vect{b}^{(\ell)} \gets | |
| 271 | + \vect{b}^{(\ell)} - \dfrac{\eta}{m}\,\Delta\vect{b}^{(\ell)}$ | |
| 272 | + \EndFor | |
| 273 | + \end{algorithmic} | |
| 274 | +\end{algorithm} | |
| 275 | + | |
| 276 | +\begin{remark}[Cost] | |
| 277 | +One backward pass costs the same order as one forward pass, | |
| 278 | +$O\bigl(\sum_\ell n_\ell n_{\ell-1}\bigr)$: the full gradient of a scalar | |
| 279 | +with respect to $P$ parameters is obtained for $O(1)$ — not $O(P)$ — | |
| 280 | +forward-pass equivalents. This efficiency of reverse-mode differentiation | |
| 281 | +is what makes deep learning computationally feasible. | |
| 282 | +\end{remark} | |
| 283 | + | |
| 284 | +\begin{remark}[Vanishing and exploding gradients]\label{rem:train-vanish} | |
| 285 | +The recursion~\eqref{eq:train-bp-rec} multiplies a $\varphi'$ factor and a | |
| 286 | +weight matrix at every layer. With sigmoid activations, | |
| 287 | +$\sigma'(z)\le 1/4$, so error signals shrink at least geometrically with | |
| 288 | +depth; with large weights they can instead grow without bound. This | |
| 289 | +vanishing/exploding behaviour motivates ReLU-family activations, careful | |
| 290 | +initialization (Section~\ref{sec:train-init}), normalization layers and | |
| 291 | +residual connections, treated in later chapters. | |
| 292 | +\end{remark} | |
| 293 | + | |
| 294 | +% ============================================================================ | |
| 295 | +\section{First-Order Optimizers} | |
| 296 | + | |
| 297 | +Throughout this section $\vect{\theta}_t$ denotes the parameters at step | |
| 298 | +$t$, $\vect{g}_t = \nabla_{\vect{\theta}}\Loss(\vect{\theta}_t)$ the | |
| 299 | +mini-batch gradient, $\eta$ the learning rate, and all operations on | |
| 300 | +vectors are elementwise. | |
| 301 | + | |
| 302 | +\subsection{Stochastic gradient descent and momentum} | |
| 303 | + | |
| 304 | +\emph{Stochastic gradient descent} (SGD) applies the elementary update | |
| 305 | +\begin{equation}\label{eq:train-sgd} | |
| 306 | + \vect{\theta}_{t+1} = \vect{\theta}_t - \eta\,\vect{g}_t . | |
| 307 | +\end{equation} | |
| 308 | +Classical stochastic-approximation theory guarantees convergence when the | |
| 309 | +step sizes satisfy $\sum_t \eta_t = \infty$ and $\sum_t \eta_t^2 < \infty$ | |
| 310 | +\cite{goodfellow2016book}. | |
| 311 | + | |
| 312 | +\emph{Momentum} (Polyak's heavy ball) accumulates an exponentially weighted | |
| 313 | +velocity, | |
| 314 | +\begin{equation}\label{eq:train-momentum} | |
| 315 | + \vect{v}_t = \beta\,\vect{v}_{t-1} + \vect{g}_t , | |
| 316 | + \qquad | |
| 317 | + \vect{\theta}_{t+1} = \vect{\theta}_t - \eta\,\vect{v}_t , | |
| 318 | +\end{equation} | |
| 319 | +with $\beta \approx 0.9$. Directions in which successive gradients agree | |
| 320 | +are amplified by up to $1/(1-\beta)$, while oscillating components cancel — | |
| 321 | +precisely the geometry of Figure~\ref{fig:train-trajectories}. | |
| 322 | +\emph{Nesterov's accelerated gradient} evaluates the gradient at a | |
| 323 | +look-ahead point, | |
| 324 | +\begin{equation}\label{eq:train-nesterov} | |
| 325 | + \vect{v}_t = \beta\,\vect{v}_{t-1} | |
| 326 | + + \nabla_{\vect{\theta}} | |
| 327 | + \Loss\bigl(\vect{\theta}_t - \eta\beta\,\vect{v}_{t-1}\bigr), | |
| 328 | + \qquad | |
| 329 | + \vect{\theta}_{t+1} = \vect{\theta}_t - \eta\,\vect{v}_t , | |
| 330 | +\end{equation} | |
| 331 | +letting the update ``see'' where it is heading and correct in advance; for | |
| 332 | +smooth convex objectives it attains the optimal $O(1/t^2)$ convergence | |
| 333 | +rate. | |
| 334 | + | |
| 335 | +\begin{figure}[htbp] | |
| 336 | + \centering | |
| 337 | + \begin{tikzpicture} | |
| 338 | + \begin{axis}[ | |
| 339 | + width=0.88\textwidth, height=6.2cm, | |
| 340 | + xmin=-10, xmax=1.5, ymin=-2.4, ymax=2.4, | |
| 341 | + xlabel={$\theta_1$}, ylabel={$\theta_2$}, | |
| 342 | + xlabel near ticks, ylabel near ticks, | |
| 343 | + tick label style={font=\scriptsize}, | |
| 344 | + label style={font=\small}, | |
| 345 | + legend style={font=\scriptsize, at={(0.98,0.04)}, | |
| 346 | + anchor=south east, draw=black!30}, | |
| 347 | + legend cell align=left, | |
| 348 | + ] | |
| 349 | + % loss contours of f = x^2/20 + y^2 (anisotropic bowl) | |
| 350 | + \addplot[domain=0:360, samples=91, smooth, black!25, forget plot] | |
| 351 | + ({2.828*cos(x)}, {0.632*sin(x)}); | |
| 352 | + \addplot[domain=0:360, samples=91, smooth, black!25, forget plot] | |
| 353 | + ({4.899*cos(x)}, {1.095*sin(x)}); | |
| 354 | + \addplot[domain=0:360, samples=91, smooth, black!25, forget plot] | |
| 355 | + ({6.928*cos(x)}, {1.549*sin(x)}); | |
| 356 | + \addplot[domain=0:360, samples=91, smooth, black!25, forget plot] | |
| 357 | + ({8.944*cos(x)}, {2.000*sin(x)}); | |
| 358 | + \addplot[domain=0:360, samples=91, smooth, black!25, forget plot] | |
| 359 | + ({11.662*cos(x)}, {2.608*sin(x)}); | |
| 360 | + % SGD trajectory (zigzag) | |
| 361 | + \addplot[coutput, thick, mark=*, mark size=1.1pt] coordinates { | |
| 362 | + (-8.500,1.800) (-7.777,-1.260) (-7.116,0.882) (-6.512,-0.617) | |
| 363 | + (-5.958,0.432) (-5.452,-0.303) (-4.988,0.212) (-4.564,-0.148) | |
| 364 | + (-4.176,0.104) (-3.821,-0.073) (-3.496,0.051) (-3.199,-0.036) | |
| 365 | + (-2.927,0.025) (-2.679,-0.017) (-2.451,0.012) (-2.243,-0.009) | |
| 366 | + (-2.052,0.006) | |
| 367 | + }; | |
| 368 | + \addlegendentry{SGD~\eqref{eq:train-sgd}} | |
| 369 | + % Momentum trajectory (smooth) | |
| 370 | + \addplot[cinput, thick, mark=*, mark size=1.1pt] coordinates { | |
| 371 | + (-8.500,1.800) (-8.381,1.296) (-8.157,0.480) (-7.840,-0.390) | |
| 372 | + (-7.446,-1.063) (-6.987,-1.371) (-6.476,-1.264) (-5.925,-0.815) | |
| 373 | + (-5.347,-0.182) (-4.751,0.439) (-4.149,0.874) (-3.548,1.022) | |
| 374 | + (-2.959,0.868) (-2.386,0.487) (-1.838,0.007) (-1.318,-0.426) | |
| 375 | + (-0.832,-0.697) | |
| 376 | + }; | |
| 377 | + \addlegendentry{Momentum~\eqref{eq:train-momentum}} | |
| 378 | + % optimum | |
| 379 | + \addplot[only marks, mark=star, mark size=3.2pt, black] | |
| 380 | + coordinates {(0,0)}; | |
| 381 | + \addlegendentry{minimum $\vect{\theta}^{\star}$} | |
| 382 | + \end{axis} | |
| 383 | + \end{tikzpicture} | |
| 384 | + \caption{Sixteen steps of SGD and momentum on the anisotropic quadratic | |
| 385 | + $\Loss(\vect{\theta}) = \theta_1^2/20 + \theta_2^2$ (grey level sets), | |
| 386 | + both computed numerically from the same starting point. SGD (red) | |
| 387 | + oscillates across the narrow valley while creeping along the shallow | |
| 388 | + direction; momentum (blue) damps the oscillation and accelerates along | |
| 389 | + the valley floor.} | |
| 390 | + \label{fig:train-trajectories} | |
| 391 | +\end{figure} | |
| 392 | + | |
| 393 | +\subsection{Adaptive methods: AdaGrad, RMSProp, Adam} | |
| 394 | + | |
| 395 | +Adaptive methods give every coordinate its own effective learning rate, | |
| 396 | +scaled by the history of gradient magnitudes. \emph{AdaGrad} accumulates | |
| 397 | +the squared gradients, | |
| 398 | +\begin{equation}\label{eq:train-adagrad} | |
| 399 | + \vect{G}_t = \vect{G}_{t-1} + \vect{g}_t^{2}, | |
| 400 | + \qquad | |
| 401 | + \vect{\theta}_{t+1} | |
| 402 | + = \vect{\theta}_t | |
| 403 | + - \frac{\eta}{\sqrt{\vect{G}_t}+\epsilon}\odot\vect{g}_t , | |
| 404 | +\end{equation} | |
| 405 | +so rarely active (sparse) coordinates receive large steps. Because | |
| 406 | +$\vect{G}_t$ grows monotonically, however, the effective step size decays | |
| 407 | +to zero. \emph{RMSProp} repairs this by replacing the sum with an | |
| 408 | +exponential moving average, | |
| 409 | +\begin{equation}\label{eq:train-rmsprop} | |
| 410 | + \E[\vect{g}^2]_t | |
| 411 | + = \rho\,\E[\vect{g}^2]_{t-1} + (1-\rho)\,\vect{g}_t^{2}, | |
| 412 | + \qquad | |
| 413 | + \vect{\theta}_{t+1} | |
| 414 | + = \vect{\theta}_t | |
| 415 | + - \frac{\eta}{\sqrt{\E[\vect{g}^2]_t}+\epsilon}\odot\vect{g}_t , | |
| 416 | +\end{equation} | |
| 417 | +with $\rho\approx 0.9$, so that stale gradients are forgotten. | |
| 418 | + | |
| 419 | +\emph{Adam}~\cite{kingma2015adam} (\emph{adaptive moment estimation}) | |
| 420 | +combines the momentum idea~\eqref{eq:train-momentum} — an exponential | |
| 421 | +moving average of the gradient, the first moment — with the RMSProp | |
| 422 | +idea~\eqref{eq:train-rmsprop} — an exponential moving average of its | |
| 423 | +square, the second moment — and corrects the initialization bias of both: | |
| 424 | +\begin{align} | |
| 425 | + \vect{m}_t &= \beta_1\,\vect{m}_{t-1} + (1-\beta_1)\,\vect{g}_t , | |
| 426 | + \qquad | |
| 427 | + \vect{v}_t = \beta_2\,\vect{v}_{t-1} + (1-\beta_2)\,\vect{g}_t^{2}, | |
| 428 | + \label{eq:train-adam-moments}\\[2pt] | |
| 429 | + \hat{\vect{m}}_t &= \frac{\vect{m}_t}{1-\beta_1^{\,t}} , | |
| 430 | + \qquad | |
| 431 | + \hat{\vect{v}}_t = \frac{\vect{v}_t}{1-\beta_2^{\,t}} , | |
| 432 | + \label{eq:train-adam-bias}\\[2pt] | |
| 433 | + \vect{\theta}_{t+1} | |
| 434 | + &= \vect{\theta}_t | |
| 435 | + - \eta\,\frac{\hat{\vect{m}}_t}{\sqrt{\hat{\vect{v}}_t}+\epsilon} . | |
| 436 | + \label{eq:train-adam-update} | |
| 437 | +\end{align} | |
| 438 | +The standard defaults are $\eta = 10^{-3}$, $\beta_1 = 0.9$, | |
| 439 | +$\beta_2 = 0.999$ and $\epsilon = 10^{-8}$. | |
| 440 | + | |
| 441 | +\begin{remark}[Why the bias correction matters] | |
| 442 | +Since $\vect{m}_0=\vect{v}_0=\vect{0}$, the raw averages | |
| 443 | +in~\eqref{eq:train-adam-moments} are biased toward zero for small $t$: | |
| 444 | +$\E[\vect{m}_t]\approx(1-\beta_1^{\,t})\,\E[\vect{g}_t]$. Dividing by | |
| 445 | +$(1-\beta_1^{\,t})$ and $(1-\beta_2^{\,t})$ | |
| 446 | +in~\eqref{eq:train-adam-bias} removes this bias exactly; without it the | |
| 447 | +first updates would be far too small — dramatically so for | |
| 448 | +$\vect{v}_t$, whose decay rate $\beta_2 = 0.999$ makes the bias persist | |
| 449 | +for roughly a thousand steps. The ratio | |
| 450 | +$\hat{\vect{m}}_t/\sqrt{\hat{\vect{v}}_t}$ acts as a per-coordinate | |
| 451 | +signal-to-noise estimate, and the magnitude of each update is bounded by | |
| 452 | +approximately $\eta$ regardless of the gradient scale, making | |
| 453 | +Adam invariant to gradient rescaling. The decoupled-weight-decay variant | |
| 454 | +AdamW, which applies the $L_2$ shrinkage outside the adaptive rescaling, | |
| 455 | +is the default optimizer for modern Transformer models. | |
| 456 | +\end{remark} | |
| 457 | + | |
| 458 | +% ============================================================================ | |
| 459 | +\section{Learning-Rate Schedules} | |
| 460 | + | |
| 461 | +The learning rate is the single most important hyperparameter of the | |
| 462 | +updates above, and it is rarely held constant. Writing $\eta_t$ for the | |
| 463 | +rate at step $t$ over a horizon of $T$ steps, the three standard schedules | |
| 464 | +are | |
| 465 | +\begin{align} | |
| 466 | + \eta_t &= \eta_0\,\gamma^{\lfloor t/s \rfloor}, | |
| 467 | + \qquad 0<\gamma<1 | |
| 468 | + &&\text{(step decay, factor $\gamma$ every $s$ steps)}, | |
| 469 | + \label{eq:train-step-decay}\\[2pt] | |
| 470 | + \eta_t &= \frac{\eta_0}{2} | |
| 471 | + \Bigl(1+\cos\frac{\pi t}{T}\Bigr) | |
| 472 | + &&\text{(cosine annealing)}, | |
| 473 | + \label{eq:train-cosine}\\[2pt] | |
| 474 | + \eta_t &= | |
| 475 | + \begin{cases} | |
| 476 | + \eta_0\, t/T_w & t \le T_w\\[2pt] | |
| 477 | + \dfrac{\eta_0}{2} | |
| 478 | + \Bigl(1+\cos\dfrac{\pi (t-T_w)}{T-T_w}\Bigr) & t > T_w | |
| 479 | + \end{cases} | |
| 480 | + &&\text{(linear warmup, then cosine)}. | |
| 481 | + \label{eq:train-warmup} | |
| 482 | +\end{align} | |
| 483 | +Warmup~\eqref{eq:train-warmup} protects the early phase of training — when | |
| 484 | +Adam's second-moment estimate $\hat{\vect{v}}_t$ is still noisy — from | |
| 485 | +destructively large steps, and is standard practice for Transformers; the | |
| 486 | +three profiles are compared in Figure~\ref{fig:train-schedules}. | |
| 487 | + | |
| 488 | +\begin{figure}[htbp] | |
| 489 | + \centering | |
| 490 | + \begin{tikzpicture} | |
| 491 | + \begin{axis}[ | |
| 492 | + width=0.8\textwidth, height=5.2cm, | |
| 493 | + xmin=0, xmax=100, ymin=0, ymax=0.115, | |
| 494 | + xlabel={training step $t$ (\% of horizon $T$)}, | |
| 495 | + ylabel={$\eta_t$}, | |
| 496 | + xlabel near ticks, ylabel near ticks, | |
| 497 | + tick label style={font=\scriptsize}, | |
| 498 | + label style={font=\small}, | |
| 499 | + legend style={font=\scriptsize, draw=black!30}, | |
| 500 | + legend cell align=left, | |
| 501 | + ] | |
| 502 | + \addplot[coutput, thick, const plot, domain=0:100, samples=201] | |
| 503 | + {0.1 * 0.5^(floor(x/30))}; | |
| 504 | + \addlegendentry{step decay~\eqref{eq:train-step-decay}} | |
| 505 | + \addplot[cinput, thick, domain=0:100, samples=201] | |
| 506 | + {0.05*(1+cos(deg(pi*x/100)))}; | |
| 507 | + \addlegendentry{cosine annealing~\eqref{eq:train-cosine}} | |
| 508 | + \addplot[cgate, thick, domain=0:10, samples=21, forget plot] | |
| 509 | + {0.1*x/10}; | |
| 510 | + \addplot[cgate, thick, domain=10:100, samples=181] | |
| 511 | + {0.05*(1+cos(deg(pi*(x-10)/90)))}; | |
| 512 | + \addlegendentry{warmup + cosine~\eqref{eq:train-warmup}} | |
| 513 | + \end{axis} | |
| 514 | + \end{tikzpicture} | |
| 515 | + \caption{Learning-rate schedules with $\eta_0=0.1$: step decay | |
| 516 | + ($\gamma=0.5$, $s=0.3\,T$), cosine annealing, and linear warmup over | |
| 517 | + the first $10\%$ of training followed by cosine annealing.} | |
| 518 | + \label{fig:train-schedules} | |
| 519 | +\end{figure} | |
| 520 | + | |
| 521 | +% ============================================================================ | |
| 522 | +\section{Weight Initialization}\label{sec:train-init} | |
| 523 | + | |
| 524 | +Remark~\ref{rem:train-vanish} showed that signals are multiplied by a | |
| 525 | +weight matrix at every layer; initialization must therefore keep the | |
| 526 | +variance of activations and of backpropagated gradients approximately | |
| 527 | +constant with depth~\cite{goodfellow2016book}. For a layer with | |
| 528 | +$n_{\mathrm{in}}$ inputs and $n_{\mathrm{out}}$ outputs, | |
| 529 | +\emph{Xavier/Glorot} initialization — appropriate for symmetric, roughly | |
| 530 | +linear-around-zero activations such as $\tanh$ — balances both passes: | |
| 531 | +\begin{equation}\label{eq:train-xavier} | |
| 532 | + \operatorname{Var}\bigl(W_{ij}\bigr) | |
| 533 | + = \frac{2}{n_{\mathrm{in}}+n_{\mathrm{out}}}, | |
| 534 | + \qquad\text{e.g.}\quad | |
| 535 | + W_{ij} \sim | |
| 536 | + \mathcal{U}\!\left[ | |
| 537 | + -\sqrt{\tfrac{6}{n_{\mathrm{in}}+n_{\mathrm{out}}}},\; | |
| 538 | + \sqrt{\tfrac{6}{n_{\mathrm{in}}+n_{\mathrm{out}}}} | |
| 539 | + \right]. | |
| 540 | +\end{equation} | |
| 541 | +\emph{He/Kaiming} initialization corrects for the fact that ReLU zeroes | |
| 542 | +half of its inputs, which halves the activation variance at each layer: | |
| 543 | +\begin{equation}\label{eq:train-he} | |
| 544 | + \operatorname{Var}\bigl(W_{ij}\bigr) = \frac{2}{n_{\mathrm{in}}}, | |
| 545 | + \qquad | |
| 546 | + W_{ij} \sim | |
| 547 | + \mathcal{N}\!\Bigl(0,\; \tfrac{2}{n_{\mathrm{in}}}\Bigr), | |
| 548 | +\end{equation} | |
| 549 | +and is the default for ReLU-family networks. Biases are initialized to | |
| 550 | +zero in both schemes. Together with the schedules of | |
| 551 | +Figure~\ref{fig:train-schedules} and an adaptive optimizer such as | |
| 552 | +Adam~\eqref{eq:train-adam-moments}--\eqref{eq:train-adam-update}, these | |
| 553 | +choices form the standard modern training recipe on which the | |
| 554 | +regularization techniques of the next chapter are layered. | |
added
chapters/03-regularization.tex
+407 −0
Large diff (407 lines) is not loaded by default to keep the page fast.Load diff
added
chapters/04-cnn.tex
+464 −0
Large diff (464 lines) is not loaded by default to keep the page fast.Load diff
added
chapters/05-rnn.tex
+501 −0
Large diff (501 lines) is not loaded by default to keep the page fast.Load diff
added
chapters/06-attention.tex
+590 −0
Large diff (590 lines) is not loaded by default to keep the page fast.Load diff
added
chapters/07-modern-transformers.tex
+678 −0
Large diff (678 lines) is not loaded by default to keep the page fast.Load diff
added
chapters/08-gnn.tex
+685 −0
Large diff (685 lines) is not loaded by default to keep the page fast.Load diff
added
chapters/09-energy.tex
+596 −0
Large diff (596 lines) is not loaded by default to keep the page fast.Load diff
added
chapters/10-generative.tex
+552 −0
Large diff (552 lines) is not loaded by default to keep the page fast.Load diff
added
chapters/11-advanced-generative.tex
+741 −0
Large diff (741 lines) is not loaded by default to keep the page fast.Load diff
added
chapters/12-bio-inspired.tex
+926 −0
Large diff (926 lines) is not loaded by default to keep the page fast.Load diff
added
chapters/13-emerging.tex
+731 −0
Large diff (731 lines) is not loaded by default to keep the page fast.Load diff
added
main.pdf
+0 −0
Binary file not shown.
added
main.tex
+511 −0
Large diff (512 lines) is not loaded by default to keep the page fast.Load diff