spb/artificial-neural-networks-book Public

1# Artificial Neural Networks — Methods, Equations and Graphical Representations23**A complete, self-contained book on neural networks — every method presented with rigorous equations, an estimation algorithm in pseudocode, and a native TikZ figure.**45*Simon-Pierre Boucher* · [contact@spboucher.ai](mailto:contact@spboucher.ai)67---89## 📖 About1011This repository contains the full LaTeX source and compiled PDF of a 119-page book covering **all major families of artificial neural networks**, from Rosenblatt's perceptron (1958) to Kolmogorov–Arnold networks (2024).1213The book is built on one strict organizing principle:1415> **Every architecture = rigorous equations + estimation/training algorithm + faithful graphical representation.**1617Every single figure is drawn natively in **TikZ/pgfplots** — no imported images — so each diagram is exactly as precise as the equations it illustrates. Notation is unified across all chapters (bold lowercase vectors, bold uppercase matrices, Hadamard product `⊙`, shared color palette for inputs/hidden/output/gates/memory).1819## 📊 At a glance2021| Metric | Count |22|---|---:|23| Pages | 119 |24| Chapters | 13 (in 5 parts) |25| Numbered equations | 256 |26| Native TikZ/pgfplots figures | 42 |27| Estimation algorithms (pseudocode) | 26 |28| Comparison tables | 7 |29| Bibliography entries (all cited) | 36 |3031## 🗂 Contents3233### Part I — Foundations and Learning341. **Foundations of Neural Networks** — artificial neuron, perceptron (Novikoff convergence theorem + learning algorithm), activation functions with derivatives, softmax, MLP, universal approximation theorem, XOR geometry, RBF contrast.352. **Training Neural Networks** — empirical risk, MSE/cross-entropy, the softmax+CE gradient, the three backpropagation equations (with computational-graph figure and full pseudocode), SGD/Momentum/Nesterov/AdaGrad/RMSProp/**Adam** with bias correction, learning-rate schedules, Xavier/He initialization.363. **Regularization** — bias–variance decomposition, L2/L1, inverted dropout, Batch & Layer Normalization, early stopping, label smoothing, gradient clipping.3738### Part II — Core Architectures394. **Convolutional Neural Networks** — convolution with stride/padding (worked 5×5 ⊛ 3×3 grid figure), pooling, canonical architecture, **ResNet** residual block and its gradient, receptive fields, conv forward-pass algorithm.405. **Recurrent Networks: RNN, LSTM and GRU** — folded/unfolded views, BPTT (Jacobian products, vanishing/exploding gradients, algorithm), the six **LSTM** equations with a detailed cell diagram, GRU, bidirectional RNNs, seq2seq.416. **Attention and Transformers** — scaled dot-product attention with the 1/√d_k variance argument, multi-head attention, sinusoidal positional encoding, Add&Norm/FFN/causal masking, full **encoder–decoder architecture figure**, greedy decoding algorithm, BERT/GPT/T5.427. **Modern Transformer Variants** — **Vision Transformer** (patches, [CLS], training algorithm), **Mixture of Experts** (noisy top-k routing, load balancing), efficient/linear attention, **state-space models and Mamba** (ZOH discretization, selective scan), scaling laws.4344### Part III — Graphs, Energy and Memory458. **Graph Neural Networks** — message-passing framework, **GCN** (spectral derivation), **GraphSAGE** (minibatch estimation algorithm), **GAT**, **GIN** with the 1-WL expressiveness theorem, task heads.469. **Energy-Based and Associative-Memory Networks** — **Hopfield** networks (energy descent theorem, capacity, recall algorithm), modern Hopfield ≈ attention, **Boltzmann machines**, **RBM** with **contrastive divergence (CD-k)**, deep belief networks with greedy layer-wise pretraining.4748### Part IV — Generative Models4910. **Autoencoders, VAEs and GANs** — hourglass autoencoder, **VAE** (ELBO with reconstruction/regularization decomposition, closed-form Gaussian KL, reparameterization trick, training algorithm), **GAN** (minimax game, optimal discriminator, Jensen–Shannon link, non-saturating loss, WGAN, alternating training algorithm), VAE/GAN/autoregressive comparison table.5011. **Normalizing Flows, Diffusion and Autoregressive Models** — change of variables, RealNVP coupling layers, **DDPM** (forward/reverse processes, variational bound, ε-parameterization, training *and* sampling algorithms), DDIM, classifier-free guidance, **WaveNet** dilated causal convolutions, energy-based models.5152### Part V — Specialized and Emerging Architectures5312. **Biologically Inspired Networks** — **spiking neurons** (LIF dynamics, f–I curve, STDP window, surrogate gradients), **self-organizing maps** (BMU, neighborhood kernel, training loop), **reservoir computing / echo state networks** (echo-state property, ridge readout).5413. **Emerging Architectures** — **capsule networks** (dynamic routing-by-agreement), **Neural ODEs** (adjoint method), **Neural Turing Machines** (full addressing pipeline), **Kolmogorov–Arnold networks** (superposition theorem, learnable edge functions).5556## 🔧 Building from source5758Requires a full TeX Live installation (pgf/pgfplots, `babel-english`, `algorithm`/`algpseudocode`, `booktabs`, `microtype`, `hyperref`).5960```bash61cd these  # repository root62pdflatex -interaction=nonstopmode main.tex63pdflatex -interaction=nonstopmode main.tex   # resolve TOC + cross-references64pdflatex -interaction=nonstopmode main.tex   # settle part/page numbers65```6667The build is clean by construction: **zero errors, zero undefined references, zero unresolved citations**.6869## 📁 Repository layout7071```72.73├── README.md74├── CLAUDE.md                 # executable build specification of the book75├── main.pdf                  # compiled book (119 pages)76├── main.tex                  # preamble, cover, preface, abstract, TOC, bibliography77└── chapters/78    ├── 01-foundations.tex79    ├── 02-training.tex80    ├── 03-regularization.tex81    ├── 04-cnn.tex82    ├── 05-rnn.tex83    ├── 06-attention.tex84    ├── 07-modern-transformers.tex85    ├── 08-gnn.tex86    ├── 09-energy.tex87    ├── 10-generative.tex88    ├── 11-advanced-generative.tex89    ├── 12-bio-inspired.tex90    └── 13-emerging.tex91```9293All global TikZ styles, the named color palette, notation macros and theorem environments are defined once in `main.tex` and reused by every chapter — guaranteeing visual and mathematical consistency across the whole book.9495## ✍️ Author9697**Simon-Pierre Boucher**98📧 [contact@spboucher.ai](mailto:contact@spboucher.ai)99100## 📜 License101102© 2026 Simon-Pierre Boucher. All rights reserved.103The source is published for reading and personal study; please contact the author for any other use.104