# Artificial Neural Networks — Methods, Equations and Graphical Representations

**A complete, self-contained book on neural networks — every method presented with rigorous equations, an estimation algorithm in pseudocode, and a native TikZ figure.**

*Simon-Pierre Boucher* · [contact@spboucher.ai](mailto:contact@spboucher.ai)

---

## 📖 About

This repository contains the full LaTeX source and compiled PDF of a 119-page book covering **all major families of artificial neural networks**, from Rosenblatt's perceptron (1958) to Kolmogorov–Arnold networks (2024).

The book is built on one strict organizing principle:

> **Every architecture = rigorous equations + estimation/training algorithm + faithful graphical representation.**

Every single figure is drawn natively in **TikZ/pgfplots** — no imported images — so each diagram is exactly as precise as the equations it illustrates. Notation is unified across all chapters (bold lowercase vectors, bold uppercase matrices, Hadamard product `⊙`, shared color palette for inputs/hidden/output/gates/memory).

## 📊 At a glance

| Metric | Count |
|---|---:|
| Pages | 119 |
| Chapters | 13 (in 5 parts) |
| Numbered equations | 256 |
| Native TikZ/pgfplots figures | 42 |
| Estimation algorithms (pseudocode) | 26 |
| Comparison tables | 7 |
| Bibliography entries (all cited) | 36 |

## 🗂 Contents

### Part I — Foundations and Learning
1. **Foundations of Neural Networks** — artificial neuron, perceptron (Novikoff convergence theorem + learning algorithm), activation functions with derivatives, softmax, MLP, universal approximation theorem, XOR geometry, RBF contrast.
2. **Training Neural Networks** — empirical risk, MSE/cross-entropy, the softmax+CE gradient, the three backpropagation equations (with computational-graph figure and full pseudocode), SGD/Momentum/Nesterov/AdaGrad/RMSProp/**Adam** with bias correction, learning-rate schedules, Xavier/He initialization.
3. **Regularization** — bias–variance decomposition, L2/L1, inverted dropout, Batch & Layer Normalization, early stopping, label smoothing, gradient clipping.

### Part II — Core Architectures
4. **Convolutional Neural Networks** — convolution with stride/padding (worked 5×5 ⊛ 3×3 grid figure), pooling, canonical architecture, **ResNet** residual block and its gradient, receptive fields, conv forward-pass algorithm.
5. **Recurrent Networks: RNN, LSTM and GRU** — folded/unfolded views, BPTT (Jacobian products, vanishing/exploding gradients, algorithm), the six **LSTM** equations with a detailed cell diagram, GRU, bidirectional RNNs, seq2seq.
6. **Attention and Transformers** — scaled dot-product attention with the 1/√d_k variance argument, multi-head attention, sinusoidal positional encoding, Add&Norm/FFN/causal masking, full **encoder–decoder architecture figure**, greedy decoding algorithm, BERT/GPT/T5.
7. **Modern Transformer Variants** — **Vision Transformer** (patches, [CLS], training algorithm), **Mixture of Experts** (noisy top-k routing, load balancing), efficient/linear attention, **state-space models and Mamba** (ZOH discretization, selective scan), scaling laws.

### Part III — Graphs, Energy and Memory
8. **Graph Neural Networks** — message-passing framework, **GCN** (spectral derivation), **GraphSAGE** (minibatch estimation algorithm), **GAT**, **GIN** with the 1-WL expressiveness theorem, task heads.
9. **Energy-Based and Associative-Memory Networks** — **Hopfield** networks (energy descent theorem, capacity, recall algorithm), modern Hopfield ≈ attention, **Boltzmann machines**, **RBM** with **contrastive divergence (CD-k)**, deep belief networks with greedy layer-wise pretraining.

### Part IV — Generative Models
10. **Autoencoders, VAEs and GANs** — hourglass autoencoder, **VAE** (ELBO with reconstruction/regularization decomposition, closed-form Gaussian KL, reparameterization trick, training algorithm), **GAN** (minimax game, optimal discriminator, Jensen–Shannon link, non-saturating loss, WGAN, alternating training algorithm), VAE/GAN/autoregressive comparison table.
11. **Normalizing Flows, Diffusion and Autoregressive Models** — change of variables, RealNVP coupling layers, **DDPM** (forward/reverse processes, variational bound, ε-parameterization, training *and* sampling algorithms), DDIM, classifier-free guidance, **WaveNet** dilated causal convolutions, energy-based models.

### Part V — Specialized and Emerging Architectures
12. **Biologically Inspired Networks** — **spiking neurons** (LIF dynamics, f–I curve, STDP window, surrogate gradients), **self-organizing maps** (BMU, neighborhood kernel, training loop), **reservoir computing / echo state networks** (echo-state property, ridge readout).
13. **Emerging Architectures** — **capsule networks** (dynamic routing-by-agreement), **Neural ODEs** (adjoint method), **Neural Turing Machines** (full addressing pipeline), **Kolmogorov–Arnold networks** (superposition theorem, learnable edge functions).

## 🔧 Building from source

Requires a full TeX Live installation (pgf/pgfplots, `babel-english`, `algorithm`/`algpseudocode`, `booktabs`, `microtype`, `hyperref`).

```bash
cd these  # repository root
pdflatex -interaction=nonstopmode main.tex
pdflatex -interaction=nonstopmode main.tex   # resolve TOC + cross-references
pdflatex -interaction=nonstopmode main.tex   # settle part/page numbers
```

The build is clean by construction: **zero errors, zero undefined references, zero unresolved citations**.

## 📁 Repository layout

```
.
├── README.md
├── CLAUDE.md                 # executable build specification of the book
├── main.pdf                  # compiled book (119 pages)
├── main.tex                  # preamble, cover, preface, abstract, TOC, bibliography
└── chapters/
    ├── 01-foundations.tex
    ├── 02-training.tex
    ├── 03-regularization.tex
    ├── 04-cnn.tex
    ├── 05-rnn.tex
    ├── 06-attention.tex
    ├── 07-modern-transformers.tex
    ├── 08-gnn.tex
    ├── 09-energy.tex
    ├── 10-generative.tex
    ├── 11-advanced-generative.tex
    ├── 12-bio-inspired.tex
    └── 13-emerging.tex
```

All global TikZ styles, the named color palette, notation macros and theorem environments are defined once in `main.tex` and reused by every chapter — guaranteeing visual and mathematical consistency across the whole book.

## ✍️ Author

**Simon-Pierre Boucher**
📧 [contact@spboucher.ai](mailto:contact@spboucher.ai)

## 📜 License

© 2026 Simon-Pierre Boucher. All rights reserved.
The source is published for reading and personal study; please contact the author for any other use.