# Artificial Neural Networks — Methods, Equations and Graphical Representations

A complete, self-contained book on neural networks — every method presented with rigorous equations, an estimation algorithm in pseudocode, and a native TikZ figure.

Simon-Pierre Boucher · contact@spboucher.ai

# 📖 About

This repository contains the full LaTeX source and compiled PDF of a 119-page book covering all major families of artificial neural networks, from Rosenblatt's perceptron (1958) to Kolmogorov–Arnold networks (2024).

The book is built on one strict organizing principle:

Every architecture = rigorous equations + estimation/training algorithm + faithful graphical representation.

Every single figure is drawn natively in TikZ/pgfplots — no imported images — so each diagram is exactly as precise as the equations it illustrates. Notation is unified across all chapters (bold lowercase vectors, bold uppercase matrices, Hadamard product ⊙, shared color palette for inputs/hidden/output/gates/memory).

# 📊 At a glance

# 🗂 Contents

# Part I — Foundations and Learning

1. Foundations of Neural Networks — artificial neuron, perceptron (Novikoff convergence theorem + learning algorithm), activation functions with derivatives, softmax, MLP, universal approximation theorem, XOR geometry, RBF contrast.
2. Training Neural Networks — empirical risk, MSE/cross-entropy, the softmax+CE gradient, the three backpropagation equations (with computational-graph figure and full pseudocode), SGD/Momentum/Nesterov/AdaGrad/RMSProp/Adam with bias correction, learning-rate schedules, Xavier/He initialization.
3. Regularization — bias–variance decomposition, L2/L1, inverted dropout, Batch & Layer Normalization, early stopping, label smoothing, gradient clipping.

# Part II — Core Architectures

4. Convolutional Neural Networks — convolution with stride/padding (worked 5×5 ⊛ 3×3 grid figure), pooling, canonical architecture, ResNet residual block and its gradient, receptive fields, conv forward-pass algorithm.
5. Recurrent Networks: RNN, LSTM and GRU — folded/unfolded views, BPTT (Jacobian products, vanishing/exploding gradients, algorithm), the six LSTM equations with a detailed cell diagram, GRU, bidirectional RNNs, seq2seq.
6. Attention and Transformers — scaled dot-product attention with the 1/√d_k variance argument, multi-head attention, sinusoidal positional encoding, Add&Norm/FFN/causal masking, full encoder–decoder architecture figure, greedy decoding algorithm, BERT/GPT/T5.
7. Modern Transformer Variants — Vision Transformer (patches, [CLS], training algorithm), Mixture of Experts (noisy top-k routing, load balancing), efficient/linear attention, state-space models and Mamba (ZOH discretization, selective scan), scaling laws.

# Part III — Graphs, Energy and Memory

8. Graph Neural Networks — message-passing framework, GCN (spectral derivation), GraphSAGE (minibatch estimation algorithm), GAT, GIN with the 1-WL expressiveness theorem, task heads.
9. Energy-Based and Associative-Memory Networks — Hopfield networks (energy descent theorem, capacity, recall algorithm), modern Hopfield ≈ attention, Boltzmann machines, RBM with contrastive divergence (CD-k), deep belief networks with greedy layer-wise pretraining.

# Part IV — Generative Models

10. Autoencoders, VAEs and GANs — hourglass autoencoder, VAE (ELBO with reconstruction/regularization decomposition, closed-form Gaussian KL, reparameterization trick, training algorithm), GAN (minimax game, optimal discriminator, Jensen–Shannon link, non-saturating loss, WGAN, alternating training algorithm), VAE/GAN/autoregressive comparison table.
11. Normalizing Flows, Diffusion and Autoregressive Models — change of variables, RealNVP coupling layers, DDPM (forward/reverse processes, variational bound, ε-parameterization, training and sampling algorithms), DDIM, classifier-free guidance, WaveNet dilated causal convolutions, energy-based models.

# Part V — Specialized and Emerging Architectures

12. Biologically Inspired Networks — spiking neurons (LIF dynamics, f–I curve, STDP window, surrogate gradients), self-organizing maps (BMU, neighborhood kernel, training loop), reservoir computing / echo state networks (echo-state property, ridge readout).
13. Emerging Architectures — capsule networks (dynamic routing-by-agreement), Neural ODEs (adjoint method), Neural Turing Machines (full addressing pipeline), Kolmogorov–Arnold networks (superposition theorem, learnable edge functions).

# 🔧 Building from source

Requires a full TeX Live installation (pgf/pgfplots, babel-english, algorithm/algpseudocode, booktabs, microtype, hyperref).

cd these  # repository root
pdflatex -interaction=nonstopmode main.tex
pdflatex -interaction=nonstopmode main.tex   # resolve TOC + cross-references
pdflatex -interaction=nonstopmode main.tex   # settle part/page numbers

The build is clean by construction: zero errors, zero undefined references, zero unresolved citations.

# 📁 Repository layout

.
├── README.md
├── CLAUDE.md                 # executable build specification of the book
├── main.pdf                  # compiled book (119 pages)
├── main.tex                  # preamble, cover, preface, abstract, TOC, bibliography
└── chapters/
    ├── 01-foundations.tex
    ├── 02-training.tex
    ├── 03-regularization.tex
    ├── 04-cnn.tex
    ├── 05-rnn.tex
    ├── 06-attention.tex
    ├── 07-modern-transformers.tex
    ├── 08-gnn.tex
    ├── 09-energy.tex
    ├── 10-generative.tex
    ├── 11-advanced-generative.tex
    ├── 12-bio-inspired.tex
    └── 13-emerging.tex

All global TikZ styles, the named color palette, notation macros and theorem environments are defined once in main.tex and reused by every chapter — guaranteeing visual and mathematical consistency across the whole book.

# ✍️ Author

Simon-Pierre Boucher 📧 contact@spboucher.ai

# 📜 License

© 2026 Simon-Pierre Boucher. All rights reserved. The source is published for reading and personal study; please contact the author for any other use.