spb/neural-networks Public

1# Specialized and Emerging Neural Network Architectures23## 1. Graph Neural Networks (GNNs)45### 1.1 The message-passing framework67Most GNNs are instances of the **Message Passing Neural Network (MPNN)** formalism unified by Gilmer et al. (2017), *Neural Message Passing for Quantum Chemistry*, ICML 2017 (arXiv:1704.01212). Given a graph $G=(V,E)$ with node features $x_v$ and edge features $e_{vw}$, the forward pass runs $T$ propagation steps:89$$m_v^{(t+1)} = \sum_{w \in \mathcal{N}(v)} M_t\!\left(h_v^{(t)}, h_w^{(t)}, e_{vw}\right), \qquad h_v^{(t+1)} = U_t\!\left(h_v^{(t)}, m_v^{(t+1)}\right)$$1011where $M_t$ is a learned **message function**, $U_t$ a learned **update function** (Gilmer et al. used a GRU), and $\mathcal{N}(v)$ the neighborhood of $v$. A permutation-invariant **readout** produces a graph-level embedding:1213$$\hat{y} = R\left(\{h_v^{(T)} \mid v \in V\}\right)$$1415The generic modern form separates *aggregation* from *combination*:1617$$a_v^{(k)} = \operatorname{AGGREGATE}^{(k)}\!\left(\{h_u^{(k-1)} : u \in \mathcal{N}(v)\}\right), \qquad h_v^{(k)} = \operatorname{COMBINE}^{(k)}\!\left(h_v^{(k-1)}, a_v^{(k)}\right)$$1819The aggregator must be permutation-invariant (sum, mean, max, attention-weighted sum). This is the single equation from which GCN, GraphSAGE, GAT and GIN are all special cases.2021### 1.2 Graph Convolutional Networks (GCN)2223Kipf & Welling (2017), *Semi-Supervised Classification with Graph Convolutional Networks*, ICLR 2017 (arXiv:1609.02907), derived a first-order localized spectral filter. Starting from spectral graph convolution $g_\theta \star x = U g_\theta(\Lambda) U^\top x$ with $L = I_N - D^{-1/2}AD^{-1/2} = U\Lambda U^\top$, they applied a first-order Chebyshev truncation and a renormalization trick, yielding the celebrated layer-wise propagation rule:2425$$H^{(l+1)} = \sigma\!\left(\tilde{D}^{-\frac{1}{2}}\,\tilde{A}\,\tilde{D}^{-\frac{1}{2}}\,H^{(l)}\,W^{(l)}\right)$$2627with $\tilde{A} = A + I_N$ (adjacency with self-loops), $\tilde{D}_{ii} = \sum_j \tilde{A}_{ij}$, $H^{(0)} = X$, $W^{(l)}$ the trainable weight matrix of layer $l$, and $\sigma$ typically ReLU. The **renormalization trick** ($A \to A+I$ before normalization) keeps the eigenvalues of the propagation operator bounded, preventing exploding/vanishing gradients in deep stacks. In node form:2829$$h_v^{(l+1)} = \sigma\!\left(\sum_{u \in \mathcal{N}(v)\cup\{v\}} \frac{1}{\sqrt{\tilde{d}_v \tilde{d}_u}} W^{(l)} h_u^{(l)}\right)$$3031The symmetric normalization $D^{-1/2}\tilde{A}D^{-1/2}$ is preferred over random-walk normalization $D^{-1}\tilde{A}$ because it keeps the operator symmetric (real spectrum) and downweights messages from high-degree hubs on both endpoints. A two-layer GCN for semi-supervised classification reads $Z = \operatorname{softmax}\!\left(\hat{A}\,\operatorname{ReLU}(\hat{A}XW^{(0)})\,W^{(1)}\right)$ with $\hat{A} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}$.3233**Limitation:** GCN is transductive (needs the full adjacency at training time) and suffers **over-smoothing** — as depth grows, node representations converge to a degree-dependent stationary vector, since repeated multiplication by $\hat{A}$ is a low-pass filter.3435### 1.3 GraphSAGE3637Hamilton, Ying & Leskovec (2017), *Inductive Representation Learning on Large Graphs*, NeurIPS 2017 (arXiv:1706.02216), made GNNs **inductive** by learning aggregator functions over *sampled* fixed-size neighborhoods rather than the full graph:3839$$h_{\mathcal{N}(v)}^{(k)} = \operatorname{AGGREGATE}_k\!\left(\{h_u^{(k-1)}, \forall u \in \mathcal{N}(v)\}\right)$$40$$h_v^{(k)} = \sigma\!\left(W^{(k)} \cdot \operatorname{CONCAT}\!\left(h_v^{(k-1)},\, h_{\mathcal{N}(v)}^{(k)}\right)\right), \qquad h_v^{(k)} \leftarrow \frac{h_v^{(k)}}{\|h_v^{(k)}\|_2}$$4142The concatenation (rather than summation) of the self-vector with the neighborhood vector is a "skip connection" that preserves the node's own identity. Three aggregators were proposed:4344- **Mean:** $h_v^{(k)} = \sigma\!\left(W \cdot \operatorname{mean}(\{h_v^{(k-1)}\} \cup \{h_u^{(k-1)}, u \in \mathcal{N}(v)\})\right)$ — the "GCN-inductive" variant.45- **LSTM:** apply an LSTM to a random permutation of the neighbors (not permutation-invariant, but expressive).46- **Max-pooling:** $\operatorname{AGGREGATE}^{\text{pool}} = \max\!\left(\{\sigma(W_{\text{pool}} h_u^{(k-1)} + b),\ \forall u \in \mathcal{N}(v)\}\right)$, element-wise max.4748Unsupervised training uses a graph-based loss with negative sampling: $J(z_u) = -\log\!\left(\sigma(z_u^\top z_v)\right) - Q\cdot\mathbb{E}_{v_n \sim P_n(v)}\log\!\left(\sigma(-z_u^\top z_{v_n})\right)$, where $v$ co-occurs with $u$ on a fixed-length random walk.4950### 1.4 Graph Attention Networks (GAT)5152Veličković et al. (2018), *Graph Attention Networks*, ICLR 2018 (arXiv:1710.10903), replaced the fixed structural coefficient $1/\sqrt{\tilde d_v \tilde d_u}$ by learned attention. Unnormalized attention logits:5354$$e_{ij} = \operatorname{LeakyReLU}\!\left(\vec{\mathbf{a}}^{\top}\left[\mathbf{W}\vec{h}_i \,\|\, \mathbf{W}\vec{h}_j\right]\right)$$5556with $\|$ concatenation, $\mathbf{W} \in \mathbb{R}^{F' \times F}$ shared, $\vec{\mathbf{a}} \in \mathbb{R}^{2F'}$ the attention vector, and LeakyReLU slope $0.2$. Normalization is a masked softmax over the first-order neighborhood (including $i$ itself):5758$$\alpha_{ij} = \operatorname{softmax}_j(e_{ij}) = \frac{\exp(e_{ij})}{\sum_{k \in \mathcal{N}_i} \exp(e_{ik})}$$5960Node update, and its multi-head ($K$ heads) versions:6162$$\vec{h}_i' = \sigma\!\left(\sum_{j \in \mathcal{N}_i} \alpha_{ij}\,\mathbf{W}\vec{h}_j\right), \qquad63\vec{h}_i' = \Big\|_{k=1}^{K} \sigma\!\left(\sum_{j\in\mathcal{N}_i}\alpha_{ij}^k \mathbf{W}^k \vec{h}_j\right)$$6465and on the final (prediction) layer, heads are **averaged** rather than concatenated: $\vec{h}_i' = \sigma\!\left(\frac{1}{K}\sum_{k=1}^{K}\sum_{j\in\mathcal{N}_i}\alpha_{ij}^k \mathbf{W}^k \vec{h}_j\right)$.6667GAT is inductive, does not require knowing the graph structure upfront, and assigns different importances to neighbors of the same node. Brody, Alon & Yahav (2022), *How Attentive are Graph Attention Networks?* (**GATv2**, ICLR 2022), showed GAT computes only *static* attention (the ranking of neighbors is shared across query nodes) and fixed it by moving the nonlinearity inside: $e_{ij} = \vec{\mathbf{a}}^\top \operatorname{LeakyReLU}\!\left(\mathbf{W}[\vec{h}_i \| \vec{h}_j]\right)$.6869### 1.5 Graph Isomorphism Network (GIN)7071Xu, Hu, Leskovec & Jegelka (2019), *How Powerful are Graph Neural Networks?*, ICLR 2019 (arXiv:1810.00826), proved that a message-passing GNN is **at most as discriminative as the 1-dimensional Weisfeiler-Lehman (1-WL) test**, and that this bound is attained iff the aggregation and readout functions are injective on multisets. Since sum is injective over multisets (mean and max are not), they proposed:7273$$h_v^{(k)} = \operatorname{MLP}^{(k)}\!\left(\left(1 + \epsilon^{(k)}\right)\cdot h_v^{(k-1)} + \sum_{u\in\mathcal{N}(v)} h_u^{(k-1)}\right)$$7475with $\epsilon^{(k)}$ a learnable or fixed scalar (GIN-$\epsilon$ vs GIN-0), which disambiguates the node's own features from its neighbors' sum. For graph-level tasks, they concatenate summed readouts across all depths to preserve both local and global structure:7677$$h_G = \operatorname{CONCAT}\!\left(\operatorname{READOUT}\left(\{h_v^{(k)} \mid v \in G\}\right) \ \Big|\ k = 0,1,\ldots,K\right)$$7879Mean aggregation fails to distinguish multisets with identical distributions but different multiplicities; max aggregation fails to distinguish multisets with identical support. This ordering (sum > mean > max in expressive power) is the central theoretical result of the paper.8081### 1.6 Applications8283- **AlphaFold 2** (Jumper et al., *Nature* 2021, "Highly accurate protein structure prediction with AlphaFold"): the Evoformer treats the pair representation as a fully-connected weighted graph over residues and enforces the **triangle inequality** through triangular multiplicative updates and triangular self-attention — a geometric, graph-structured attention mechanism. The Structure Module then performs Invariant Point Attention (IPA) in SE(3).84- **Molecular property prediction:** MPNN/GIN on QM9, ChEMBL; Stokes et al. (*Cell* 2020) discovered the antibiotic **halicin** with a directed-MPNN (Chemprop).85- **Recommender systems:** PinSAGE (Ying et al., KDD 2018), a GraphSAGE variant deployed on Pinterest's 3-billion-node graph.86- **Physics simulation:** Sanchez-Gonzalez et al. (ICML 2020), *Learning to Simulate Complex Physics with Graph Networks*.87- **Combinatorial optimization, traffic forecasting** (Google Maps ETA), **fraud detection**, **weather** (GraphCast, Lam et al., *Science* 2023).8889---9091## 2. Spiking Neural Networks (SNNs)9293SNNs are the "third generation" of neural networks (Maass, 1997, *Networks of spiking neurons: The third generation of neural network models*, Neural Networks 10(9)). Information is carried by discrete, asynchronous events (spikes) in continuous time rather than by real-valued activations.9495### 2.1 Leaky Integrate-and-Fire (LIF)9697The workhorse model, tracing back to Lapicque (1907):9899$$\tau_m \frac{dV(t)}{dt} = -\left(V(t) - V_{\text{rest}}\right) + R_m I(t)$$100101with $\tau_m = R_m C_m$ the membrane time constant (typically 10–30 ms in cortex). When $V(t)$ crosses threshold $V_{\text{th}}$, a spike is emitted and the potential is reset: $V \leftarrow V_{\text{reset}}$, followed by an absolute refractory period $t_{\text{ref}}$. The equivalent circuit is a leaky RC compartment. The discrete-time form used in deep SNN frameworks (snnTorch, SpikingJelly, Norse):102103$$V[t] = \beta V[t-1] + \sum_j W_j S_j[t] - S[t-1]\,V_{\text{th}}, \qquad S[t] = \Theta\!\left(V[t] - V_{\text{th}}\right)$$104105with decay $\beta = e^{-\Delta t/\tau_m}$ and $\Theta$ the Heaviside step. The last term implements **soft reset** (subtraction) versus **hard reset** ($V \leftarrow 0$).106107Because $\Theta'$ is a Dirac delta, gradient descent requires a **surrogate gradient** (Neftci, Mostafa & Zenke, 2019, *Surrogate Gradient Learning in Spiking Neural Networks*, IEEE Signal Processing Magazine), e.g. the fast sigmoid derivative $\frac{\partial S}{\partial V} \approx \frac{1}{(1 + \gamma|V - V_{\text{th}}|)^2}$ or an arctan/box surrogate. This enables BPTT training of deep SNNs.108109### 2.2 Hodgkin–Huxley110111Hodgkin & Huxley (1952), *A quantitative description of membrane current and its application to conduction and excitation in nerve*, J. Physiol. 117:500–544 (Nobel Prize 1963). The biophysically complete, four-dimensional model of the squid giant axon:112113$$C_m \frac{dV}{dt} = I_{\text{ext}} - \bar{g}_{\text{Na}} m^3 h\,(V - E_{\text{Na}}) - \bar{g}_{\text{K}} n^4 (V - E_{\text{K}}) - \bar{g}_L (V - E_L)$$114115with three gating variables $m$ (Na$^+$ activation), $h$ (Na$^+$ inactivation), $n$ (K$^+$ activation), each obeying first-order kinetics:116117$$\frac{dx}{dt} = \alpha_x(V)(1 - x) - \beta_x(V)\,x = \frac{x_\infty(V) - x}{\tau_x(V)}, \qquad x \in \{m,h,n\}$$118119where $x_\infty = \alpha_x/(\alpha_x + \beta_x)$ and $\tau_x = 1/(\alpha_x + \beta_x)$. Typical parameters: $\bar g_{\text{Na}} = 120$, $\bar g_{\text{K}} = 36$, $\bar g_L = 0.3$ mS/cm², $C_m = 1\,\mu$F/cm². HH is accurate but costs ~1200 FLOPs per 1 ms of simulation, making it impractical for large networks.120121### 2.3 Izhikevich model122123Izhikevich (2003), *Simple Model of Spiking Neurons*, IEEE Transactions on Neural Networks 14(6):1569–1572. A two-dimensional reduction (via normal-form theory of the saddle-node-on-invariant-circle bifurcation) that retains HH-like richness at ~13 FLOPs/ms:124125$$\frac{dv}{dt} = 0.04v^2 + 5v + 140 - u + I, \qquad \frac{du}{dt} = a\,(bv - u)$$126127with the auxiliary after-spike reset:128129$$\text{if } v \geq 30\ \text{mV}, \quad \text{then } \begin{cases} v \leftarrow c \\ u \leftarrow u + d\end{cases}$$130131$v$ is membrane potential (mV), $u$ a recovery variable (K$^+$ activation and Na$^+$ inactivation). The four parameters $(a,b,c,d)$ select the firing regime: regular spiking $(0.02, 0.2, -65, 8)$, intrinsically bursting $(0.02,0.2,-55,4)$, chattering $(0.02,0.2,-50,2)$, fast spiking $(0.1,0.2,-65,2)$, low-threshold spiking, resonator, etc. — reproducing ~20 documented cortical firing patterns.132133### 2.4 STDP134135Spike-Timing-Dependent Plasticity, characterized experimentally by Bi & Poo (1998), *J. Neurosci.* 18(24):10464–10472, and by Markram et al. (1997), *Science*. The classical pair-based, additive exponential window with $\Delta t = t_{\text{post}} - t_{\text{pre}}$:136137$$\Delta w = \begin{cases} A_+ \exp\!\left(-\dfrac{\Delta t}{\tau_+}\right) & \text{if } \Delta t > 0 \quad \text{(LTP: pre before post — causal)} \\[2ex] -A_- \exp\!\left(\dfrac{\Delta t}{\tau_-}\right) & \text{if } \Delta t \leq 0 \quad \text{(LTD: post before pre)} \end{cases}$$138139with $\tau_+ \approx \tau_- \approx 20$ ms and $A_\pm$ the learning rates. This is a **local, unsupervised, Hebbian** rule ("neurons that fire together, wire together" — Hebb, 1949) that implements a causality detector. Online implementation uses pre- and post-synaptic **eligibility traces** $x_{\text{pre}}, x_{\text{post}}$:140141$$\tau_+ \frac{dx_{\text{pre}}}{dt} = -x_{\text{pre}} + \sum_f \delta(t - t^f_{\text{pre}}), \qquad \frac{dw}{dt} = A_+ x_{\text{pre}}\,S_{\text{post}}(t) - A_- x_{\text{post}}\,S_{\text{pre}}(t)$$142143Variants include **multiplicative STDP** (weight-dependent, $A_+ \propto (w_{\max}-w)$, guaranteeing bounded weights), **triplet STDP** (Pfister & Gerstner, 2006), and **R-STDP / dopamine-modulated STDP** where a global reward signal gates the eligibility trace, yielding a biologically plausible reinforcement learning rule.144145### 2.5 Neural coding146147- **Rate coding:** information in the spike count over a window; simple, robust, but high latency and energy.148- **Temporal / latency coding (TTFS, time-to-first-spike):** information in the precise spike time; a single spike per neuron suffices, extremely energy-efficient.149- **Phase coding:** spike time relative to a background oscillation.150- **Population / rank-order coding** (Thorpe et al.): information in the *order* in which neurons in a population fire.151152### 2.6 Neuromorphic hardware153154- **IBM TrueNorth** (Merolla et al., *Science* 345:668–673, 2014): 4096 neurosynaptic cores, **1 million** digital neurons, **256 million** synapses, 5.4 billion transistors on Samsung 28 nm, running at ~**65–70 mW** with ~46 GSOPS/W. Fully event-driven, no clock in the neural fabric.155- **Intel Loihi** (Davies et al., *IEEE Micro* 38(1):82–99, 2018): 128 neuromorphic cores + 3 x86 cores, 130,000 LIF neurons, 130 million synapses, 14 nm, with **on-chip programmable learning rules** (including STDP) via a microcode "learning engine". Discretized dynamics: $u_i[t] = u_i[t-1](1 - \delta^{(u)}_i 2^{-12}) + 2^6\sum_j w_{ij}s_j[t]$, $v_i[t] = v_i[t-1](1-\delta^{(v)}_i 2^{-12}) + u_i[t] + u_{\text{bias}}$.156- **Loihi 2** (Intel, 2021): Intel 4 (pre-production 7 nm), 31 mm², 2.3 billion transistors, up to **1 million neurons** and 120 million synapses, 128 neuromorphic cores of 8192 neurons, 6 x86 cores, **programmable neuron models** (not just LIF) and **graded spikes** (spikes carry a 32-bit payload). Programmed via the open-source **Lava** framework. Hala Point (2024) assembles 1152 Loihi 2 chips for 1.15 billion neurons.157- **SpiNNaker / SpiNNaker2** (Furber et al., University of Manchester): ARM-based massively parallel simulator, 1 million cores in SpiNNaker1.158- **BrainScaleS** (Heidelberg): analog above-threshold, 1000–10000× accelerated against biological real-time.159- **Others:** Tianjic (Tsinghua, *Nature* 2019), Akida (BrainChip), DYNAP-SE (SynSense), IBM NorthPole (2023).160161---162163## 3. Self-Organizing Maps (Kohonen)164165Kohonen (1982), *Self-Organized Formation of Topologically Correct Feature Maps*, Biological Cybernetics 43:59–69; book: *Self-Organizing Maps*, Springer, 1995/2001. An unsupervised, competitive-learning algorithm that projects a high-dimensional input space onto a low-dimensional (usually 2D) discrete lattice while **preserving topology**.166167Each unit $i$ on the lattice carries a codebook (prototype) vector $m_i \in \mathbb{R}^n$. Two steps per sample $x(t)$:168169**(a) Competition — Best Matching Unit (BMU):**170171$$c = \arg\min_i \|x(t) - m_i(t)\| \quad \Longleftrightarrow \quad \|x - m_c\| = \min_i \|x - m_i\|$$172173**(b) Cooperation & adaptation:**174175$$m_i(t+1) = m_i(t) + \alpha(t)\,h_{ci}(t)\,\left[x(t) - m_i(t)\right]$$176177with $\alpha(t) \in (0,1)$ a monotonically decreasing learning rate (e.g. $\alpha(t) = \alpha_0 e^{-t/\lambda}$) and $h_{ci}(t)$ the **neighborhood kernel** measured in *lattice* (not input) space:178179$$h_{ci}(t) = \exp\!\left(-\frac{\|r_c - r_i\|^2}{2\sigma^2(t)}\right), \qquad \sigma(t) = \sigma_0 \exp\!\left(-\frac{t}{\lambda}\right)$$180181or the "bubble" kernel $h_{ci} = \mathbb{1}[\|r_c - r_i\| \le \sigma(t)]$. The radius $\sigma(t)$ starts large (global ordering phase) and shrinks (fine-tuning / convergence phase); this annealing is what produces the topology-preserving unfolding. A **batch SOM** variant exists: $m_i = \frac{\sum_t h_{c(t)i}\,x(t)}{\sum_t h_{c(t)i}}$.182183Quality is assessed with **quantization error** (mean $\|x - m_c\|$) and **topographic error** (fraction of samples whose first and second BMUs are non-adjacent). Applications: exploratory data visualization (U-matrix), WEBSOM document clustering, process monitoring, financial/macroprudential dashboards, gene expression clustering. SOMs are conceptually the ancestor of vector-quantization layers in VQ-VAE.184185---186187## 4. Capsule Networks188189Sabour, Frosst & Hinton (2017), *Dynamic Routing Between Capsules*, NeurIPS 2017 (arXiv:1710.09829), building on Hinton, Krizhevsky & Wang (2011), *Transforming Auto-encoders*. A **capsule** is a group of neurons whose activity *vector* encodes the instantiation parameters (pose, deformation, hue, texture) of an entity, with the **length** of the vector encoding the probability that the entity is present. This addresses CNNs' loss of precise spatial relationships under max-pooling ("equivariance instead of invariance").190191**Squashing nonlinearity** (vector-valued, preserves orientation, maps length into $[0,1)$):192193$$\mathbf{v}_j = \frac{\|\mathbf{s}_j\|^2}{1 + \|\mathbf{s}_j\|^2}\,\frac{\mathbf{s}_j}{\|\mathbf{s}_j\|}$$194195**Prediction vectors** ("votes") from lower capsule $i$ to higher capsule $j$ via a learned pose transformation matrix $\mathbf{W}_{ij}$:196197$$\hat{\mathbf{u}}_{j|i} = \mathbf{W}_{ij}\,\mathbf{u}_i, \qquad \mathbf{s}_j = \sum_i c_{ij}\,\hat{\mathbf{u}}_{j|i}$$198199**Coupling coefficients** by routing softmax over the output capsules:200201$$c_{ij} = \frac{\exp(b_{ij})}{\sum_k \exp(b_{ik})}$$202203**Routing-by-agreement** (typically $r=3$ iterations): initialize $b_{ij} \leftarrow 0$, then repeat204205$$b_{ij} \leftarrow b_{ij} + \hat{\mathbf{u}}_{j|i} \cdot \mathbf{v}_j$$206207i.e. the log-prior is increased when a lower capsule's vote agrees (large scalar product) with the current output of the higher capsule — a form of clustering in pose space that implements part–whole assignment.208209**Margin loss** per class capsule $k$:210211$$L_k = T_k \max(0,\, m^+ - \|\mathbf{v}_k\|)^2 + \lambda\,(1 - T_k)\max(0,\, \|\mathbf{v}_k\| - m^-)^2$$212213with $m^+ = 0.9$, $m^- = 0.1$, $\lambda = 0.5$, $T_k = 1$ iff class $k$ is present. A reconstruction decoder adds a scaled-down ($0.0005$) MSE regularizer. CapsNet reached 0.25% error on MNIST and was notably strong on overlapping digits (MultiMNIST). Follow-up: **EM routing** with matrix capsules (Hinton, Sabour & Frosst, ICLR 2018) and **Stacked Capsule Autoencoders** (Kosiorek et al., NeurIPS 2019). Capsules remain computationally expensive and have not scaled to ImageNet-class problems.214215---216217## 5. Neural Ordinary Differential Equations218219Chen, Rubanova, Bettencourt & Duvenaud (2018), *Neural Ordinary Differential Equations*, NeurIPS 2018 **Best Paper** (arXiv:1806.07366). Observing that a residual block $h_{t+1} = h_t + f(h_t, \theta_t)$ is an Euler discretization, they take the continuous limit:220221$$\frac{d\mathbf{h}(t)}{dt} = f\!\left(\mathbf{h}(t), t, \theta\right), \qquad \mathbf{h}(t_1) = \mathbf{h}(t_0) + \int_{t_0}^{t_1} f(\mathbf{h}(t), t, \theta)\,dt = \operatorname{ODESolve}(\mathbf{h}(t_0), f, t_0, t_1, \theta)$$222223The network becomes a **continuous-depth** model; the ODE solver (Dormand–Prince, adaptive Runge–Kutta) chooses the number of function evaluations, trading accuracy for compute *at test time*.224225**Adjoint sensitivity method** (Pontryagin et al., 1962) gives $O(1)$ memory in depth, since no intermediate activations need storing. Define the adjoint $\mathbf{a}(t) = \partial L/\partial \mathbf{h}(t)$. It obeys a backward ODE:226227$$\frac{d\mathbf{a}(t)}{dt} = -\mathbf{a}(t)^{\top}\frac{\partial f(\mathbf{h}(t), t, \theta)}{\partial \mathbf{h}}$$228229and the parameter gradient is a single quadrature:230231$$\frac{dL}{d\theta} = -\int_{t_1}^{t_0} \mathbf{a}(t)^{\top}\,\frac{\partial f(\mathbf{h}(t), t, \theta)}{\partial \theta}\,dt$$232233In practice one integrates the **augmented state** $[\mathbf{h}, \mathbf{a}, \partial L/\partial\theta]$ backwards in time in a single solver call; the vector-Jacobian products $\mathbf{a}^\top \partial f/\partial \mathbf{h}$ are obtained by ordinary reverse-mode autodiff on $f$ alone.234235**Consequences and descendants:**236- **Continuous Normalizing Flows / FFJORD**: the instantaneous change of variables $\frac{\partial \log p(\mathbf{z}(t))}{\partial t} = -\operatorname{tr}\!\left(\frac{\partial f}{\partial \mathbf{z}(t)}\right)$ replaces the $O(d^3)$ log-determinant of discrete flows by an $O(d)$ trace (Hutchinson estimator).237- **Latent ODEs / ODE-RNN** for irregularly-sampled time series.238- **Augmented Neural ODEs** (Dupont, Doucet & Teh, NeurIPS 2019): ODE flows are homeomorphisms and cannot cross trajectories, so some functions are unrepresentable; lifting to $[\mathbf{x}; \mathbf{a}]$ with extra dimensions fixes this.239- **Neural SDEs, Neural CDEs** (Kidger et al., 2020), **Hamiltonian/Lagrangian Neural Networks**.240241---242243## 6. Physics-Informed Neural Networks (PINNs)244245Raissi, Perdikaris & Karniadakis (2019), *Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations*, Journal of Computational Physics 378:686–707 (preprints arXiv:1711.10561/10566). Antecedent: Lagaris, Likas & Fotiadis (1998).246247Consider a PDE in general form on $\Omega \times [0,T]$:248249$$u_t + \mathcal{N}[u;\lambda] = 0, \quad x \in \Omega,\ t\in[0,T]$$250251Approximate $u(x,t) \approx u_\theta(x,t)$ by an MLP and define the **PDE residual**, computed exactly by automatic differentiation of the network with respect to its inputs:252253$$r_\theta(x,t) := \frac{\partial u_\theta}{\partial t} + \mathcal{N}\!\left[u_\theta; \lambda\right]$$254255The **composite loss** is a weighted sum of mean-squared terms:256257$$\mathcal{L}(\theta) = \underbrace{\lambda_r \frac{1}{N_r}\sum_{i=1}^{N_r}\left|r_\theta(x_r^i, t_r^i)\right|^2}_{\text{PDE residual (collocation points)}} + \underbrace{\lambda_b \frac{1}{N_b}\sum_{i=1}^{N_b}\left|u_\theta(x_b^i,t_b^i) - g^i\right|^2}_{\text{boundary conditions}} + \underbrace{\lambda_0\frac{1}{N_0}\sum_{i=1}^{N_0}\left|u_\theta(x_0^i,0) - u_0^i\right|^2}_{\text{initial condition}} + \underbrace{\lambda_d \frac{1}{N_d}\sum_{i=1}^{N_d}\left|u_\theta(x_d^i,t_d^i) - u_d^i\right|^2}_{\text{observed data}}$$258259Key properties:260- **Mesh-free**: collocation points are sampled (uniformly, Latin hypercube, or adaptively) rather than discretized on a grid; the curse of dimensionality is much milder than for finite elements.261- **Inverse problems come for free**: the unknown physical parameters $\lambda$ (e.g. viscosity, diffusivity) are simply additional trainable variables optimized jointly with $\theta$ against sparse data.262- **Training**: typically Adam followed by L-BFGS; `tanh` activations (need smooth higher derivatives).263264**Known pathologies:** severe **loss-term imbalance** (gradient pathologies, Wang, Teng & Perdikaris, SIAM J. Sci. Comput. 2021 — fixed by learning-rate annealing or NTK-based weighting), **spectral bias** against high-frequency solutions, and failure on stiff/convection-dominated regimes ("failure modes", Krishnapriyan et al., NeurIPS 2021). Remedies include hard-constrained boundary conditions ($u_\theta = g + B(x)\,\text{NN}(x)$), domain decomposition (XPINN, cPINN), causal training weights, and Fourier feature embeddings. Extensions: **DeepONet** (Lu, Jin & Karniadakis, *Nature Machine Intelligence* 2021) and **Fourier Neural Operator** (Li et al., ICLR 2021), which learn operators between function spaces rather than single solutions.265266---267268## 7. Neural Radiance Fields (NeRF)269270Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi & Ng (2020), *NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis*, ECCV 2020 **Best Paper Honorable Mention** (arXiv:2003.08934). A scene is a *single* MLP:271272$$F_\Theta : (\mathbf{x}, \mathbf{d}) = (x,y,z,\theta,\phi) \longmapsto (\mathbf{c}, \sigma) = (r,g,b,\sigma)$$273274with volume density $\sigma$ predicted from position only (enforcing multi-view consistent geometry) and view-dependent color $\mathbf{c}$ from position **and** viewing direction (enabling specularities).275276**Volume rendering** along a camera ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$, between near and far bounds $[t_n, t_f]$ (classical emission-absorption model, Kajiya & Von Herzen 1984):277278$$C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,\mathbf{c}(\mathbf{r}(t), \mathbf{d})\,dt, \qquad T(t) = \exp\!\left(-\int_{t_n}^{t}\sigma(\mathbf{r}(s))\,ds\right)$$279280where $T(t)$ is the **accumulated transmittance** (probability the ray travels from $t_n$ to $t$ without being absorbed). The quadrature approximation over $N$ stratified samples with intervals $\delta_i = t_{i+1} - t_i$:281282$$\hat{C}(\mathbf{r}) = \sum_{i=1}^{N} T_i\,\alpha_i\,\mathbf{c}_i, \qquad \alpha_i = 1 - \exp(-\sigma_i\delta_i), \qquad T_i = \exp\!\left(-\sum_{j=1}^{i-1}\sigma_j\delta_j\right) = \prod_{j=1}^{i-1}(1-\alpha_j)$$283284This is exactly alpha compositing and is **fully differentiable**, so the whole pipeline trains by photometric loss alone: $\mathcal{L} = \sum_{\mathbf{r}\in\mathcal{R}} \left[\|\hat{C}_c(\mathbf{r}) - C(\mathbf{r})\|_2^2 + \|\hat{C}_f(\mathbf{r}) - C(\mathbf{r})\|_2^2\right]$ over the coarse and fine networks (hierarchical sampling: the coarse network's weights $w_i = T_i\alpha_i$ define a PDF used for inverse-transform sampling of the fine network).285286**Positional encoding** is essential: MLPs exhibit spectral bias toward low frequencies, so raw $(x,y,z)$ inputs give blurry reconstructions. NeRF maps each scalar coordinate through:287288$$\gamma(p) = \left(\sin(2^0\pi p), \cos(2^0 \pi p), \sin(2^1 \pi p), \cos(2^1\pi p), \ldots, \sin(2^{L-1}\pi p), \cos(2^{L-1}\pi p)\right)$$289290with $L=10$ for $\mathbf{x}$ (60 dims) and $L=4$ for $\mathbf{d}$ (24 dims). Tancik et al. (NeurIPS 2020), *Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains*, explained this via NTK: the encoding converts the NTK into a stationary, tunable-bandwidth kernel.291292Descendants: **Mip-NeRF** (anti-aliasing via integrated positional encoding of conical frustums), **Instant-NGP** (Müller et al., SIGGRAPH 2022 — multiresolution hash encoding, seconds instead of days), **Plenoxels**, **NeRF in the Wild**, **Zip-NeRF**, and the successor paradigm **3D Gaussian Splatting** (Kerbl et al., SIGGRAPH 2023), which replaces the MLP with explicit anisotropic Gaussians and rasterization for real-time rendering.293294---295296## 8. Implicit Neural Representations and SIREN297298An **implicit neural representation** (INR) encodes a signal as a continuous function $\Phi_\theta : \mathbb{R}^n \to \mathbb{R}^m$ parameterized by network weights, rather than as a discrete grid — resolution-independent, memory scaling with signal complexity rather than resolution. Examples: **DeepSDF** (Park et al., CVPR 2019, signed distance functions), **Occupancy Networks** (Mescheder et al., CVPR 2019), NeRF.299300**SIREN** — Sitzmann, Martel, Bergman, Lindell & Wetzstein (2020), *Implicit Neural Representations with Periodic Activation Functions*, NeurIPS 2020 **Oral** (arXiv:2006.09661). Replace ReLU by a sine:301302$$\Phi(\mathbf{x}) = \mathbf{W}_n\left(\phi_{n-1}\circ\phi_{n-2}\circ\cdots\circ\phi_0\right)(\mathbf{x}) + \mathbf{b}_n, \qquad \phi_i(\mathbf{x}_i) = \sin\!\left(\omega_0\,\mathbf{W}_i\mathbf{x}_i + \mathbf{b}_i\right)$$303304Crucially, **the derivative of a SIREN is itself a SIREN** (since $\frac{d}{dx}\sin(x) = \cos(x) = \sin(x + \pi/2)$), so all higher-order derivatives are well-behaved and nonzero — unlike ReLU networks, whose second derivative vanishes everywhere. This lets SIRENs be supervised directly on derivative constraints: solving the **eikonal equation** $\|\nabla_\mathbf{x}\Phi\| = 1$ for SDFs, the **Poisson equation** from gradients/Laplacians only, or the Helmholtz/wave equations.305306**Principled initialization** is required to keep activation distributions stable across depth: draw $w_i \sim \mathcal{U}\!\left(-\sqrt{6/\text{fan\_in}},\, \sqrt{6/\text{fan\_in}}\right)$, so that the pre-activations are normally distributed and the post-activations are arcsine-distributed, with $\omega_0 = 30$ for the first layer to span the input frequency spectrum.307308Related: **Fourier Feature Networks** (Tancik et al. 2020), **WIRE** (Gabor wavelet activations, CVPR 2023), **BACON**, **MFN** (multiplicative filter networks), and **functa**/**hypernetwork-conditioned INRs** for generative modeling over signals.309310---311312## 9. Networks for Deep Reinforcement Learning313314### 9.1 Deep Q-Networks (DQN)315316Mnih et al. (2015), *Human-level control through deep reinforcement learning*, *Nature* 518:529–533 (NIPS workshop version 2013). The optimal action-value function obeys the **Bellman optimality equation**:317318$$Q^*(s,a) = \mathbb{E}_{s'\sim\mathcal{E}}\!\left[r + \gamma \max_{a'} Q^*(s',a') \,\Big|\, s,a\right]$$319320DQN approximates $Q^*(s,a) \approx Q(s,a;\theta)$ with a CNN over raw pixels and minimizes the **TD loss**:321322$$L_i(\theta_i) = \mathbb{E}_{(s,a,r,s')\sim U(\mathcal{D})}\left[\left(\underbrace{r + \gamma\max_{a'}Q(s',a';\theta^-)}_{\text{TD target } y} - Q(s,a;\theta_i)\right)^2\right]$$323324Two stabilizing innovations: (i) **experience replay** — sample uniformly from a buffer $\mathcal{D}$ to break temporal correlations; (ii) a **target network** with frozen parameters $\theta^-$, synchronized every $C$ steps ($\theta^- \leftarrow \theta$), which prevents the target from chasing the prediction. Gradient: $\nabla_{\theta_i}L_i = \mathbb{E}\left[(y - Q(s,a;\theta_i))\nabla_{\theta_i}Q(s,a;\theta_i)\right]$; Huber loss is used in practice for robustness.325326Extensions, consolidated in **Rainbow** (Hessel et al., AAAI 2018): **Double DQN** (van Hasselt et al., AAAI 2016 — decouple selection from evaluation, $y = r + \gamma Q(s', \arg\max_{a'}Q(s',a';\theta);\theta^-)$, curing overestimation bias); **Dueling networks** (Wang et al., ICML 2016 — $Q(s,a) = V(s) + A(s,a) - \frac{1}{|\mathcal{A}|}\sum_{a'}A(s,a')$); **Prioritized Experience Replay** (Schaul et al., ICLR 2016 — $p_i \propto |\delta_i|^\alpha$ with importance-sampling correction); **Noisy Nets**; **distributional RL / C51** (Bellemare et al., ICML 2017); **n-step returns**.327328### 9.2 Policy gradients and REINFORCE329330The **policy gradient theorem** (Sutton, McAllester, Singh & Mansour, NeurIPS 2000) for $J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}[R(\tau)]$:331332$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau\sim\pi_\theta}\left[\sum_{t=0}^{T}\nabla_\theta \log \pi_\theta(a_t|s_t)\, \Psi_t\right]$$333334**REINFORCE** (Williams, 1992, *Simple statistical gradient-following algorithms for connectionist reinforcement learning*, Machine Learning 8:229–256) takes $\Psi_t = G_t = \sum_{k=t}^{T}\gamma^{k-t}r_k$, giving the update $\theta \leftarrow \theta + \alpha\,\gamma^t G_t \nabla_\theta \log\pi_\theta(a_t|s_t)$. It is unbiased but has very high variance; subtracting a state-dependent **baseline** $b(s_t)$ leaves it unbiased (since $\mathbb{E}[\nabla_\theta \log\pi_\theta(a|s)\,b(s)] = 0$) and reduces variance:335336$$\nabla_\theta J = \mathbb{E}\left[\sum_t \nabla_\theta \log\pi_\theta(a_t|s_t)\left(G_t - b(s_t)\right)\right]$$337338### 9.3 Actor-Critic and A3C339340Setting $b(s) = V^\pi_\phi(s)$ and $\Psi_t = A^\pi(s_t,a_t) = Q^\pi(s_t,a_t) - V^\pi(s_t)$ gives **advantage actor-critic**. With the $n$-step estimator $\hat{A}_t = \sum_{i=0}^{n-1}\gamma^i r_{t+i} + \gamma^n V_\phi(s_{t+n}) - V_\phi(s_t)$, the combined objective is:341342$$\mathcal{L} = -\log\pi_\theta(a_t|s_t)\,\hat{A}_t + c_v\left(V_\phi(s_t) - G_t\right)^2 - c_e\,H\!\left(\pi_\theta(\cdot|s_t)\right)$$343344where $H$ is the policy entropy, encouraging exploration (typically $c_e = 0.01$, $c_v = 0.5$).345346**A3C** — Mnih et al. (2016), *Asynchronous Methods for Deep Reinforcement Learning*, ICML 2016. Multiple CPU actor-learners explore in parallel and apply **Hogwild!**-style asynchronous updates to shared parameters; the decorrelation induced by parallel exploration replaces the replay buffer, allowing on-policy learning and training on CPUs alone. **A2C** is the synchronous, batched variant that dominates in practice on GPUs.347348**GAE** (Schulman et al., ICLR 2016) interpolates bias/variance: $\hat{A}_t^{\text{GAE}(\gamma,\lambda)} = \sum_{l=0}^{\infty}(\gamma\lambda)^l\delta_{t+l}$ with $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$.349350### 9.4 TRPO and PPO351352TRPO (Schulman et al., ICML 2015) maximizes a surrogate under a KL trust region: $\max_\theta \mathbb{E}\left[\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}\hat A\right]$ s.t. $\mathbb{E}\left[D_{\text{KL}}(\pi_{\theta_{\text{old}}}\|\pi_\theta)\right] \le \delta$, requiring conjugate gradients and Fisher-vector products.353354**PPO** — Schulman, Wolski, Dhariwal, Radford & Klimov (2017), *Proximal Policy Optimization Algorithms* (arXiv:1707.06347) — achieves the same effect first-order. With the probability ratio $r_t(\theta) = \dfrac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$, the **clipped surrogate objective** is:355356$$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t\left[\min\Big(r_t(\theta)\,\hat{A}_t,\ \operatorname{clip}\big(r_t(\theta),\,1-\epsilon,\,1+\epsilon\big)\,\hat{A}_t\Big)\right]$$357358typically $\epsilon = 0.2$. The $\min$ makes the objective a **pessimistic lower bound**: when $\hat A_t > 0$ the gain is capped at $1+\epsilon$; when $\hat A_t < 0$ the loss is capped at $1-\epsilon$; but the unclipped branch is retained when it is *worse*, so a policy that has moved too far in the wrong direction is still pulled back. The full loss combines value and entropy terms:359360$$L^{\text{CLIP+VF+S}}(\theta) = \hat{\mathbb{E}}_t\left[L^{\text{CLIP}}_t(\theta) - c_1 L^{\text{VF}}_t(\theta) + c_2\,S[\pi_\theta](s_t)\right]$$361362PPO allows multiple epochs of minibatch SGD on the same rollout, is simple and robust, and became the default for continuous control, Dota 2 (OpenAI Five), and **RLHF for large language models** (Ouyang et al., NeurIPS 2022, InstructGPT), where the reward is a learned preference model and a per-token KL penalty to the reference policy is added. An alternative penalized form is $L^{\text{KLPEN}} = \hat{\mathbb{E}}\left[r_t(\theta)\hat A_t - \beta\,\text{KL}[\pi_{\theta_{\text{old}}}, \pi_\theta]\right]$ with adaptive $\beta$.363364Off-policy actor-critics complete the picture: **DDPG** (Lillicrap et al., ICLR 2016), **TD3** (Fujimoto et al., ICML 2018), **SAC** (Haarnoja et al., ICML 2018) with the maximum-entropy objective $J(\pi) = \sum_t \mathbb{E}\left[r_t + \alpha H(\pi(\cdot|s_t))\right]$.365366### 9.5 AlphaGo / AlphaZero / MuZero367368- **AlphaGo** (Silver et al., *Nature* 529:484–489, 2016): SL policy network trained on human expert moves, an RL policy network refined by self-play, a fast rollout policy, and a value network, all combined in MCTS. Defeated Lee Sedol 4–1 in 2016.369- **AlphaGo Zero** (Silver et al., *Nature* 550:354–359, 2017) and **AlphaZero** (Silver et al., *Science* 362:1140–1144, 2018): a single residual CNN $(\mathbf{p}, v) = f_\theta(s)$ outputs a move-prior vector and a scalar value, trained **tabula rasa** purely from self-play, no human data, generalized to chess and shogi.370371MCTS selection uses a **PUCT** rule (Rosin 2011, adapted):372373$$a_t = \arg\max_a\left(Q(s_t,a) + U(s_t,a)\right), \qquad U(s,a) = c_{\text{puct}}\,P(s,a)\,\frac{\sqrt{\sum_b N(s,b)}}{1 + N(s,a)}$$374375where $P(s,a)$ is the network prior, $N(s,a)$ the visit count and $Q(s,a) = \frac{1}{N(s,a)}\sum_{s'|s,a\to s'} V(s')$ the mean value. Search acts as a **policy improvement operator**: the visit-count distribution $\pi_a \propto N(s,a)^{1/\tau}$ is stronger than the raw network prior, and the network is then trained to imitate it:376377$$\ell = (z - v)^2 - \boldsymbol{\pi}^{\top}\log \mathbf{p} + c\|\theta\|^2$$378379with $z \in \{-1,0,+1\}$ the self-play game outcome. Dirichlet noise is added at the root ($P(s,a) = (1-\varepsilon)p_a + \varepsilon\eta_a$, $\eta \sim \text{Dir}(0.03)$) for exploration.380381**MuZero** (Schrittwieser et al., *Nature* 588:604–609, 2020) removes the need for a known simulator by learning a representation function $h$, dynamics function $g$ and prediction function $f$ in a latent space, trained only to predict reward, value and policy — a **learned model** for planning.382383---384385## 10. Siamese Networks and Metric Learning386387Introduced by Bromley, Guyon, LeCun, Säckinger & Shah (1993), *Signature Verification using a "Siamese" Time Delay Neural Network*, NeurIPS 1993. Two (or more) identical towers with **shared weights** map inputs into an embedding space $f_\theta: \mathcal{X}\to\mathbb{R}^d$ where semantic similarity becomes geometric distance. This enables **one-shot / few-shot** recognition and open-set verification, where the class set is not fixed at training time.388389**Contrastive loss** — Hadsell, Chopra & LeCun (2006), *Dimensionality Reduction by Learning an Invariant Mapping*, CVPR 2006 (see also Chopra, Hadsell & LeCun, CVPR 2005). With $Y=0$ for a similar pair and $Y=1$ for dissimilar, and $D_W = \|f_\theta(x_1) - f_\theta(x_2)\|_2$:390391$$\mathcal{L}(W, Y, x_1, x_2) = (1-Y)\,\frac{1}{2}\,D_W^2 \;+\; Y\,\frac{1}{2}\left\{\max\left(0,\, m - D_W\right)\right\}^2$$392393Similar pairs are pulled together with a quadratic spring; dissimilar pairs are pushed apart only until they reach margin $m$, after which they contribute no gradient (otherwise all negatives would be pushed to infinity).394395**Triplet loss** — Schroff, Kalenichenko & Philbin (2015), *FaceNet: A Unified Embedding for Face Recognition and Clustering*, CVPR 2015; earlier ranking form in Weinberger & Saul (LMNN, JMLR 2009). Given an anchor $a$, positive $p$ (same class) and negative $n$, with embeddings constrained to the unit hypersphere $\|f(x)\|_2 = 1$:396397$$\mathcal{L}_{\text{triplet}} = \sum_{i}^{N}\left[\left\|f(x_i^a) - f(x_i^p)\right\|_2^2 - \left\|f(x_i^a) - f(x_i^n)\right\|_2^2 + \alpha\right]_{+}$$398399with $[\cdot]_+ = \max(0,\cdot)$ and margin $\alpha$ (0.2 in FaceNet). The loss is **relative** rather than absolute: it only demands the negative be farther than the positive by $\alpha$, which is a weaker and better-behaved constraint than contrastive loss's absolute distance targets.400401**Mining is critical**: random triplets are quickly satisfied and give zero gradient. FaceNet uses *semi-hard* online mining within a large minibatch (~1800): negatives with $\|f_a - f_p\|^2 < \|f_a - f_n\|^2 < \|f_a - f_p\|^2 + \alpha$ — harder than the positive but still inside the margin. Selecting the *hardest* negatives causes collapse to $f(x) = 0$.402403**Extensions:** Koch, Zemel & Salakhutdinov (ICML Deep Learning Workshop 2015) on one-shot Omniglot with binary cross-entropy over the L1 distance; **N-pair loss** (Sohn, NeurIPS 2016); **Lifted structured loss**; **angular margin softmax** losses — SphereFace (A-Softmax), CosFace, and **ArcFace** (Deng et al., CVPR 2019) with $\mathcal{L} = -\log\frac{e^{s\cos(\theta_{y_i}+m)}}{e^{s\cos(\theta_{y_i}+m)} + \sum_{j\ne y_i}e^{s\cos\theta_j}}$; and the **InfoNCE** loss (Oord et al. 2018) at the heart of SimCLR/MoCo/CLIP: $\mathcal{L} = -\log\frac{\exp(\text{sim}(z_i,z_j)/\tau)}{\sum_{k\ne i}\exp(\text{sim}(z_i,z_k)/\tau)}$. Non-contrastive siamese methods (BYOL, Grill et al. NeurIPS 2020; SimSiam, Chen & He CVPR 2021; **Barlow Twins**; **VICReg**) avoid negatives entirely via stop-gradient, predictor heads, or redundancy-reduction terms.404405---406407## 11. Other Specialized and Emerging Architectures408409### 11.1 Extreme Learning Machines (ELM)410411Huang, Zhu & Siew (2006), *Extreme learning machine: Theory and applications*, Neurocomputing 70(1–3):489–501 (ICNN 2004 conference version). For a single-hidden-layer feedforward network with $L$ hidden nodes, **randomly** assign input weights $\mathbf{w}_i$ and biases $b_i$ and never tune them; only the output weights $\boldsymbol{\beta}$ are learned. The hidden-layer output matrix is $H_{ij} = g(\mathbf{w}_j\cdot\mathbf{x}_i + b_j)$, and training reduces to the linear least-squares problem $\mathbf{H}\boldsymbol{\beta} = \mathbf{T}$, whose minimum-norm solution is:412413$$\hat{\boldsymbol{\beta}} = \mathbf{H}^{\dagger}\mathbf{T}$$414415with $\mathbf{H}^\dagger$ the **Moore–Penrose pseudoinverse**; the regularized version is $\hat{\boldsymbol{\beta}} = \left(\mathbf{H}^\top\mathbf{H} + \frac{\mathbf{I}}{C}\right)^{-1}\mathbf{H}^\top\mathbf{T}$. Training is non-iterative and orders of magnitude faster than backpropagation, with universal approximation guarantees. Criticism (notably by Wang & Li, 2017) is that ELM largely restates Schmidt et al. (1992) and the Random Vector Functional Link (Pao et al., 1994). The idea is the feedforward cousin of **reservoir computing** — Echo State Networks (Jaeger, 2001) and Liquid State Machines (Maass, Natschläger & Markram, 2002) — where a fixed random recurrent reservoir is read out linearly.416417### 11.2 Deep Equilibrium Models (DEQ)418419Bai, Kolter & Koltun (2019), *Deep Equilibrium Models*, NeurIPS 2019 (arXiv:1909.01377). Instead of stacking $L$ layers, directly solve for the **fixed point** of a single weight-tied layer, equivalent to an infinite-depth network:420421$$\mathbf{z}^\star = f_\theta(\mathbf{z}^\star; \mathbf{x})$$422423found with a black-box root solver (Broyden's method, Anderson acceleration) on $g_\theta(\mathbf{z}) = f_\theta(\mathbf{z};\mathbf{x}) - \mathbf{z} = 0$. Backpropagation uses the **implicit function theorem**, not the solver's trajectory:424425$$\frac{\partial \ell}{\partial(\cdot)} = \frac{\partial \ell}{\partial \mathbf{z}^\star}\left(I - J_{f_\theta}(\mathbf{z}^\star)\right)^{-1}\frac{\partial f_\theta(\mathbf{z}^\star;\mathbf{x})}{\partial(\cdot)}, \qquad J_{f_\theta}(\mathbf{z}^\star) = \left.\frac{\partial f_\theta}{\partial \mathbf{z}}\right|_{\mathbf{z}^\star}$$426427The inverse-Jacobian-vector product is itself computed by solving a linear fixed-point system, so **memory is $O(1)$ in depth** (up to 88% reduction reported), independent of the number of solver iterations. Concerns are ill-conditioning and non-existence/instability of the fixed point, addressed by **Jacobian regularization** (Bai, Koltun et al., NeurIPS 2021, penalizing $\|J\|_F$ via Hutchinson estimation) and monotone operator formulations (**monDEQ**, Winston & Kolter, NeurIPS 2020). DEQ is the discrete-implicit sibling of Neural ODEs; both belong to the family of **implicit layers** (see also OptNet, differentiable convex optimization layers).428429### 11.3 HyperNetworks430431Ha, Dai & Le (2017), *HyperNetworks*, ICLR 2017 (arXiv:1609.09106). A small network $g_\psi$ **generates the weights** of a larger primary network:432433$$\theta^{(l)} = g_\psi\!\left(\mathbf{e}^{(l)}\right)$$434435where $\mathbf{e}^{(l)}$ is a learned per-layer embedding. In the static case (CNNs) this is a form of relaxed weight sharing and drastic compression; in the dynamic case (recurrent HyperLSTM) the hypernetwork is itself recurrent and produces **input-dependent, time-varying** weights — a mechanism closely related to fast weights (Schmidhuber, 1992) and to linear attention. Modern descendants: **hypernetwork-conditioned INRs**, **HyperMorph** for medical registration, **task-conditioned hypernetworks** for continual learning and meta-learning, and the weight-generation view of **LoRA**-style adapters.436437### 11.4 Neural Architecture Search (NAS)438439- **RL-based**: Zoph & Le (2017), *Neural Architecture Search with Reinforcement Learning*, ICLR 2017 — an RNN controller samples architecture descriptions and is trained with REINFORCE on validation accuracy $R$: $\nabla_{\theta_c}J = \sum_t \mathbb{E}\left[\nabla_{\theta_c}\log P(a_t|a_{(t-1):1};\theta_c)(R - b)\right]$. Cost: ~800 GPUs for weeks. NASNet (Zoph et al., CVPR 2018) searched transferable cells rather than whole networks.440- **Evolutionary**: Real et al. (AAAI 2019), *Regularized Evolution for Image Classifier Architecture Search* (AmoebaNet), with age-based tournament selection.441- **Differentiable**: Liu, Simonyan & Yang (2019), *DARTS: Differentiable Architecture Search*, ICLR 2019. Relax the categorical choice of operation on edge $(i,j)$ into a softmax mixture over the operation set $\mathcal{O}$:442443$$\bar{o}^{(i,j)}(x) = \sum_{o\in\mathcal{O}}\frac{\exp\left(\alpha_o^{(i,j)}\right)}{\sum_{o'\in\mathcal{O}}\exp\left(\alpha_{o'}^{(i,j)}\right)}\,o(x)$$444445and solve the **bilevel** problem $\min_\alpha \mathcal{L}_{\text{val}}(w^*(\alpha), \alpha)$ s.t. $w^*(\alpha) = \arg\min_w \mathcal{L}_{\text{train}}(w,\alpha)$ with a one-step (second-order or first-order) approximation. Search cost drops to ~1 GPU-day. Discretization keeps $o^{(i,j)} = \arg\max_o \alpha^{(i,j)}_o$. Known failure mode: collapse to parameter-free skip connections; fixes include DARTS+, PC-DARTS, Fair DARTS, and early stopping on the Hessian eigenvalue.446- **One-shot / weight sharing**: ENAS, Once-for-All (Cai et al., ICLR 2020), BigNAS; **zero-cost proxies** and NAS benchmarks (NAS-Bench-101/201/301). Practical outcomes: EfficientNet (compound scaling), MnasNet, MobileNetV3.447448### 11.5 Binarized and Quantized Neural Networks449450**BinaryConnect** (Courbariaux, Bengio & David, NeurIPS 2015), **BinaryNet/BNN** (Hubara, Courbariaux et al., NeurIPS 2016), **XNOR-Net** (Rastegari et al., ECCV 2016). Weights and/or activations are constrained to $\{-1,+1\}$:451452$$x^b = \operatorname{sign}(x) = \begin{cases} +1 & x \ge 0 \\ -1 & x < 0\end{cases}$$453454so that the dot product collapses to XNOR + popcount, giving up to 32× memory compression and ~58× speedup on CPU. Since $\partial\,\text{sign}/\partial x = 0$ a.e., training uses the **straight-through estimator** (Hinton, 2012; Bengio, Léonard & Courville, 2013) with a clipping/hard-tanh window:455456$$\frac{\partial \mathcal{L}}{\partial x} \approx \frac{\partial\mathcal{L}}{\partial x^b}\cdot\mathbb{1}_{|x|\le 1}$$457458while a **latent real-valued** shadow copy of the weights accumulates the small gradient updates. XNOR-Net adds per-channel scaling factors $\alpha = \frac{1}{n}\|W\|_{\ell 1}$ to reduce the quantization error $\|W - \alpha B\|^2$. Modern relatives: **quantization-aware training** (Jacob et al., CVPR 2018), **LSQ** (learned step size), post-training quantization (GPTQ, AWQ), and 1-bit LLMs (**BitNet b1.58**, Ma et al. 2024, with ternary $\{-1,0,1\}$ weights).459460### 11.6 Bayesian Neural Networks461462Foundations: MacKay (1992, evidence framework), Neal (1995, HMC and the NN–GP correspondence). BNNs place a prior $p(\mathbf{w})$ over weights and seek the posterior $p(\mathbf{w}|\mathcal{D}) = \frac{p(\mathcal{D}|\mathbf{w})p(\mathbf{w})}{p(\mathcal{D})}$, giving calibrated **epistemic uncertainty**; prediction marginalizes: $p(y^*|x^*,\mathcal{D}) = \int p(y^*|x^*,\mathbf{w})\,p(\mathbf{w}|\mathcal{D})\,d\mathbf{w}$.463464The posterior is intractable, so **variational inference** fits $q_\phi(\mathbf{w})$ (typically fully-factorized Gaussian, $\mathbf{w}\sim\mathcal{N}(\mu, \sigma^2)$) by minimizing the variational free energy / maximizing the **ELBO**:465466$$\mathcal{F}(\phi) = D_{\text{KL}}\!\left(q_\phi(\mathbf{w})\,\|\,p(\mathbf{w})\right) - \mathbb{E}_{q_\phi(\mathbf{w})}\left[\log p(\mathcal{D}|\mathbf{w})\right] = -\text{ELBO}$$467468**Bayes by Backprop** — Blundell, Cornebise, Kavukcuoglu & Wierstra (2015), *Weight Uncertainty in Neural Networks*, ICML 2015 — makes this trainable by the reparameterization trick $\mathbf{w} = \mu + \log(1+e^{\rho})\odot\boldsymbol{\epsilon}$, $\boldsymbol{\epsilon}\sim\mathcal{N}(0,I)$, yielding the unbiased minibatch estimator:469470$$\mathcal{F} \approx \sum_{i=1}^{n}\left[\log q_\phi(\mathbf{w}^{(i)}) - \log p(\mathbf{w}^{(i)}) - \log p(\mathcal{D}|\mathbf{w}^{(i)})\right]$$471472with a scale-mixture-of-Gaussians prior. Cheaper practical alternatives: **MC Dropout** (Gal & Ghahramani, ICML 2016 — dropout at test time approximates a deep GP), **Deep Ensembles** (Lakshminarayanan et al., NeurIPS 2017 — often the strongest baseline), **SWAG**, **Laplace approximation** (Ritter et al. 2018; Daxberger et al. 2021), and **SG-MCMC** (SGLD, Welling & Teh 2011; cyclical SG-MCMC, Zhang et al. 2020).473474### 11.7 Liquid Neural Networks475476Hasani, Lechner, Amini, Rus & Grosu (2021), *Liquid Time-constant Networks*, AAAI 2021 (arXiv:2006.04439), inspired by the *C. elegans* nervous system. A continuous-time RNN whose time constant is itself **state- and input-dependent** ("liquid"):477478$$\frac{d\mathbf{x}(t)}{dt} = -\left[\frac{1}{\tau} + f\big(\mathbf{x}(t), \mathbf{I}(t), t, \theta\big)\right]\odot\mathbf{x}(t) + f\big(\mathbf{x}(t),\mathbf{I}(t),t,\theta\big)\odot A$$479480so the effective time constant is $\tau_{\text{sys}} = \dfrac{\tau}{1 + \tau f(\mathbf{x},\mathbf{I},t,\theta)}$, varying with the input — this is what makes the model adaptive after training and gives it bounded dynamics and stable state ($\mathbf{x}$ bounded between $\min(0,A)$ and $\max(0,A)$). Training uses a fused (implicit/explicit) ODE solver with BPTT. LTCs were shown to have higher **expressivity** (measured by trajectory length) than classical CT-RNNs and to be causally more robust in end-to-end drone/car navigation with as few as 19 neurons (**Neural Circuit Policies**, Lechner et al., *Nature Machine Intelligence* 2020).481482Because the LTC ODE has no known analytic solution, Hasani et al. (2022), *Closed-form Continuous-time Neural Networks*, *Nature Machine Intelligence* 4:992–1003 (arXiv:2106.13898), derived **CfC** — an approximate closed-form solution that removes the numerical solver entirely:483484$$\mathbf{x}(t) = \sigma\big(-f(\mathbf{x},\mathbf{I};\theta_f)\,t\big)\odot g(\mathbf{x},\mathbf{I};\theta_g) + \left[1 - \sigma\big(-f(\mathbf{x},\mathbf{I};\theta_f)\,t\big)\right]\odot h(\mathbf{x},\mathbf{I};\theta_h)$$485486with $\sigma$ a sigmoidal time-gate, yielding 1–5 orders of magnitude faster training and inference at comparable accuracy on irregularly sampled time series. Commercialized by **Liquid AI** (MIT spin-off, 2023), which extended the ideas to Liquid Foundation Models (LFM).487488### 11.8 World Models and JEPA489490**World Models** — Ha & Schmidhuber (2018), *World Models*, NeurIPS 2018 (arXiv:1803.10122). A three-component agent: **V** (a VAE compressing frames into $z_t$), **M** (an MDN-RNN predicting $p(z_{t+1}|a_t, z_t, h_t)$ as a mixture of Gaussians with temperature $\tau$), and **C** (a tiny linear controller $a_t = W_c[z_t\,h_t] + b_c$ evolved with CMA-ES). Crucially the agent can be trained **entirely inside its own hallucinated dream** (the "car racing in the dream" and "DoomTakeCover" experiments) and transferred back to the real environment. Successors: **PlaNet** (Hafner et al., ICML 2019), **Dreamer / DreamerV2 / DreamerV3** (Hafner et al., 2020/2021/*Nature* 2025), which learn latent dynamics with an RSSM and train an actor-critic on imagined rollouts, and **Genie** (Bruce et al., ICML 2024) for action-controllable video world models.491492**JEPA** — LeCun (2022), *A Path Towards Autonomous Machine Intelligence*, OpenReview position paper. The core critique is that generative/reconstructive objectives waste capacity on unpredictable pixel-level detail. A **Joint-Embedding Predictive Architecture** instead predicts *in representation space*: an encoder $s_x = \text{Enc}_x(x)$, a target encoder $s_y = \text{Enc}_y(y)$, and a predictor $\hat{s}_y = \text{Pred}(s_x, z)$ conditioned on a latent variable $z$ capturing the residual unpredictability. Training minimizes an **energy** $E(x,y) = \|s_y - \hat s_y\|$ with a mechanism to prevent representation collapse (asymmetric architecture + EMA target encoder, or VICReg-style variance/covariance regularization) rather than negative sampling.493494- **I-JEPA** — Assran et al. (2023), *Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture*, CVPR 2023: predict representations of several large, sufficiently-informative target blocks of an image from a single context block, using ViT encoders and an EMA target — strong semantic representations without hand-crafted data augmentations.495- **V-JEPA** (Bardes et al., 2024) and **V-JEPA 2** (Meta, 2025): feature prediction over masked spatio-temporal video patches, then action-conditioned fine-tuning for zero-shot robot planning.496- The family now spans audio (A-JEPA), point clouds, graphs, time series and scientific domains. JEPA is Meta AI's flagship bet on non-generative, planning-capable **world models** as the route to human-like machine intelligence, in explicit contrast to autoregressive LLMs.497498---499500**Sources:**501- [Kipf & Welling, GCN (TensorFlow implementation)](https://github.com/tkipf/gcn)502- [Veličković et al., Graph Attention Networks](https://www.emergentmind.com/topics/graph-attention-networks-gat)503- [Gilmer et al., Neural Message Passing for Quantum Chemistry (ICML 2017)](https://proceedings.mlr.press/v70/gilmer17a/gilmer17a.pdf)504- [Xu et al., How Powerful are Graph Neural Networks? (arXiv:1810.00826)](https://arxiv.org/pdf/2411.05464)505- [Orhan, The Leaky Integrate-and-Fire Neuron Model](https://www.cns.nyu.edu/~eorhan/notes/lif-neuron.pdf)506- [Izhikevich, Simple Model of Spiking Neurons](https://www.izhikevich.org/publications/spikes.pdf)507- [NESTML STDP windows tutorial](https://nestml.readthedocs.io/en/latest/tutorials/stdp_windows/stdp_windows.html)508- [Intel, Loihi 2 Technology Brief](https://download.intel.com/newsroom/2021/new-technologies/neuromorphic-computing-loihi-2-brief.pdf)509- [Open Neuromorphic, TrueNorth Deep Dive](https://open-neuromorphic.org/blog/truenorth-deep-dive-ibm-neuromorphic-chip-design/)510- [Hamarsheh, Self-Organizing Maps (Kohonen Maps)](https://www.philadelphia.edu.jo/academics/qhamarsheh/uploads/Lecture%2015_Self-Organizing%20Maps%20(Kohonen%20Maps).pdf)511- [Sabour, Frosst & Hinton, Dynamic Routing Between Capsules (arXiv:1710.09829)](https://arxiv.org/pdf/1710.09829)512- [Chen et al., Neural ODEs — adjoint method overview](https://arxiv.org/pdf/2209.06886)513- [Raissi et al., PINNs — comprehensive review](https://link.springer.com/article/10.1007/s10462-025-11322-7)514- [Mildenhall et al., NeRF (arXiv:2003.08934)](https://arxiv.org/pdf/2003.08934)515- [Sitzmann et al., Implicit Neural Representations with Periodic Activation Functions (arXiv:2006.09661)](https://arxiv.org/abs/2006.09661)516- [Schulman et al., PPO — implementation details](https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/)517- [AlphaZero PUCT / neural MCTS](https://arxiv.org/pdf/2101.06619)518- [Hadsell/Chopra/LeCun contrastive vs. triplet loss analysis](https://arxiv.org/pdf/2510.02161)519- [Bai, Kolter & Koltun, Deep Equilibrium Models](http://implicit-layers-tutorial.org/deep_equilibrium_models/)520- [Ha, Dai & Le, HyperNetworks (arXiv:1609.09106)](https://deepsense.ai/wp-content/uploads/2023/03/1609.09106.pdf)521- [Liu, Simonyan & Yang, DARTS](https://www.cl.cam.ac.uk/~ey204/teaching/ACS/R244_2018_2019/papers/Liu_ArXiv_2018.pdf)522- [Straight-Through Estimators overview](https://www.emergentmind.com/topics/straight-through-estimators-ste)523- [Blundell et al., Bayes by Backprop / Bayesian RNNs](https://www.gatsby.ucl.ac.uk/~ucgtcbl/papers/ForBluVin2017a.pdf)524- [Hasani et al., Liquid Time-Constant Networks (AAAI 2021)](https://cdn.aaai.org/ojs/16936/16936-13-20430-1-2-20210518.pdf)525- [Hasani et al., Closed-form Continuous-time Neural Networks (arXiv:2106.13898)](https://arxiv.org/pdf/2106.13898)526- [Ha & Schmidhuber, World Models (arXiv:1803.10122)](https://arxiv.org/abs/1803.10122)527- [Meta AI, I-JEPA](https://ai.meta.com/blog/yann-lecun-ai-model-i-jepa/)528- [Extreme Learning Machine overview](https://www.sciencedirect.com/topics/computer-science/extreme-learning-machine)529