# Specialized and Emerging Neural Network Architectures

# 1. Graph Neural Networks (GNNs)

# 1.1 The message-passing framework

Most GNNs are instances of the Message Passing Neural Network (MPNN) formalism unified by Gilmer et al. (2017), Neural Message Passing for Quantum Chemistry, ICML 2017 (arXiv:1704.01212). Given a graph $G=(V,E)$ with node features $x_v$ and edge features $e_{vw}$, the forward pass runs $T$ propagation steps:

$$m_v^{(t+1)} = \sum_{w \in \mathcal{N}(v)} M_t!\left(h_v^{(t)}, h_w^{(t)}, e_{vw}\right), \qquad h_v^{(t+1)} = U_t!\left(h_v^{(t)}, m_v^{(t+1)}\right)$$

where $M_t$ is a learned message function, $U_t$ a learned update function (Gilmer et al. used a GRU), and $\mathcal{N}(v)$ the neighborhood of $v$. A permutation-invariant readout produces a graph-level embedding:

$$\hat{y} = R\left({h_v^{(T)} \mid v \in V}\right)$$

The generic modern form separates aggregation from combination:

$$a_v^{(k)} = \operatorname{AGGREGATE}^{(k)}!\left({h_u^{(k-1)} : u \in \mathcal{N}(v)}\right), \qquad h_v^{(k)} = \operatorname{COMBINE}^{(k)}!\left(h_v^{(k-1)}, a_v^{(k)}\right)$$

The aggregator must be permutation-invariant (sum, mean, max, attention-weighted sum). This is the single equation from which GCN, GraphSAGE, GAT and GIN are all special cases.

# 1.2 Graph Convolutional Networks (GCN)

Kipf & Welling (2017), Semi-Supervised Classification with Graph Convolutional Networks, ICLR 2017 (arXiv:1609.02907), derived a first-order localized spectral filter. Starting from spectral graph convolution $g_\theta \star x = U g_\theta(\Lambda) U^\top x$ with $L = I_N - D^{-1/2}AD^{-1/2} = U\Lambda U^\top$, they applied a first-order Chebyshev truncation and a renormalization trick, yielding the celebrated layer-wise propagation rule:

$$H^{(l+1)} = \sigma!\left(\tilde{D}^{-\frac{1}{2}},\tilde{A},\tilde{D}^{-\frac{1}{2}},H^{(l)},W^{(l)}\right)$$

with $\tilde{A} = A + I_N$ (adjacency with self-loops), $\tilde{D}{ii} = \sum_j \tilde{A}{ij}$, $H^{(0)} = X$, $W^{(l)}$ the trainable weight matrix of layer $l$, and $\sigma$ typically ReLU. The renormalization trick ($A \to A+I$ before normalization) keeps the eigenvalues of the propagation operator bounded, preventing exploding/vanishing gradients in deep stacks. In node form:

$$h_v^{(l+1)} = \sigma!\left(\sum_{u \in \mathcal{N}(v)\cup{v}} \frac{1}{\sqrt{\tilde{d}_v \tilde{d}_u}} W^{(l)} h_u^{(l)}\right)$$

The symmetric normalization $D^{-1/2}\tilde{A}D^{-1/2}$ is preferred over random-walk normalization $D^{-1}\tilde{A}$ because it keeps the operator symmetric (real spectrum) and downweights messages from high-degree hubs on both endpoints. A two-layer GCN for semi-supervised classification reads $Z = \operatorname{softmax}!\left(\hat{A},\operatorname{ReLU}(\hat{A}XW^{(0)}),W^{(1)}\right)$ with $\hat{A} = \tilde{D}^{-1/2}\tilde{A}\tilde{D}^{-1/2}$.

Limitation: GCN is transductive (needs the full adjacency at training time) and suffers over-smoothing — as depth grows, node representations converge to a degree-dependent stationary vector, since repeated multiplication by $\hat{A}$ is a low-pass filter.

# 1.3 GraphSAGE

Hamilton, Ying & Leskovec (2017), Inductive Representation Learning on Large Graphs, NeurIPS 2017 (arXiv:1706.02216), made GNNs inductive by learning aggregator functions over sampled fixed-size neighborhoods rather than the full graph:

$$h_{\mathcal{N}(v)}^{(k)} = \operatorname{AGGREGATE}k!\left({h_u^{(k-1)}, \forall u \in \mathcal{N}(v)}\right)$$ $$h_v^{(k)} = \sigma!\left(W^{(k)} \cdot \operatorname{CONCAT}!\left(h_v^{(k-1)},, h{\mathcal{N}(v)}^{(k)}\right)\right), \qquad h_v^{(k)} \leftarrow \frac{h_v^{(k)}}{|h_v^{(k)}|_2}$$

The concatenation (rather than summation) of the self-vector with the neighborhood vector is a "skip connection" that preserves the node's own identity. Three aggregators were proposed:

Mean: $h_v^{(k)} = \sigma!\left(W \cdot \operatorname{mean}({h_v^{(k-1)}} \cup {h_u^{(k-1)}, u \in \mathcal{N}(v)})\right)$ — the "GCN-inductive" variant.
LSTM: apply an LSTM to a random permutation of the neighbors (not permutation-invariant, but expressive).
Max-pooling: $\operatorname{AGGREGATE}^{\text{pool}} = \max!\left({\sigma(W_{\text{pool}} h_u^{(k-1)} + b),\ \forall u \in \mathcal{N}(v)}\right)$, element-wise max.

Unsupervised training uses a graph-based loss with negative sampling: $J(z_u) = -\log!\left(\sigma(z_u^\top z_v)\right) - Q\cdot\mathbb{E}{v_n \sim P_n(v)}\log!\left(\sigma(-z_u^\top z{v_n})\right)$, where $v$ co-occurs with $u$ on a fixed-length random walk.

# 1.4 Graph Attention Networks (GAT)

Veličković et al. (2018), Graph Attention Networks, ICLR 2018 (arXiv:1710.10903), replaced the fixed structural coefficient $1/\sqrt{\tilde d_v \tilde d_u}$ by learned attention. Unnormalized attention logits:

$$e_{ij} = \operatorname{LeakyReLU}!\left(\vec{\mathbf{a}}^{\top}\left[\mathbf{W}\vec{h}_i ,|, \mathbf{W}\vec{h}_j\right]\right)$$

with $|$ concatenation, $\mathbf{W} \in \mathbb{R}^{F' \times F}$ shared, $\vec{\mathbf{a}} \in \mathbb{R}^{2F'}$ the attention vector, and LeakyReLU slope $0.2$. Normalization is a masked softmax over the first-order neighborhood (including $i$ itself):

$$\alpha_{ij} = \operatorname{softmax}j(e{ij}) = \frac{\exp(e_{ij})}{\sum_{k \in \mathcal{N}i} \exp(e{ik})}$$

Node update, and its multi-head ($K$ heads) versions:

$$\vec{h}i' = \sigma!\left(\sum{j \in \mathcal{N}i} \alpha{ij},\mathbf{W}\vec{h}j\right), \qquad \vec{h}i' = \Big|{k=1}^{K} \sigma!\left(\sum{j\in\mathcal{N}i}\alpha{ij}^k \mathbf{W}^k \vec{h}_j\right)$$

and on the final (prediction) layer, heads are averaged rather than concatenated: $\vec{h}i' = \sigma!\left(\frac{1}{K}\sum{k=1}^{K}\sum_{j\in\mathcal{N}i}\alpha{ij}^k \mathbf{W}^k \vec{h}_j\right)$.

GAT is inductive, does not require knowing the graph structure upfront, and assigns different importances to neighbors of the same node. Brody, Alon & Yahav (2022), How Attentive are Graph Attention Networks? (GATv2, ICLR 2022), showed GAT computes only static attention (the ranking of neighbors is shared across query nodes) and fixed it by moving the nonlinearity inside: $e_{ij} = \vec{\mathbf{a}}^\top \operatorname{LeakyReLU}!\left(\mathbf{W}[\vec{h}_i | \vec{h}_j]\right)$.

# 1.5 Graph Isomorphism Network (GIN)

Xu, Hu, Leskovec & Jegelka (2019), How Powerful are Graph Neural Networks?, ICLR 2019 (arXiv:1810.00826), proved that a message-passing GNN is at most as discriminative as the 1-dimensional Weisfeiler-Lehman (1-WL) test, and that this bound is attained iff the aggregation and readout functions are injective on multisets. Since sum is injective over multisets (mean and max are not), they proposed:

$$h_v^{(k)} = \operatorname{MLP}^{(k)}!\left(\left(1 + \epsilon^{(k)}\right)\cdot h_v^{(k-1)} + \sum_{u\in\mathcal{N}(v)} h_u^{(k-1)}\right)$$

with $\epsilon^{(k)}$ a learnable or fixed scalar (GIN-$\epsilon$ vs GIN-0), which disambiguates the node's own features from its neighbors' sum. For graph-level tasks, they concatenate summed readouts across all depths to preserve both local and global structure:

$$h_G = \operatorname{CONCAT}!\left(\operatorname{READOUT}\left({h_v^{(k)} \mid v \in G}\right) \ \Big|\ k = 0,1,\ldots,K\right)$$

Mean aggregation fails to distinguish multisets with identical distributions but different multiplicities; max aggregation fails to distinguish multisets with identical support. This ordering (sum > mean > max in expressive power) is the central theoretical result of the paper.

# 1.6 Applications

AlphaFold 2 (Jumper et al., Nature 2021, "Highly accurate protein structure prediction with AlphaFold"): the Evoformer treats the pair representation as a fully-connected weighted graph over residues and enforces the triangle inequality through triangular multiplicative updates and triangular self-attention — a geometric, graph-structured attention mechanism. The Structure Module then performs Invariant Point Attention (IPA) in SE(3).
Molecular property prediction: MPNN/GIN on QM9, ChEMBL; Stokes et al. (Cell 2020) discovered the antibiotic halicin with a directed-MPNN (Chemprop).
Recommender systems: PinSAGE (Ying et al., KDD 2018), a GraphSAGE variant deployed on Pinterest's 3-billion-node graph.
Physics simulation: Sanchez-Gonzalez et al. (ICML 2020), Learning to Simulate Complex Physics with Graph Networks.
Combinatorial optimization, traffic forecasting (Google Maps ETA), fraud detection, weather (GraphCast, Lam et al., Science 2023).

# 2. Spiking Neural Networks (SNNs)

SNNs are the "third generation" of neural networks (Maass, 1997, Networks of spiking neurons: The third generation of neural network models, Neural Networks 10(9)). Information is carried by discrete, asynchronous events (spikes) in continuous time rather than by real-valued activations.

# 2.1 Leaky Integrate-and-Fire (LIF)

The workhorse model, tracing back to Lapicque (1907):

$$\tau_m \frac{dV(t)}{dt} = -\left(V(t) - V_{\text{rest}}\right) + R_m I(t)$$

with $\tau_m = R_m C_m$ the membrane time constant (typically 10–30 ms in cortex). When $V(t)$ crosses threshold $V_{\text{th}}$, a spike is emitted and the potential is reset: $V \leftarrow V_{\text{reset}}$, followed by an absolute refractory period $t_{\text{ref}}$. The equivalent circuit is a leaky RC compartment. The discrete-time form used in deep SNN frameworks (snnTorch, SpikingJelly, Norse):

$$V[t] = \beta V[t-1] + \sum_j W_j S_j[t] - S[t-1],V_{\text{th}}, \qquad S[t] = \Theta!\left(V[t] - V_{\text{th}}\right)$$

with decay $\beta = e^{-\Delta t/\tau_m}$ and $\Theta$ the Heaviside step. The last term implements soft reset (subtraction) versus hard reset ($V \leftarrow 0$).

Because $\Theta'$ is a Dirac delta, gradient descent requires a surrogate gradient (Neftci, Mostafa & Zenke, 2019, Surrogate Gradient Learning in Spiking Neural Networks, IEEE Signal Processing Magazine), e.g. the fast sigmoid derivative $\frac{\partial S}{\partial V} \approx \frac{1}{(1 + \gamma|V - V_{\text{th}}|)^2}$ or an arctan/box surrogate. This enables BPTT training of deep SNNs.

# 2.2 Hodgkin–Huxley

Hodgkin & Huxley (1952), A quantitative description of membrane current and its application to conduction and excitation in nerve, J. Physiol. 117:500–544 (Nobel Prize 1963). The biophysically complete, four-dimensional model of the squid giant axon:

$$C_m \frac{dV}{dt} = I_{\text{ext}} - \bar{g}{\text{Na}} m^3 h,(V - E{\text{Na}}) - \bar{g}{\text{K}} n^4 (V - E{\text{K}}) - \bar{g}_L (V - E_L)$$

with three gating variables $m$ (Na$^+$ activation), $h$ (Na$^+$ inactivation), $n$ (K$^+$ activation), each obeying first-order kinetics:

$$\frac{dx}{dt} = \alpha_x(V)(1 - x) - \beta_x(V),x = \frac{x_\infty(V) - x}{\tau_x(V)}, \qquad x \in {m,h,n}$$

where $x_\infty = \alpha_x/(\alpha_x + \beta_x)$ and $\tau_x = 1/(\alpha_x + \beta_x)$. Typical parameters: $\bar g_{\text{Na}} = 120$, $\bar g_{\text{K}} = 36$, $\bar g_L = 0.3$ mS/cm², $C_m = 1,\mu$F/cm². HH is accurate but costs ~1200 FLOPs per 1 ms of simulation, making it impractical for large networks.

# 2.3 Izhikevich model

Izhikevich (2003), Simple Model of Spiking Neurons, IEEE Transactions on Neural Networks 14(6):1569–1572. A two-dimensional reduction (via normal-form theory of the saddle-node-on-invariant-circle bifurcation) that retains HH-like richness at ~13 FLOPs/ms:

$$\frac{dv}{dt} = 0.04v^2 + 5v + 140 - u + I, \qquad \frac{du}{dt} = a,(bv - u)$$

with the auxiliary after-spike reset:

$$\text{if } v \geq 30\ \text{mV}, \quad \text{then } \begin{cases} v \leftarrow c \ u \leftarrow u + d\end{cases}$$

$v$ is membrane potential (mV), $u$ a recovery variable (K$^+$ activation and Na$^+$ inactivation). The four parameters $(a,b,c,d)$ select the firing regime: regular spiking $(0.02, 0.2, -65, 8)$, intrinsically bursting $(0.02,0.2,-55,4)$, chattering $(0.02,0.2,-50,2)$, fast spiking $(0.1,0.2,-65,2)$, low-threshold spiking, resonator, etc. — reproducing ~20 documented cortical firing patterns.

# 2.4 STDP

Spike-Timing-Dependent Plasticity, characterized experimentally by Bi & Poo (1998), J. Neurosci. 18(24):10464–10472, and by Markram et al. (1997), Science. The classical pair-based, additive exponential window with $\Delta t = t_{\text{post}} - t_{\text{pre}}$:

$$\Delta w = \begin{cases} A_+ \exp!\left(-\dfrac{\Delta t}{\tau_+}\right) & \text{if } \Delta t > 0 \quad \text{(LTP: pre before post — causal)} \[2ex] -A_- \exp!\left(\dfrac{\Delta t}{\tau_-}\right) & \text{if } \Delta t \leq 0 \quad \text{(LTD: post before pre)} \end{cases}$$

with $\tau_+ \approx \tau_- \approx 20$ ms and $A_\pm$ the learning rates. This is a local, unsupervised, Hebbian rule ("neurons that fire together, wire together" — Hebb, 1949) that implements a causality detector. Online implementation uses pre- and post-synaptic eligibility traces $x_{\text{pre}}, x_{\text{post}}$:

$$\tau_+ \frac{dx_{\text{pre}}}{dt} = -x_{\text{pre}} + \sum_f \delta(t - t^f_{\text{pre}}), \qquad \frac{dw}{dt} = A_+ x_{\text{pre}},S_{\text{post}}(t) - A_- x_{\text{post}},S_{\text{pre}}(t)$$

Variants include multiplicative STDP (weight-dependent, $A_+ \propto (w_{\max}-w)$, guaranteeing bounded weights), triplet STDP (Pfister & Gerstner, 2006), and R-STDP / dopamine-modulated STDP where a global reward signal gates the eligibility trace, yielding a biologically plausible reinforcement learning rule.

# 2.5 Neural coding

Rate coding: information in the spike count over a window; simple, robust, but high latency and energy.
Temporal / latency coding (TTFS, time-to-first-spike): information in the precise spike time; a single spike per neuron suffices, extremely energy-efficient.
Phase coding: spike time relative to a background oscillation.
Population / rank-order coding (Thorpe et al.): information in the order in which neurons in a population fire.

# 2.6 Neuromorphic hardware

IBM TrueNorth (Merolla et al., Science 345:668–673, 2014): 4096 neurosynaptic cores, 1 million digital neurons, 256 million synapses, 5.4 billion transistors on Samsung 28 nm, running at ~65–70 mW with ~46 GSOPS/W. Fully event-driven, no clock in the neural fabric.
Intel Loihi (Davies et al., IEEE Micro 38(1):82–99, 2018): 128 neuromorphic cores + 3 x86 cores, 130,000 LIF neurons, 130 million synapses, 14 nm, with on-chip programmable learning rules (including STDP) via a microcode "learning engine". Discretized dynamics: $u_i[t] = u_i[t-1](1 - \delta^{(u)}i 2^{-12}) + 2^6\sum_j w{ij}s_j[t]$, $v_i[t] = v_i[t-1](1-\delta^{(v)}i 2^{-12}) + u_i[t] + u{\text{bias}}$.
Loihi 2 (Intel, 2021): Intel 4 (pre-production 7 nm), 31 mm², 2.3 billion transistors, up to 1 million neurons and 120 million synapses, 128 neuromorphic cores of 8192 neurons, 6 x86 cores, programmable neuron models (not just LIF) and graded spikes (spikes carry a 32-bit payload). Programmed via the open-source Lava framework. Hala Point (2024) assembles 1152 Loihi 2 chips for 1.15 billion neurons.
SpiNNaker / SpiNNaker2 (Furber et al., University of Manchester): ARM-based massively parallel simulator, 1 million cores in SpiNNaker1.
BrainScaleS (Heidelberg): analog above-threshold, 1000–10000× accelerated against biological real-time.
Others: Tianjic (Tsinghua, Nature 2019), Akida (BrainChip), DYNAP-SE (SynSense), IBM NorthPole (2023).

# 3. Self-Organizing Maps (Kohonen)

Kohonen (1982), Self-Organized Formation of Topologically Correct Feature Maps, Biological Cybernetics 43:59–69; book: Self-Organizing Maps, Springer, 1995/2001. An unsupervised, competitive-learning algorithm that projects a high-dimensional input space onto a low-dimensional (usually 2D) discrete lattice while preserving topology.

Each unit $i$ on the lattice carries a codebook (prototype) vector $m_i \in \mathbb{R}^n$. Two steps per sample $x(t)$:

(a) Competition — Best Matching Unit (BMU):

$$c = \arg\min_i |x(t) - m_i(t)| \quad \Longleftrightarrow \quad |x - m_c| = \min_i |x - m_i|$$

(b) Cooperation & adaptation:

$$m_i(t+1) = m_i(t) + \alpha(t),h_{ci}(t),\left[x(t) - m_i(t)\right]$$

with $\alpha(t) \in (0,1)$ a monotonically decreasing learning rate (e.g. $\alpha(t) = \alpha_0 e^{-t/\lambda}$) and $h_{ci}(t)$ the neighborhood kernel measured in lattice (not input) space:

$$h_{ci}(t) = \exp!\left(-\frac{|r_c - r_i|^2}{2\sigma^2(t)}\right), \qquad \sigma(t) = \sigma_0 \exp!\left(-\frac{t}{\lambda}\right)$$

or the "bubble" kernel $h_{ci} = \mathbb{1}[|r_c - r_i| \le \sigma(t)]$. The radius $\sigma(t)$ starts large (global ordering phase) and shrinks (fine-tuning / convergence phase); this annealing is what produces the topology-preserving unfolding. A batch SOM variant exists: $m_i = \frac{\sum_t h_{c(t)i},x(t)}{\sum_t h_{c(t)i}}$.

Quality is assessed with quantization error (mean $|x - m_c|$) and topographic error (fraction of samples whose first and second BMUs are non-adjacent). Applications: exploratory data visualization (U-matrix), WEBSOM document clustering, process monitoring, financial/macroprudential dashboards, gene expression clustering. SOMs are conceptually the ancestor of vector-quantization layers in VQ-VAE.

# 4. Capsule Networks

Sabour, Frosst & Hinton (2017), Dynamic Routing Between Capsules, NeurIPS 2017 (arXiv:1710.09829), building on Hinton, Krizhevsky & Wang (2011), Transforming Auto-encoders. A capsule is a group of neurons whose activity vector encodes the instantiation parameters (pose, deformation, hue, texture) of an entity, with the length of the vector encoding the probability that the entity is present. This addresses CNNs' loss of precise spatial relationships under max-pooling ("equivariance instead of invariance").

Squashing nonlinearity (vector-valued, preserves orientation, maps length into $[0,1)$):

$$\mathbf{v}_j = \frac{|\mathbf{s}_j|^2}{1 + |\mathbf{s}_j|^2},\frac{\mathbf{s}_j}{|\mathbf{s}_j|}$$

Prediction vectors ("votes") from lower capsule $i$ to higher capsule $j$ via a learned pose transformation matrix $\mathbf{W}_{ij}$:

$$\hat{\mathbf{u}}{j|i} = \mathbf{W}{ij},\mathbf{u}i, \qquad \mathbf{s}j = \sum_i c{ij},\hat{\mathbf{u}}{j|i}$$

Coupling coefficients by routing softmax over the output capsules:

$$c_{ij} = \frac{\exp(b_{ij})}{\sum_k \exp(b_{ik})}$$

Routing-by-agreement (typically $r=3$ iterations): initialize $b_{ij} \leftarrow 0$, then repeat

$$b_{ij} \leftarrow b_{ij} + \hat{\mathbf{u}}_{j|i} \cdot \mathbf{v}_j$$

i.e. the log-prior is increased when a lower capsule's vote agrees (large scalar product) with the current output of the higher capsule — a form of clustering in pose space that implements part–whole assignment.

Margin loss per class capsule $k$:

$$L_k = T_k \max(0,, m^+ - |\mathbf{v}_k|)^2 + \lambda,(1 - T_k)\max(0,, |\mathbf{v}_k| - m^-)^2$$

with $m^+ = 0.9$, $m^- = 0.1$, $\lambda = 0.5$, $T_k = 1$ iff class $k$ is present. A reconstruction decoder adds a scaled-down ($0.0005$) MSE regularizer. CapsNet reached 0.25% error on MNIST and was notably strong on overlapping digits (MultiMNIST). Follow-up: EM routing with matrix capsules (Hinton, Sabour & Frosst, ICLR 2018) and Stacked Capsule Autoencoders (Kosiorek et al., NeurIPS 2019). Capsules remain computationally expensive and have not scaled to ImageNet-class problems.

# 5. Neural Ordinary Differential Equations

Chen, Rubanova, Bettencourt & Duvenaud (2018), Neural Ordinary Differential Equations, NeurIPS 2018 Best Paper (arXiv:1806.07366). Observing that a residual block $h_{t+1} = h_t + f(h_t, \theta_t)$ is an Euler discretization, they take the continuous limit:

$$\frac{d\mathbf{h}(t)}{dt} = f!\left(\mathbf{h}(t), t, \theta\right), \qquad \mathbf{h}(t_1) = \mathbf{h}(t_0) + \int_{t_0}^{t_1} f(\mathbf{h}(t), t, \theta),dt = \operatorname{ODESolve}(\mathbf{h}(t_0), f, t_0, t_1, \theta)$$

The network becomes a continuous-depth model; the ODE solver (Dormand–Prince, adaptive Runge–Kutta) chooses the number of function evaluations, trading accuracy for compute at test time.

Adjoint sensitivity method (Pontryagin et al., 1962) gives $O(1)$ memory in depth, since no intermediate activations need storing. Define the adjoint $\mathbf{a}(t) = \partial L/\partial \mathbf{h}(t)$. It obeys a backward ODE:

$$\frac{d\mathbf{a}(t)}{dt} = -\mathbf{a}(t)^{\top}\frac{\partial f(\mathbf{h}(t), t, \theta)}{\partial \mathbf{h}}$$

and the parameter gradient is a single quadrature:

$$\frac{dL}{d\theta} = -\int_{t_1}^{t_0} \mathbf{a}(t)^{\top},\frac{\partial f(\mathbf{h}(t), t, \theta)}{\partial \theta},dt$$

In practice one integrates the augmented state $[\mathbf{h}, \mathbf{a}, \partial L/\partial\theta]$ backwards in time in a single solver call; the vector-Jacobian products $\mathbf{a}^\top \partial f/\partial \mathbf{h}$ are obtained by ordinary reverse-mode autodiff on $f$ alone.

Consequences and descendants:

Continuous Normalizing Flows / FFJORD: the instantaneous change of variables $\frac{\partial \log p(\mathbf{z}(t))}{\partial t} = -\operatorname{tr}!\left(\frac{\partial f}{\partial \mathbf{z}(t)}\right)$ replaces the $O(d^3)$ log-determinant of discrete flows by an $O(d)$ trace (Hutchinson estimator).
Latent ODEs / ODE-RNN for irregularly-sampled time series.
Augmented Neural ODEs (Dupont, Doucet & Teh, NeurIPS 2019): ODE flows are homeomorphisms and cannot cross trajectories, so some functions are unrepresentable; lifting to $[\mathbf{x}; \mathbf{a}]$ with extra dimensions fixes this.
Neural SDEs, Neural CDEs (Kidger et al., 2020), Hamiltonian/Lagrangian Neural Networks.

# 6. Physics-Informed Neural Networks (PINNs)

Raissi, Perdikaris & Karniadakis (2019), Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics 378:686–707 (preprints arXiv:1711.10561/10566). Antecedent: Lagaris, Likas & Fotiadis (1998).

Consider a PDE in general form on $\Omega \times [0,T]$:

$$u_t + \mathcal{N}[u;\lambda] = 0, \quad x \in \Omega,\ t\in[0,T]$$

Approximate $u(x,t) \approx u_\theta(x,t)$ by an MLP and define the PDE residual, computed exactly by automatic differentiation of the network with respect to its inputs:

$$r_\theta(x,t) := \frac{\partial u_\theta}{\partial t} + \mathcal{N}!\left[u_\theta; \lambda\right]$$

The composite loss is a weighted sum of mean-squared terms:

$$\mathcal{L}(\theta) = \underbrace{\lambda_r \frac{1}{N_r}\sum_{i=1}^{N_r}\left|r_\theta(x_r^i, t_r^i)\right|^2}{\text{PDE residual (collocation points)}} + \underbrace{\lambda_b \frac{1}{N_b}\sum{i=1}^{N_b}\left|u_\theta(x_b^i,t_b^i) - g^i\right|^2}{\text{boundary conditions}} + \underbrace{\lambda_0\frac{1}{N_0}\sum{i=1}^{N_0}\left|u_\theta(x_0^i,0) - u_0^i\right|^2}{\text{initial condition}} + \underbrace{\lambda_d \frac{1}{N_d}\sum{i=1}^{N_d}\left|u_\theta(x_d^i,t_d^i) - u_d^i\right|^2}_{\text{observed data}}$$

Key properties:

Mesh-free: collocation points are sampled (uniformly, Latin hypercube, or adaptively) rather than discretized on a grid; the curse of dimensionality is much milder than for finite elements.
Inverse problems come for free: the unknown physical parameters $\lambda$ (e.g. viscosity, diffusivity) are simply additional trainable variables optimized jointly with $\theta$ against sparse data.
Training: typically Adam followed by L-BFGS; tanh activations (need smooth higher derivatives).

Known pathologies: severe loss-term imbalance (gradient pathologies, Wang, Teng & Perdikaris, SIAM J. Sci. Comput. 2021 — fixed by learning-rate annealing or NTK-based weighting), spectral bias against high-frequency solutions, and failure on stiff/convection-dominated regimes ("failure modes", Krishnapriyan et al., NeurIPS 2021). Remedies include hard-constrained boundary conditions ($u_\theta = g + B(x),\text{NN}(x)$), domain decomposition (XPINN, cPINN), causal training weights, and Fourier feature embeddings. Extensions: DeepONet (Lu, Jin & Karniadakis, Nature Machine Intelligence 2021) and Fourier Neural Operator (Li et al., ICLR 2021), which learn operators between function spaces rather than single solutions.

# 7. Neural Radiance Fields (NeRF)

Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi & Ng (2020), NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis, ECCV 2020 Best Paper Honorable Mention (arXiv:2003.08934). A scene is a single MLP:

$$F_\Theta : (\mathbf{x}, \mathbf{d}) = (x,y,z,\theta,\phi) \longmapsto (\mathbf{c}, \sigma) = (r,g,b,\sigma)$$

with volume density $\sigma$ predicted from position only (enforcing multi-view consistent geometry) and view-dependent color $\mathbf{c}$ from position and viewing direction (enabling specularities).

Volume rendering along a camera ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$, between near and far bounds $[t_n, t_f]$ (classical emission-absorption model, Kajiya & Von Herzen 1984):

$$C(\mathbf{r}) = \int_{t_n}^{t_f} T(t),\sigma(\mathbf{r}(t)),\mathbf{c}(\mathbf{r}(t), \mathbf{d}),dt, \qquad T(t) = \exp!\left(-\int_{t_n}^{t}\sigma(\mathbf{r}(s)),ds\right)$$

where $T(t)$ is the accumulated transmittance (probability the ray travels from $t_n$ to $t$ without being absorbed). The quadrature approximation over $N$ stratified samples with intervals $\delta_i = t_{i+1} - t_i$:

$$\hat{C}(\mathbf{r}) = \sum_{i=1}^{N} T_i,\alpha_i,\mathbf{c}i, \qquad \alpha_i = 1 - \exp(-\sigma_i\delta_i), \qquad T_i = \exp!\left(-\sum{j=1}^{i-1}\sigma_j\delta_j\right) = \prod_{j=1}^{i-1}(1-\alpha_j)$$

This is exactly alpha compositing and is fully differentiable, so the whole pipeline trains by photometric loss alone: $\mathcal{L} = \sum_{\mathbf{r}\in\mathcal{R}} \left[|\hat{C}_c(\mathbf{r}) - C(\mathbf{r})|_2^2 + |\hat{C}_f(\mathbf{r}) - C(\mathbf{r})|_2^2\right]$ over the coarse and fine networks (hierarchical sampling: the coarse network's weights $w_i = T_i\alpha_i$ define a PDF used for inverse-transform sampling of the fine network).

Positional encoding is essential: MLPs exhibit spectral bias toward low frequencies, so raw $(x,y,z)$ inputs give blurry reconstructions. NeRF maps each scalar coordinate through:

$$\gamma(p) = \left(\sin(2^0\pi p), \cos(2^0 \pi p), \sin(2^1 \pi p), \cos(2^1\pi p), \ldots, \sin(2^{L-1}\pi p), \cos(2^{L-1}\pi p)\right)$$

with $L=10$ for $\mathbf{x}$ (60 dims) and $L=4$ for $\mathbf{d}$ (24 dims). Tancik et al. (NeurIPS 2020), Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, explained this via NTK: the encoding converts the NTK into a stationary, tunable-bandwidth kernel.

Descendants: Mip-NeRF (anti-aliasing via integrated positional encoding of conical frustums), Instant-NGP (Müller et al., SIGGRAPH 2022 — multiresolution hash encoding, seconds instead of days), Plenoxels, NeRF in the Wild, Zip-NeRF, and the successor paradigm 3D Gaussian Splatting (Kerbl et al., SIGGRAPH 2023), which replaces the MLP with explicit anisotropic Gaussians and rasterization for real-time rendering.

# 8. Implicit Neural Representations and SIREN

An implicit neural representation (INR) encodes a signal as a continuous function $\Phi_\theta : \mathbb{R}^n \to \mathbb{R}^m$ parameterized by network weights, rather than as a discrete grid — resolution-independent, memory scaling with signal complexity rather than resolution. Examples: DeepSDF (Park et al., CVPR 2019, signed distance functions), Occupancy Networks (Mescheder et al., CVPR 2019), NeRF.

SIREN — Sitzmann, Martel, Bergman, Lindell & Wetzstein (2020), Implicit Neural Representations with Periodic Activation Functions, NeurIPS 2020 Oral (arXiv:2006.09661). Replace ReLU by a sine:

$$\Phi(\mathbf{x}) = \mathbf{W}n\left(\phi{n-1}\circ\phi_{n-2}\circ\cdots\circ\phi_0\right)(\mathbf{x}) + \mathbf{b}_n, \qquad \phi_i(\mathbf{x}_i) = \sin!\left(\omega_0,\mathbf{W}_i\mathbf{x}_i + \mathbf{b}_i\right)$$

Crucially, the derivative of a SIREN is itself a SIREN (since $\frac{d}{dx}\sin(x) = \cos(x) = \sin(x + \pi/2)$), so all higher-order derivatives are well-behaved and nonzero — unlike ReLU networks, whose second derivative vanishes everywhere. This lets SIRENs be supervised directly on derivative constraints: solving the eikonal equation $|\nabla_\mathbf{x}\Phi| = 1$ for SDFs, the Poisson equation from gradients/Laplacians only, or the Helmholtz/wave equations.

Principled initialization is required to keep activation distributions stable across depth: draw $w_i \sim \mathcal{U}!\left(-\sqrt{6/\text{fan_in}},, \sqrt{6/\text{fan_in}}\right)$, so that the pre-activations are normally distributed and the post-activations are arcsine-distributed, with $\omega_0 = 30$ for the first layer to span the input frequency spectrum.

Related: Fourier Feature Networks (Tancik et al. 2020), WIRE (Gabor wavelet activations, CVPR 2023), BACON, MFN (multiplicative filter networks), and functa/hypernetwork-conditioned INRs for generative modeling over signals.

# 9. Networks for Deep Reinforcement Learning

# 9.1 Deep Q-Networks (DQN)

Mnih et al. (2015), Human-level control through deep reinforcement learning, Nature 518:529–533 (NIPS workshop version 2013). The optimal action-value function obeys the Bellman optimality equation:

$$Q^(s,a) = \mathbb{E}{s'\sim\mathcal{E}}!\left[r + \gamma \max{a'} Q^(s',a') ,\Big|, s,a\right]$$

DQN approximates $Q^*(s,a) \approx Q(s,a;\theta)$ with a CNN over raw pixels and minimizes the TD loss:

$$L_i(\theta_i) = \mathbb{E}{(s,a,r,s')\sim U(\mathcal{D})}\left[\left(\underbrace{r + \gamma\max{a'}Q(s',a';\theta^-)}_{\text{TD target } y} - Q(s,a;\theta_i)\right)^2\right]$$

Two stabilizing innovations: (i) experience replay — sample uniformly from a buffer $\mathcal{D}$ to break temporal correlations; (ii) a target network with frozen parameters $\theta^-$, synchronized every $C$ steps ($\theta^- \leftarrow \theta$), which prevents the target from chasing the prediction. Gradient: $\nabla_{\theta_i}L_i = \mathbb{E}\left[(y - Q(s,a;\theta_i))\nabla_{\theta_i}Q(s,a;\theta_i)\right]$; Huber loss is used in practice for robustness.

Extensions, consolidated in Rainbow (Hessel et al., AAAI 2018): Double DQN (van Hasselt et al., AAAI 2016 — decouple selection from evaluation, $y = r + \gamma Q(s', \arg\max_{a'}Q(s',a';\theta);\theta^-)$, curing overestimation bias); Dueling networks (Wang et al., ICML 2016 — $Q(s,a) = V(s) + A(s,a) - \frac{1}{|\mathcal{A}|}\sum_{a'}A(s,a')$); Prioritized Experience Replay (Schaul et al., ICLR 2016 — $p_i \propto |\delta_i|^\alpha$ with importance-sampling correction); Noisy Nets; distributional RL / C51 (Bellemare et al., ICML 2017); n-step returns.

# 9.2 Policy gradients and REINFORCE

The policy gradient theorem (Sutton, McAllester, Singh & Mansour, NeurIPS 2000) for $J(\theta) = \mathbb{E}{\tau\sim\pi\theta}[R(\tau)]$:

$$\nabla_\theta J(\theta) = \mathbb{E}{\tau\sim\pi\theta}\left[\sum_{t=0}^{T}\nabla_\theta \log \pi_\theta(a_t|s_t), \Psi_t\right]$$

REINFORCE (Williams, 1992, Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine Learning 8:229–256) takes $\Psi_t = G_t = \sum_{k=t}^{T}\gamma^{k-t}r_k$, giving the update $\theta \leftarrow \theta + \alpha,\gamma^t G_t \nabla_\theta \log\pi_\theta(a_t|s_t)$. It is unbiased but has very high variance; subtracting a state-dependent baseline $b(s_t)$ leaves it unbiased (since $\mathbb{E}[\nabla_\theta \log\pi_\theta(a|s),b(s)] = 0$) and reduces variance:

$$\nabla_\theta J = \mathbb{E}\left[\sum_t \nabla_\theta \log\pi_\theta(a_t|s_t)\left(G_t - b(s_t)\right)\right]$$

# 9.3 Actor-Critic and A3C

Setting $b(s) = V^\pi_\phi(s)$ and $\Psi_t = A^\pi(s_t,a_t) = Q^\pi(s_t,a_t) - V^\pi(s_t)$ gives advantage actor-critic. With the $n$-step estimator $\hat{A}t = \sum{i=0}^{n-1}\gamma^i r_{t+i} + \gamma^n V_\phi(s_{t+n}) - V_\phi(s_t)$, the combined objective is:

$$\mathcal{L} = -\log\pi_\theta(a_t|s_t),\hat{A}t + c_v\left(V\phi(s_t) - G_t\right)^2 - c_e,H!\left(\pi_\theta(\cdot|s_t)\right)$$

where $H$ is the policy entropy, encouraging exploration (typically $c_e = 0.01$, $c_v = 0.5$).

A3C — Mnih et al. (2016), Asynchronous Methods for Deep Reinforcement Learning, ICML 2016. Multiple CPU actor-learners explore in parallel and apply Hogwild!-style asynchronous updates to shared parameters; the decorrelation induced by parallel exploration replaces the replay buffer, allowing on-policy learning and training on CPUs alone. A2C is the synchronous, batched variant that dominates in practice on GPUs.

GAE (Schulman et al., ICLR 2016) interpolates bias/variance: $\hat{A}t^{\text{GAE}(\gamma,\lambda)} = \sum{l=0}^{\infty}(\gamma\lambda)^l\delta_{t+l}$ with $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$.

# 9.4 TRPO and PPO

TRPO (Schulman et al., ICML 2015) maximizes a surrogate under a KL trust region: $\max_\theta \mathbb{E}\left[\frac{\pi_\theta(a|s)}{\pi_{\theta_{\text{old}}}(a|s)}\hat A\right]$ s.t. $\mathbb{E}\left[D_{\text{KL}}(\pi_{\theta_{\text{old}}}|\pi_\theta)\right] \le \delta$, requiring conjugate gradients and Fisher-vector products.

PPO — Schulman, Wolski, Dhariwal, Radford & Klimov (2017), Proximal Policy Optimization Algorithms (arXiv:1707.06347) — achieves the same effect first-order. With the probability ratio $r_t(\theta) = \dfrac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$, the clipped surrogate objective is:

$$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t\left[\min\Big(r_t(\theta),\hat{A}_t,\ \operatorname{clip}\big(r_t(\theta),,1-\epsilon,,1+\epsilon\big),\hat{A}_t\Big)\right]$$

typically $\epsilon = 0.2$. The $\min$ makes the objective a pessimistic lower bound: when $\hat A_t > 0$ the gain is capped at $1+\epsilon$; when $\hat A_t < 0$ the loss is capped at $1-\epsilon$; but the unclipped branch is retained when it is worse, so a policy that has moved too far in the wrong direction is still pulled back. The full loss combines value and entropy terms:

$$L^{\text{CLIP+VF+S}}(\theta) = \hat{\mathbb{E}}_t\left[L^{\text{CLIP}}_t(\theta) - c_1 L^{\text{VF}}_t(\theta) + c_2,S\pi_\theta\right]$$

PPO allows multiple epochs of minibatch SGD on the same rollout, is simple and robust, and became the default for continuous control, Dota 2 (OpenAI Five), and RLHF for large language models (Ouyang et al., NeurIPS 2022, InstructGPT), where the reward is a learned preference model and a per-token KL penalty to the reference policy is added. An alternative penalized form is $L^{\text{KLPEN}} = \hat{\mathbb{E}}\left[r_t(\theta)\hat A_t - \beta,\text{KL}[\pi_{\theta_{\text{old}}}, \pi_\theta]\right]$ with adaptive $\beta$.

Off-policy actor-critics complete the picture: DDPG (Lillicrap et al., ICLR 2016), TD3 (Fujimoto et al., ICML 2018), SAC (Haarnoja et al., ICML 2018) with the maximum-entropy objective $J(\pi) = \sum_t \mathbb{E}\left[r_t + \alpha H(\pi(\cdot|s_t))\right]$.

# 9.5 AlphaGo / AlphaZero / MuZero

AlphaGo (Silver et al., Nature 529:484–489, 2016): SL policy network trained on human expert moves, an RL policy network refined by self-play, a fast rollout policy, and a value network, all combined in MCTS. Defeated Lee Sedol 4–1 in 2016.
AlphaGo Zero (Silver et al., Nature 550:354–359, 2017) and AlphaZero (Silver et al., Science 362:1140–1144, 2018): a single residual CNN $(\mathbf{p}, v) = f_\theta(s)$ outputs a move-prior vector and a scalar value, trained tabula rasa purely from self-play, no human data, generalized to chess and shogi.

MCTS selection uses a PUCT rule (Rosin 2011, adapted):

$$a_t = \arg\max_a\left(Q(s_t,a) + U(s_t,a)\right), \qquad U(s,a) = c_{\text{puct}},P(s,a),\frac{\sqrt{\sum_b N(s,b)}}{1 + N(s,a)}$$

where $P(s,a)$ is the network prior, $N(s,a)$ the visit count and $Q(s,a) = \frac{1}{N(s,a)}\sum_{s'|s,a\to s'} V(s')$ the mean value. Search acts as a policy improvement operator: the visit-count distribution $\pi_a \propto N(s,a)^{1/\tau}$ is stronger than the raw network prior, and the network is then trained to imitate it:

$$\ell = (z - v)^2 - \boldsymbol{\pi}^{\top}\log \mathbf{p} + c|\theta|^2$$

with $z \in {-1,0,+1}$ the self-play game outcome. Dirichlet noise is added at the root ($P(s,a) = (1-\varepsilon)p_a + \varepsilon\eta_a$, $\eta \sim \text{Dir}(0.03)$) for exploration.

MuZero (Schrittwieser et al., Nature 588:604–609, 2020) removes the need for a known simulator by learning a representation function $h$, dynamics function $g$ and prediction function $f$ in a latent space, trained only to predict reward, value and policy — a learned model for planning.

# 10. Siamese Networks and Metric Learning

Introduced by Bromley, Guyon, LeCun, Säckinger & Shah (1993), Signature Verification using a "Siamese" Time Delay Neural Network, NeurIPS 1993. Two (or more) identical towers with shared weights map inputs into an embedding space $f_\theta: \mathcal{X}\to\mathbb{R}^d$ where semantic similarity becomes geometric distance. This enables one-shot / few-shot recognition and open-set verification, where the class set is not fixed at training time.

Contrastive loss — Hadsell, Chopra & LeCun (2006), Dimensionality Reduction by Learning an Invariant Mapping, CVPR 2006 (see also Chopra, Hadsell & LeCun, CVPR 2005). With $Y=0$ for a similar pair and $Y=1$ for dissimilar, and $D_W = |f_\theta(x_1) - f_\theta(x_2)|_2$:

$$\mathcal{L}(W, Y, x_1, x_2) = (1-Y),\frac{1}{2},D_W^2 ;+; Y,\frac{1}{2}\left{\max\left(0,, m - D_W\right)\right}^2$$

Similar pairs are pulled together with a quadratic spring; dissimilar pairs are pushed apart only until they reach margin $m$, after which they contribute no gradient (otherwise all negatives would be pushed to infinity).

Triplet loss — Schroff, Kalenichenko & Philbin (2015), FaceNet: A Unified Embedding for Face Recognition and Clustering, CVPR 2015; earlier ranking form in Weinberger & Saul (LMNN, JMLR 2009). Given an anchor $a$, positive $p$ (same class) and negative $n$, with embeddings constrained to the unit hypersphere $|f(x)|_2 = 1$:

$$\mathcal{L}{\text{triplet}} = \sum{i}^{N}\left[\left|f(x_i^a) - f(x_i^p)\right|_2^2 - \left|f(x_i^a) - f(x_i^n)\right|2^2 + \alpha\right]{+}$$

with $[\cdot]_+ = \max(0,\cdot)$ and margin $\alpha$ (0.2 in FaceNet). The loss is relative rather than absolute: it only demands the negative be farther than the positive by $\alpha$, which is a weaker and better-behaved constraint than contrastive loss's absolute distance targets.

Mining is critical: random triplets are quickly satisfied and give zero gradient. FaceNet uses semi-hard online mining within a large minibatch (~1800): negatives with $|f_a - f_p|^2 < |f_a - f_n|^2 < |f_a - f_p|^2 + \alpha$ — harder than the positive but still inside the margin. Selecting the hardest negatives causes collapse to $f(x) = 0$.

Extensions: Koch, Zemel & Salakhutdinov (ICML Deep Learning Workshop 2015) on one-shot Omniglot with binary cross-entropy over the L1 distance; N-pair loss (Sohn, NeurIPS 2016); Lifted structured loss; angular margin softmax losses — SphereFace (A-Softmax), CosFace, and ArcFace (Deng et al., CVPR 2019) with $\mathcal{L} = -\log\frac{e^{s\cos(\theta_{y_i}+m)}}{e^{s\cos(\theta_{y_i}+m)} + \sum_{j\ne y_i}e^{s\cos\theta_j}}$; and the InfoNCE loss (Oord et al. 2018) at the heart of SimCLR/MoCo/CLIP: $\mathcal{L} = -\log\frac{\exp(\text{sim}(z_i,z_j)/\tau)}{\sum_{k\ne i}\exp(\text{sim}(z_i,z_k)/\tau)}$. Non-contrastive siamese methods (BYOL, Grill et al. NeurIPS 2020; SimSiam, Chen & He CVPR 2021; Barlow Twins; VICReg) avoid negatives entirely via stop-gradient, predictor heads, or redundancy-reduction terms.

# 11. Other Specialized and Emerging Architectures

# 11.1 Extreme Learning Machines (ELM)

Huang, Zhu & Siew (2006), Extreme learning machine: Theory and applications, Neurocomputing 70(1–3):489–501 (ICNN 2004 conference version). For a single-hidden-layer feedforward network with $L$ hidden nodes, randomly assign input weights $\mathbf{w}i$ and biases $b_i$ and never tune them; only the output weights $\boldsymbol{\beta}$ are learned. The hidden-layer output matrix is $H{ij} = g(\mathbf{w}_j\cdot\mathbf{x}_i + b_j)$, and training reduces to the linear least-squares problem $\mathbf{H}\boldsymbol{\beta} = \mathbf{T}$, whose minimum-norm solution is:

$$\hat{\boldsymbol{\beta}} = \mathbf{H}^{\dagger}\mathbf{T}$$

with $\mathbf{H}^\dagger$ the Moore–Penrose pseudoinverse; the regularized version is $\hat{\boldsymbol{\beta}} = \left(\mathbf{H}^\top\mathbf{H} + \frac{\mathbf{I}}{C}\right)^{-1}\mathbf{H}^\top\mathbf{T}$. Training is non-iterative and orders of magnitude faster than backpropagation, with universal approximation guarantees. Criticism (notably by Wang & Li, 2017) is that ELM largely restates Schmidt et al. (1992) and the Random Vector Functional Link (Pao et al., 1994). The idea is the feedforward cousin of reservoir computing — Echo State Networks (Jaeger, 2001) and Liquid State Machines (Maass, Natschläger & Markram, 2002) — where a fixed random recurrent reservoir is read out linearly.

# 11.2 Deep Equilibrium Models (DEQ)

Bai, Kolter & Koltun (2019), Deep Equilibrium Models, NeurIPS 2019 (arXiv:1909.01377). Instead of stacking $L$ layers, directly solve for the fixed point of a single weight-tied layer, equivalent to an infinite-depth network:

$$\mathbf{z}^\star = f_\theta(\mathbf{z}^\star; \mathbf{x})$$

found with a black-box root solver (Broyden's method, Anderson acceleration) on $g_\theta(\mathbf{z}) = f_\theta(\mathbf{z};\mathbf{x}) - \mathbf{z} = 0$. Backpropagation uses the implicit function theorem, not the solver's trajectory:

$$\frac{\partial \ell}{\partial(\cdot)} = \frac{\partial \ell}{\partial \mathbf{z}^\star}\left(I - J_{f_\theta}(\mathbf{z}^\star)\right)^{-1}\frac{\partial f_\theta(\mathbf{z}^\star;\mathbf{x})}{\partial(\cdot)}, \qquad J_{f_\theta}(\mathbf{z}^\star) = \left.\frac{\partial f_\theta}{\partial \mathbf{z}}\right|_{\mathbf{z}^\star}$$

The inverse-Jacobian-vector product is itself computed by solving a linear fixed-point system, so memory is $O(1)$ in depth (up to 88% reduction reported), independent of the number of solver iterations. Concerns are ill-conditioning and non-existence/instability of the fixed point, addressed by Jacobian regularization (Bai, Koltun et al., NeurIPS 2021, penalizing $|J|_F$ via Hutchinson estimation) and monotone operator formulations (monDEQ, Winston & Kolter, NeurIPS 2020). DEQ is the discrete-implicit sibling of Neural ODEs; both belong to the family of implicit layers (see also OptNet, differentiable convex optimization layers).

# 11.3 HyperNetworks

Ha, Dai & Le (2017), HyperNetworks, ICLR 2017 (arXiv:1609.09106). A small network $g_\psi$ generates the weights of a larger primary network:

$$\theta^{(l)} = g_\psi!\left(\mathbf{e}^{(l)}\right)$$

where $\mathbf{e}^{(l)}$ is a learned per-layer embedding. In the static case (CNNs) this is a form of relaxed weight sharing and drastic compression; in the dynamic case (recurrent HyperLSTM) the hypernetwork is itself recurrent and produces input-dependent, time-varying weights — a mechanism closely related to fast weights (Schmidhuber, 1992) and to linear attention. Modern descendants: hypernetwork-conditioned INRs, HyperMorph for medical registration, task-conditioned hypernetworks for continual learning and meta-learning, and the weight-generation view of LoRA-style adapters.

# 11.4 Neural Architecture Search (NAS)

RL-based: Zoph & Le (2017), Neural Architecture Search with Reinforcement Learning, ICLR 2017 — an RNN controller samples architecture descriptions and is trained with REINFORCE on validation accuracy $R$: $\nabla_{\theta_c}J = \sum_t \mathbb{E}\left[\nabla_{\theta_c}\log P(a_t|a_{(t-1):1};\theta_c)(R - b)\right]$. Cost: ~800 GPUs for weeks. NASNet (Zoph et al., CVPR 2018) searched transferable cells rather than whole networks.
Evolutionary: Real et al. (AAAI 2019), Regularized Evolution for Image Classifier Architecture Search (AmoebaNet), with age-based tournament selection.
Differentiable: Liu, Simonyan & Yang (2019), DARTS: Differentiable Architecture Search, ICLR 2019. Relax the categorical choice of operation on edge $(i,j)$ into a softmax mixture over the operation set $\mathcal{O}$:

$$\bar{o}^{(i,j)}(x) = \sum_{o\in\mathcal{O}}\frac{\exp\left(\alpha_o^{(i,j)}\right)}{\sum_{o'\in\mathcal{O}}\exp\left(\alpha_{o'}^{(i,j)}\right)},o(x)$$

and solve the bilevel problem $\min_\alpha \mathcal{L}{\text{val}}(w^(\alpha), \alpha)$ s.t. $w^(\alpha) = \arg\min_w \mathcal{L}{\text{train}}(w,\alpha)$ with a one-step (second-order or first-order) approximation. Search cost drops to ~1 GPU-day. Discretization keeps $o^{(i,j)} = \arg\max_o \alpha^{(i,j)}_o$. Known failure mode: collapse to parameter-free skip connections; fixes include DARTS+, PC-DARTS, Fair DARTS, and early stopping on the Hessian eigenvalue.

One-shot / weight sharing: ENAS, Once-for-All (Cai et al., ICLR 2020), BigNAS; zero-cost proxies and NAS benchmarks (NAS-Bench-101/201/301). Practical outcomes: EfficientNet (compound scaling), MnasNet, MobileNetV3.

# 11.5 Binarized and Quantized Neural Networks

BinaryConnect (Courbariaux, Bengio & David, NeurIPS 2015), BinaryNet/BNN (Hubara, Courbariaux et al., NeurIPS 2016), XNOR-Net (Rastegari et al., ECCV 2016). Weights and/or activations are constrained to ${-1,+1}$:

$$x^b = \operatorname{sign}(x) = \begin{cases} +1 & x \ge 0 \ -1 & x < 0\end{cases}$$

so that the dot product collapses to XNOR + popcount, giving up to 32× memory compression and ~58× speedup on CPU. Since $\partial,\text{sign}/\partial x = 0$ a.e., training uses the straight-through estimator (Hinton, 2012; Bengio, Léonard & Courville, 2013) with a clipping/hard-tanh window:

$$\frac{\partial \mathcal{L}}{\partial x} \approx \frac{\partial\mathcal{L}}{\partial x^b}\cdot\mathbb{1}_{|x|\le 1}$$

while a latent real-valued shadow copy of the weights accumulates the small gradient updates. XNOR-Net adds per-channel scaling factors $\alpha = \frac{1}{n}|W|_{\ell 1}$ to reduce the quantization error $|W - \alpha B|^2$. Modern relatives: quantization-aware training (Jacob et al., CVPR 2018), LSQ (learned step size), post-training quantization (GPTQ, AWQ), and 1-bit LLMs (BitNet b1.58, Ma et al. 2024, with ternary ${-1,0,1}$ weights).

# 11.6 Bayesian Neural Networks

Foundations: MacKay (1992, evidence framework), Neal (1995, HMC and the NN–GP correspondence). BNNs place a prior $p(\mathbf{w})$ over weights and seek the posterior $p(\mathbf{w}|\mathcal{D}) = \frac{p(\mathcal{D}|\mathbf{w})p(\mathbf{w})}{p(\mathcal{D})}$, giving calibrated epistemic uncertainty; prediction marginalizes: $p(y^|x^,\mathcal{D}) = \int p(y^|x^,\mathbf{w}),p(\mathbf{w}|\mathcal{D}),d\mathbf{w}$.

The posterior is intractable, so variational inference fits $q_\phi(\mathbf{w})$ (typically fully-factorized Gaussian, $\mathbf{w}\sim\mathcal{N}(\mu, \sigma^2)$) by minimizing the variational free energy / maximizing the ELBO:

$$\mathcal{F}(\phi) = D_{\text{KL}}!\left(q_\phi(\mathbf{w}),|,p(\mathbf{w})\right) - \mathbb{E}{q\phi(\mathbf{w})}\left[\log p(\mathcal{D}|\mathbf{w})\right] = -\text{ELBO}$$

Bayes by Backprop — Blundell, Cornebise, Kavukcuoglu & Wierstra (2015), Weight Uncertainty in Neural Networks, ICML 2015 — makes this trainable by the reparameterization trick $\mathbf{w} = \mu + \log(1+e^{\rho})\odot\boldsymbol{\epsilon}$, $\boldsymbol{\epsilon}\sim\mathcal{N}(0,I)$, yielding the unbiased minibatch estimator:

$$\mathcal{F} \approx \sum_{i=1}^{n}\left[\log q_\phi(\mathbf{w}^{(i)}) - \log p(\mathbf{w}^{(i)}) - \log p(\mathcal{D}|\mathbf{w}^{(i)})\right]$$

with a scale-mixture-of-Gaussians prior. Cheaper practical alternatives: MC Dropout (Gal & Ghahramani, ICML 2016 — dropout at test time approximates a deep GP), Deep Ensembles (Lakshminarayanan et al., NeurIPS 2017 — often the strongest baseline), SWAG, Laplace approximation (Ritter et al. 2018; Daxberger et al. 2021), and SG-MCMC (SGLD, Welling & Teh 2011; cyclical SG-MCMC, Zhang et al. 2020).

# 11.7 Liquid Neural Networks

Hasani, Lechner, Amini, Rus & Grosu (2021), Liquid Time-constant Networks, AAAI 2021 (arXiv:2006.04439), inspired by the C. elegans nervous system. A continuous-time RNN whose time constant is itself state- and input-dependent ("liquid"):

$$\frac{d\mathbf{x}(t)}{dt} = -\left[\frac{1}{\tau} + f\big(\mathbf{x}(t), \mathbf{I}(t), t, \theta\big)\right]\odot\mathbf{x}(t) + f\big(\mathbf{x}(t),\mathbf{I}(t),t,\theta\big)\odot A$$

so the effective time constant is $\tau_{\text{sys}} = \dfrac{\tau}{1 + \tau f(\mathbf{x},\mathbf{I},t,\theta)}$, varying with the input — this is what makes the model adaptive after training and gives it bounded dynamics and stable state ($\mathbf{x}$ bounded between $\min(0,A)$ and $\max(0,A)$). Training uses a fused (implicit/explicit) ODE solver with BPTT. LTCs were shown to have higher expressivity (measured by trajectory length) than classical CT-RNNs and to be causally more robust in end-to-end drone/car navigation with as few as 19 neurons (Neural Circuit Policies, Lechner et al., Nature Machine Intelligence 2020).

Because the LTC ODE has no known analytic solution, Hasani et al. (2022), Closed-form Continuous-time Neural Networks, Nature Machine Intelligence 4:992–1003 (arXiv:2106.13898), derived CfC — an approximate closed-form solution that removes the numerical solver entirely:

$$\mathbf{x}(t) = \sigma\big(-f(\mathbf{x},\mathbf{I};\theta_f),t\big)\odot g(\mathbf{x},\mathbf{I};\theta_g) + \left[1 - \sigma\big(-f(\mathbf{x},\mathbf{I};\theta_f),t\big)\right]\odot h(\mathbf{x},\mathbf{I};\theta_h)$$

with $\sigma$ a sigmoidal time-gate, yielding 1–5 orders of magnitude faster training and inference at comparable accuracy on irregularly sampled time series. Commercialized by Liquid AI (MIT spin-off, 2023), which extended the ideas to Liquid Foundation Models (LFM).

# 11.8 World Models and JEPA

World Models — Ha & Schmidhuber (2018), World Models, NeurIPS 2018 (arXiv:1803.10122). A three-component agent: V (a VAE compressing frames into $z_t$), M (an MDN-RNN predicting $p(z_{t+1}|a_t, z_t, h_t)$ as a mixture of Gaussians with temperature $\tau$), and C (a tiny linear controller $a_t = W_c[z_t,h_t] + b_c$ evolved with CMA-ES). Crucially the agent can be trained entirely inside its own hallucinated dream (the "car racing in the dream" and "DoomTakeCover" experiments) and transferred back to the real environment. Successors: PlaNet (Hafner et al., ICML 2019), Dreamer / DreamerV2 / DreamerV3 (Hafner et al., 2020/2021/Nature 2025), which learn latent dynamics with an RSSM and train an actor-critic on imagined rollouts, and Genie (Bruce et al., ICML 2024) for action-controllable video world models.

JEPA — LeCun (2022), A Path Towards Autonomous Machine Intelligence, OpenReview position paper. The core critique is that generative/reconstructive objectives waste capacity on unpredictable pixel-level detail. A Joint-Embedding Predictive Architecture instead predicts in representation space: an encoder $s_x = \text{Enc}_x(x)$, a target encoder $s_y = \text{Enc}_y(y)$, and a predictor $\hat{s}_y = \text{Pred}(s_x, z)$ conditioned on a latent variable $z$ capturing the residual unpredictability. Training minimizes an energy $E(x,y) = |s_y - \hat s_y|$ with a mechanism to prevent representation collapse (asymmetric architecture + EMA target encoder, or VICReg-style variance/covariance regularization) rather than negative sampling.

I-JEPA — Assran et al. (2023), Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, CVPR 2023: predict representations of several large, sufficiently-informative target blocks of an image from a single context block, using ViT encoders and an EMA target — strong semantic representations without hand-crafted data augmentations.
V-JEPA (Bardes et al., 2024) and V-JEPA 2 (Meta, 2025): feature prediction over masked spatio-temporal video patches, then action-conditioned fine-tuning for zero-shot robot planning.
The family now spans audio (A-JEPA), point clouds, graphs, time series and scientific domains. JEPA is Meta AI's flagship bet on non-generative, planning-capable world models as the route to human-like machine intelligence, in explicit contrast to autoregressive LLMs.

Sources: