spb/artificial-neural-networks-book Public
Artificial Neural Networks — Methods, Equations and Graphical Representations: a complete book, every method with rigorous equations, pseudocode and native TikZ figures.
TeX 100%
1% ============================================================================2% Artificial Neural Networks — Methods, Equations and Graphical3% Representations4% Author : Simon-Pierre Boucher — contact@spboucher.ai5% Chapter 13 : Emerging Architectures (chapters/13-emerging.tex)6% ============================================================================7\chapter{Emerging Architectures: Capsules, Neural ODEs, Memory-Augmented8 Networks and KANs}\label{chap:emerging}910The architectures of the preceding chapters share a common template: fixed11computational graphs whose edges carry scalar weights and whose nodes apply12fixed nonlinearities. This final chapter examines four families that each13break the template along a different axis. \emph{Capsule networks} replace14scalar neurons by vector-valued units routed by iterative agreement.15\emph{Neural ordinary differential equations} replace discrete layers by a16continuous-time flow whose depth is chosen by a numerical solver.17\emph{Memory-augmented networks} couple a neural controller to an external,18differentiable random-access memory. \emph{Kolmogorov--Arnold networks}19move the learnable nonlinearity from the nodes onto the edges themselves.20In every case we follow the discipline of this book: the defining21equations, the estimation algorithm, and a faithful graphical22representation.2324% ============================================================================25\section{Capsule Networks and Routing by Agreement}26\label{sec:em-capsules}2728Convolutional networks achieve tolerance to small translations through29pooling, but in doing so they discard the precise spatial relationships30between parts — a face detector built on max-pooling responds to a jumble31of eyes and mouth almost as strongly as to a correctly arranged face.32Capsule networks \cite{sabour2017} address this by making the network's33elementary unit a \emph{vector} rather than a scalar.3435\begin{definition}[Capsule]\label{def:em-capsule}36A \emph{capsule} is a group of neurons whose activity vector37$\vect{u} \in \R^{d}$ encodes the instantiation parameters (pose,38deformation, texture) of an entity, and whose length39$\lVert\vect{u}\rVert \in [0,1)$ encodes the probability that the entity40is present in the input.41\end{definition}4243\subsection{Prediction, coupling and squashing}4445Let $\vect{u}_i$ be the output of capsule $i$ in layer $\ell$. Capsule $i$46casts a \emph{vote} for the pose of each capsule $j$ in layer $\ell+1$47through a learned transformation matrix $\mat{W}_{ij}$, and the votes are48aggregated with data-dependent \emph{coupling coefficients} $c_{ij}$:49\begin{align}50 \hat{\vect{u}}_{j|i} &= \mat{W}_{ij}\,\vect{u}_i,51 \label{eq:em-caps-pred}\\52 \vect{s}_j &= \sum_i c_{ij}\,\hat{\vect{u}}_{j|i}.53 \label{eq:em-caps-sum}54\end{align}55The couplings are a softmax over \emph{routing logits} $b_{ij}$,56initialized to zero, so that each lower capsule distributes a unit budget57of attention across the capsules of the next layer:58\begin{equation}59 c_{ij} \;=\; \frac{\exp(b_{ij})}{\sum_k \exp(b_{ik})}.60 \label{eq:em-caps-coupling}61\end{equation}62The total input $\vect{s}_j$ is converted to an output vector by the63\emph{squashing} nonlinearity, which preserves orientation while mapping64length into $[0,1)$ so it can act as a probability:65\begin{equation}66 \vect{v}_j \;=\;67 \frac{\lVert\vect{s}_j\rVert^2}{1+\lVert\vect{s}_j\rVert^2}\,68 \frac{\vect{s}_j}{\lVert\vect{s}_j\rVert}.69 \label{eq:em-caps-squash}70\end{equation}7172\subsection{Dynamic routing as an estimation algorithm}7374The logits are refined by \emph{routing by agreement}: a vote that aligns75with the emerging consensus $\vect{v}_j$ has its coupling reinforced,76\begin{equation}77 b_{ij} \;\leftarrow\; b_{ij} + \hat{\vect{u}}_{j|i} \cdot \vect{v}_j,78 \label{eq:em-caps-agree}79\end{equation}80which is a soft clustering in pose space: parts whose predicted poses81agree are assigned to the same whole. Algorithm~\ref{alg:em-routing}82summarizes the procedure of \cite{sabour2017}, typically run for $r=3$83iterations between consecutive capsule layers.8485\begin{algorithm}[t]86\caption{Dynamic routing by agreement \cite{sabour2017}}87\label{alg:em-routing}88\begin{algorithmic}[1]89\Require prediction vectors $\hat{\vect{u}}_{j|i}$ from90 \eqref{eq:em-caps-pred}, number of iterations $r$91\Ensure output capsules $\vect{v}_j$92\State $b_{ij} \gets 0$ for all $i, j$93\For{$t = 1, \dots, r$}94 \State $c_{ij} \gets \softmax_j(b_{ij})$95 \Comment{coupling coefficients, \eqref{eq:em-caps-coupling}}96 \State $\vect{s}_j \gets \sum_i c_{ij}\,\hat{\vect{u}}_{j|i}$97 \Comment{weighted vote aggregation, \eqref{eq:em-caps-sum}}98 \State $\vect{v}_j \gets99 \dfrac{\lVert\vect{s}_j\rVert^2}{1+\lVert\vect{s}_j\rVert^2}\,100 \dfrac{\vect{s}_j}{\lVert\vect{s}_j\rVert}$101 \Comment{squash, \eqref{eq:em-caps-squash}}102 \State $b_{ij} \gets b_{ij} + \hat{\vect{u}}_{j|i}\cdot\vect{v}_j$103 \Comment{agreement update, \eqref{eq:em-caps-agree}}104\EndFor105\State \Return $\vect{v}_j$106\end{algorithmic}107\end{algorithm}108109Figure~\ref{fig:em-capsules} shows two capsule layers after routing has110converged: the width and opacity of each arrow is proportional to the111coupling coefficient $c_{ij}$, so the diagram makes visible the112part--whole assignment that \eqref{eq:em-caps-agree} computes.113114\begin{figure}[t]115 \centering116 \begin{tikzpicture}[scale=1.0]117 % primary capsules118 \node[nhidden, minimum size=9mm] (u1) at (0, 1.7) {$\vect{u}_1$};119 \node[nhidden, minimum size=9mm] (u2) at (0, 0.0) {$\vect{u}_2$};120 \node[nhidden, minimum size=9mm] (u3) at (0,-1.7) {$\vect{u}_3$};121 % output capsules122 \node[noutput, minimum size=10mm] (v1) at (5.6, 0.9) {$\vect{v}_1$};123 \node[noutput, minimum size=10mm] (v2) at (5.6,-0.9) {$\vect{v}_2$};124 % couplings: widths proportional to c_ij125 \draw[fleche, line width=1.9pt, cinput!90!black]126 (u1) -- (v1)127 node[pos=0.40, above=1pt, sloped, font=\scriptsize, text=black]128 {$\hat{\vect{u}}_{1|1}$};129 \draw[fleche, line width=0.5pt, cinput!90!black, opacity=0.30]130 (u1) -- (v2);131 \draw[fleche, line width=1.5pt, cinput!90!black, opacity=0.85]132 (u2) -- (v1);133 \draw[fleche, line width=0.7pt, cinput!90!black, opacity=0.40]134 (u2) -- (v2);135 \draw[fleche, line width=0.5pt, cinput!90!black, opacity=0.30]136 (u3) -- (v1);137 \draw[fleche, line width=1.9pt, cinput!90!black]138 (u3) -- (v2);139 % annotations140 \node[etiquette, align=center] at (0, 2.85)141 {primary capsules\\(layer $\ell$)};142 \node[etiquette, align=center] at (5.6, 2.05)143 {output capsules\\(layer $\ell+1$)};144 \node[etiquette, align=left, anchor=west] at (6.55, 0.9)145 {$\lVert\vect{v}_1\rVert \approx 1$:\\entity present};146 \node[etiquette, align=left, anchor=west] at (6.55,-0.9)147 {$\lVert\vect{v}_2\rVert \approx 1$:\\entity present};148 \node[etiquette, align=center] at (2.8,-2.75)149 {arrow width $\propto$ coupling coefficient $c_{ij}$150 of \eqref{eq:em-caps-coupling}};151 \end{tikzpicture}152 \caption{Two capsule layers after routing by agreement153 (Algorithm~\ref{alg:em-routing}). Each primary capsule sends a154 prediction vector $\hat{\vect{u}}_{j|i} = \mat{W}_{ij}\vect{u}_i$ to155 every output capsule; iterating156 \eqref{eq:em-caps-coupling}--\eqref{eq:em-caps-agree} concentrates the157 couplings (arrow width) on the wholes whose pose the parts agree on:158 here $\vect{u}_1, \vect{u}_2$ are routed to $\vect{v}_1$ and159 $\vect{u}_3$ to $\vect{v}_2$.}160 \label{fig:em-capsules}161\end{figure}162163\subsection{Margin loss}164165Since capsule lengths are probabilities, classification uses a per-class166\emph{margin loss}: with $T_k = 1$ if class $k$ is present and $0$167otherwise,168\begin{equation}169 L_k \;=\; T_k \max\bigl(0,\, m^{+} - \lVert\vect{v}_k\rVert\bigr)^2170 \;+\; \lambda\,(1-T_k)\max\bigl(0,\,171 \lVert\vect{v}_k\rVert - m^{-}\bigr)^2,172 \label{eq:em-caps-margin}173\end{equation}174with $m^{+}=0.9$, $m^{-}=0.1$ and $\lambda = 0.5$ in \cite{sabour2017};175the total loss $\Loss = \sum_k L_k$ is complemented by a small176reconstruction penalty that regularizes the capsule poses.177178\begin{remark}179Routing replaces pooling: instead of discarding position information, the180network performs an explicit, differentiable assignment of parts to181wholes. The price is computational — routing is an inner-loop iteration182per layer and per example — and capsule networks have not yet scaled183beyond mid-sized benchmarks, but the equivariance principle they embody184continues to influence architecture design.185\end{remark}186187% ============================================================================188\section{Neural Ordinary Differential Equations}189\label{sec:em-node}190191\subsection{From residual blocks to continuous depth}192193A residual block computes194$\vect{h}_{t+1} = \vect{h}_t + f(\vect{h}_t, \theta_t)$. Reading the layer195index $t$ as time, this is exactly one step of the explicit Euler scheme196with unit step size. Neural ODEs \cite{chen2018} take the continuous197limit: the hidden state is defined as the solution of an initial value198problem,199\begin{equation}200 \frac{d\vect{h}(t)}{dt} = f\bigl(\vect{h}(t), t, \theta\bigr),201 \qquad202 \vect{h}(t_1) = \vect{h}(t_0)203 + \int_{t_0}^{t_1} f\bigl(\vect{h}(t), t, \theta\bigr)\,dt,204 \label{eq:em-node-dynamics}205\end{equation}206evaluated by a black-box solver207$\vect{h}(t_1) = \mathrm{ODESolve}(\vect{h}(t_0), f, t_0, t_1, \theta)$208such as an adaptive Runge--Kutta method.209210\begin{remark}[ResNet as Euler discretization]\label{rem:em-euler}211With step size $\Delta t$, the Euler scheme applied to212\eqref{eq:em-node-dynamics} reads213\begin{equation}214 \vect{h}_{t+1} \;=\; \vect{h}_t215 + \Delta t\, f(\vect{h}_t, t, \theta),216 \label{eq:em-node-euler}217\end{equation}218which for $\Delta t = 1$ is precisely the residual block of a ResNet. A219neural ODE is thus a residual network whose number of layers — the number220of solver steps — is chosen adaptively at run time, trading accuracy221against compute without retraining.222\end{remark}223224\subsection{The adjoint method}225226Naively backpropagating through every internal solver step would store all227intermediate states. The \emph{adjoint sensitivity method} avoids this.228Define the adjoint state as the sensitivity of the loss to the hidden229state at each instant,230\begin{equation}231 \vect{a}(t) \;=\; \frac{\partial \Loss}{\partial \vect{h}(t)}.232 \label{eq:em-node-adjoint}233\end{equation}234The adjoint obeys its own linear ODE, integrated \emph{backwards} in time235from $\vect{a}(t_1) = \partial\Loss/\partial\vect{h}(t_1)$:236\begin{equation}237 \frac{d\vect{a}(t)}{dt}238 \;=\; -\,\vect{a}(t)\transp\,239 \frac{\partial f\bigl(\vect{h}(t), t, \theta\bigr)}{\partial \vect{h}},240 \label{eq:em-node-adjoint-ode}241\end{equation}242and the parameter gradient is obtained by a single quadrature along the243same backward pass:244\begin{equation}245 \frac{d\Loss}{d\theta}246 \;=\; -\int_{t_1}^{t_0} \vect{a}(t)\transp\,247 \frac{\partial f\bigl(\vect{h}(t), t, \theta\bigr)}{\partial \theta}248 \,dt.249 \label{eq:em-node-grad}250\end{equation}251In practice one concatenates the state, the adjoint and the accumulating252gradient into a single \emph{augmented state} and makes one backward253solver call:254\begin{equation}255 \frac{d}{dt}256 \begin{bmatrix} \vect{h}(t)\\[2pt] \vect{a}(t)\\[2pt]257 \dfrac{d\Loss}{d\theta}(t) \end{bmatrix}258 =259 \begin{bmatrix} f(\vect{h}, t, \theta)\\[2pt]260 -\vect{a}\transp\,\partial f/\partial\vect{h}\\[2pt]261 -\vect{a}\transp\,\partial f/\partial\theta \end{bmatrix},262 \label{eq:em-node-augmented}263\end{equation}264where the vector--Jacobian products are computed by ordinary reverse-mode265automatic differentiation of $f$ alone.266Algorithm~\ref{alg:em-adjoint} assembles the full gradient computation.267268\begin{property}[Constant memory in depth]\label{prop:em-memory}269Training with270\eqref{eq:em-node-adjoint-ode}--\eqref{eq:em-node-augmented} requires271storing only the endpoint states, so the memory cost of a neural ODE is272$O(1)$ in the effective depth, whereas backpropagation through an273$L$-layer residual network stores $O(L)$ activations. The hidden274trajectory $\vect{h}(t)$ needed inside \eqref{eq:em-node-adjoint-ode} is275recovered on the fly by integrating \eqref{eq:em-node-dynamics}276backwards alongside the adjoint.277\end{property}278279\begin{algorithm}[t]280\caption{Adjoint-method gradient for a neural ODE \cite{chen2018}}281\label{alg:em-adjoint}282\begin{algorithmic}[1]283\Require dynamics $f_\theta$, initial state $\vect{h}(t_0)$, loss $\Loss$284\State $\vect{h}(t_1) \gets285 \mathrm{ODESolve}\bigl(\vect{h}(t_0), f, t_0, t_1, \theta\bigr)$286 \Comment{forward solve of \eqref{eq:em-node-dynamics}}287\State $\vect{a}(t_1) \gets \partial\Loss/\partial\vect{h}(t_1)$288 \Comment{terminal adjoint, \eqref{eq:em-node-adjoint}}289\State $\vect{s}(t_1) \gets290 \bigl[\vect{h}(t_1),\ \vect{a}(t_1),\ \vect{0}\bigr]$291 \Comment{augmented state}292\State $\bigl[\vect{h}(t_0), \vect{a}(t_0), d\Loss/d\theta\bigr] \gets293 \mathrm{ODESolve}\bigl(\vect{s}(t_1),294 \text{dynamics \eqref{eq:em-node-augmented}}, t_1, t_0\bigr)$295 \Comment{backward solve}296\State \Return $d\Loss/d\theta$ \ and \297 $\vect{a}(t_0) = \partial\Loss/\partial\vect{h}(t_0)$298\end{algorithmic}299\end{algorithm}300301Figure~\ref{fig:em-node} contrasts the two views of depth: the discrete302Euler/ResNet staircase of \eqref{eq:em-node-euler} against the smooth flow303of \eqref{eq:em-node-dynamics}, drawn over the vector field304$f(h) = 0.8\,h\,(1-h/3)$ that both are following.305306\begin{figure}[t]307 \centering308 \begin{tikzpicture}309 \begin{axis}[310 width=0.82\textwidth, height=6.6cm,311 xlabel={$t$ (depth)}, ylabel={$h(t)$},312 xmin=-0.15, xmax=4.45, ymin=0, ymax=3.15,313 legend style={at={(0.97,0.06)}, anchor=south east,314 font=\scriptsize, draw=black!30},315 tick label style={font=\scriptsize},316 label style={font=\small},317 ]318 % faint vector field of the dynamics f(h)=0.8 h (1-h/3)319 \foreach \t in {0.1,0.6,1.1,1.6,2.1,2.6,3.1,3.6,4.1}{320 \foreach \h in {0.35,0.75,1.15,1.55,1.95,2.35,2.75}{321 \edef\temp{\noexpand\draw[-{Stealth[length=1.0mm]},322 black!25, thin]323 (axis cs:\t,\h) --324 (axis cs:{\t+0.16},{\h+0.16*0.8*\h*(1-\h/3)});}325 \temp326 }327 }328 % continuous ODE trajectory (logistic solution)329 \addplot[cinput, very thick, smooth, domain=0:4.3, samples=80]330 {3/(1+5*exp(-0.8*x))};331 \addlegendentry{continuous flow, \eqref{eq:em-node-dynamics}}332 % Euler / ResNet steps, dt = 1333 \addplot[chidden!80!black, thick, dashed,334 mark=*, mark options={solid, fill=chidden}]335 coordinates {(0,0.5) (1,0.8333) (2,1.3148) (3,1.9056)336 (4,2.4617)};337 \addlegendentry{Euler / ResNet steps, $\Delta t = 1$,338 \eqref{eq:em-node-euler}}339 \end{axis}340 \end{tikzpicture}341 \caption{A neural ODE as the continuous limit of a residual network.342 Grey arrows: the learned vector field $f(h,t,\theta)$. Blue: the343 exact trajectory of \eqref{eq:em-node-dynamics}. Orange: the Euler344 discretization \eqref{eq:em-node-euler} with $\Delta t = 1$, i.e.\ a345 four-block residual network following the same field with346 accumulating discretization error. An adaptive solver places as many347 evaluations as the requested tolerance demands.}348 \label{fig:em-node}349\end{figure}350351% ============================================================================352\section{Memory-Augmented Networks: Neural Turing Machines}353\label{sec:em-ntm}354355Recurrent networks store all their knowledge in a fixed-size hidden state.356Neural Turing Machines \cite{graves2014} decouple computation from357storage: a controller network (typically an LSTM) interacts with an358external memory matrix $\mat{M}_t \in \R^{N\times W}$ ($N$ locations of359width $W$) through read and write \emph{heads} that are differentiable360end-to-end, so the whole system is trained by ordinary gradient descent.361362\subsection{Reading and writing}363364Every interaction is mediated by a normalized attention weighting365$\vect{w}_t \in \R^N$, $\sum_i w_t(i) = 1$, over memory locations. Reading366returns the weighted average of the rows,367\begin{equation}368 \vect{r}_t \;=\; \sum_{i=1}^{N} w_t(i)\,\mat{M}_t(i),369 \label{eq:em-ntm-read}370\end{equation}371and writing decomposes, like an LSTM gate pair, into an erase followed by372an add, with $\vect{e}_t \in [0,1]^W$ and $\vect{a}_t \in \R^W$ emitted by373the controller:374\begin{align}375 \tilde{\mat{M}}_t(i) &= \mat{M}_{t-1}(i) \odot376 \bigl[\vect{1} - w_t(i)\,\vect{e}_t\bigr],377 \label{eq:em-ntm-erase}\\378 \mat{M}_t(i) &= \tilde{\mat{M}}_t(i) + w_t(i)\,\vect{a}_t.379 \label{eq:em-ntm-add}380\end{align}381382\subsection{The differentiable addressing pipeline}383384The weighting $\vect{w}_t$ is produced by a four-stage pipeline that385blends content-based and location-based addressing. First, the controller386emits a key $\vect{k}_t$ compared to every row by cosine similarity,387\begin{equation}388 K[\vect{u},\vect{v}] \;=\;389 \frac{\vect{u}\cdot\vect{v}}{\lVert\vect{u}\rVert\,390 \lVert\vect{v}\rVert},391 \label{eq:em-ntm-cosine}392\end{equation}393sharpened by a scalar $\beta_t > 0$ and normalized:394\begin{equation}395 w_t^{c}(i) \;=\;396 \frac{\exp\bigl(\beta_t\,K[\vect{k}_t, \mat{M}_t(i)]\bigr)}397 {\sum_j \exp\bigl(\beta_t\,K[\vect{k}_t, \mat{M}_t(j)]\bigr)}.398 \label{eq:em-ntm-content}399\end{equation}400The content weighting is then interpolated with the previous weighting by401a gate $g_t \in [0,1]$,402\begin{equation}403 \vect{w}_t^{g} \;=\; g_t\,\vect{w}_t^{c}404 + (1-g_t)\,\vect{w}_{t-1},405 \label{eq:em-ntm-interp}406\end{equation}407rotated by a circular convolution with a shift distribution $\vect{s}_t$408(location-based addressing, enabling ``move one slot to the right''),409\begin{equation}410 \tilde{w}_t(i) \;=\; \sum_{j=0}^{N-1} w_t^{g}(j)\,411 s_t\bigl((i-j) \bmod N\bigr),412 \label{eq:em-ntm-shift}413\end{equation}414and finally re-sharpened with $\gamma_t \geq 1$ to undo the blurring415introduced by the convolution:416\begin{equation}417 w_t(i) \;=\;418 \frac{\tilde{w}_t(i)^{\gamma_t}}{\sum_j \tilde{w}_t(j)^{\gamma_t}}.419 \label{eq:em-ntm-sharpen}420\end{equation}421Algorithm~\ref{alg:em-ntm} chains422\eqref{eq:em-ntm-cosine}--\eqref{eq:em-ntm-sharpen} with the memory update423\eqref{eq:em-ntm-erase}--\eqref{eq:em-ntm-add}; every operation is smooth424in its inputs, so gradients flow from the task loss into the controller,425the heads and the memory itself. The complete system is drawn in426Figure~\ref{fig:em-ntm}.427428\begin{algorithm}[t]429\caption{One NTM head at timestep $t$ \cite{graves2014}}430\label{alg:em-ntm}431\begin{algorithmic}[1]432\Require controller outputs $\vect{k}_t, \beta_t, g_t, \vect{s}_t,433 \gamma_t$ (and $\vect{e}_t, \vect{a}_t$ for a write head),434 previous weighting $\vect{w}_{t-1}$, memory $\mat{M}_{t-1}$435\State $w_t^c(i) \gets \softmax_i\bigl(\beta_t\,436 K[\vect{k}_t,\mat{M}_{t-1}(i)]\bigr)$437 \Comment{content addressing,438 \eqref{eq:em-ntm-cosine}--\eqref{eq:em-ntm-content}}439\State $\vect{w}_t^{g} \gets g_t \vect{w}_t^{c}440 + (1-g_t)\,\vect{w}_{t-1}$441 \Comment{interpolation, \eqref{eq:em-ntm-interp}}442\State $\tilde{w}_t(i) \gets \sum_j w^{g}_t(j)\,s_t((i-j)\bmod N)$443 \Comment{circular shift, \eqref{eq:em-ntm-shift}}444\State $w_t(i) \gets \tilde{w}_t(i)^{\gamma_t} \big/445 \sum_j \tilde{w}_t(j)^{\gamma_t}$446 \Comment{sharpening, \eqref{eq:em-ntm-sharpen}}447\If{write head}448 \State apply erase then add,449 \eqref{eq:em-ntm-erase}--\eqref{eq:em-ntm-add}450\Else451 \State \Return read vector452 $\vect{r}_t = \sum_i w_t(i)\,\mat{M}_t(i)$,453 \eqref{eq:em-ntm-read}454\EndIf455\end{algorithmic}456\end{algorithm}457458\begin{figure}[t]459 \centering460 \begin{tikzpicture}[scale=1.0]461 % controller462 \node[blochidden, minimum width=2.5cm, minimum height=1.5cm,463 align=center]464 (ctrl) at (0,0) {Controller\\(LSTM)};465 \draw[fleche] (-2.6,-0.45) node[left, font=\small] {$\vect{x}_t$}466 -- (-1.25,-0.45);467 \draw[fleche] (-1.25, 0.45)468 -- (-2.6, 0.45) node[left, font=\small] {$\vect{y}_t$};469 % heads470 \node[gate, minimum width=1.9cm] (wh) at (3.7, 1.5)471 {write head};472 \node[gate, minimum width=1.9cm] (rh) at (3.7,-1.5)473 {read head};474 \draw[fleche] (1.25, 0.45) -| (wh.south);475 \draw[fleche] (1.25,-0.45) -| (rh.north);476 \node[etiquette, anchor=south, align=center] at (1.85, 0.55)477 {$\vect{k}_t, \beta_t, g_t, \vect{s}_t,$\\478 $\gamma_t, \vect{e}_t, \vect{a}_t$};479 \node[etiquette, anchor=north] at (2.15,-0.55)480 {$\vect{k}_t, \beta_t, g_t, \vect{s}_t, \gamma_t$};481 % memory grid: 4 rows (locations N) x 5 columns (width W)482 \begin{scope}[shift={(6.3,-1.24)}]483 % highlighted row (addressed): row index 2 (third from bottom)484 \fill[cmem!35] (0, 1.24) rectangle (3.10, 1.86);485 \foreach \r in {0,1,2,3}{486 \foreach \c in {0,1,2,3,4}{487 \draw[black!60] (\c*0.62, \r*0.62)488 rectangle ++(0.62,0.62);489 }490 }491 \node[etiquette] at (1.55, 2.85)492 {memory $\mat{M}_t \in \R^{N\times W}$};493 \node[etiquette, anchor=west] at (3.22, 1.55) {$w_t$};494 \end{scope}495 % head <-> memory arrows (enter the addressed row's left edge)496 \draw[fleche, cgate!60!black] (wh.east)497 -- (5.9, 1.5) -- (5.9, 0.48) -- (6.28, 0.48);498 \draw[fleche, cgate!60!black] (rh.east)499 -- (5.9, -1.5) -- (5.9, 0.14) -- (6.28, 0.14);500 \node[etiquette, anchor=south, align=center] at (5.45, 1.58)501 {erase, add\\502 \eqref{eq:em-ntm-erase}--\eqref{eq:em-ntm-add}};503 \node[etiquette, anchor=north] at (5.42, -1.62)504 {address $w_t$};505 % read vector returned to controller (dashed)506 \draw[flechep, cmem!70!black]507 (rh.south) |- (0,-2.75)508 node[pos=0.75, above, etiquette] {read vector $\vect{r}_t$,509 \eqref{eq:em-ntm-read}}510 -- (0,-0.75);511 \end{tikzpicture}512 \caption{The Neural Turing Machine. The controller emits addressing513 parameters for each head; the write head modifies the memory by erase514 \eqref{eq:em-ntm-erase} then add \eqref{eq:em-ntm-add}, the read head515 returns the attention-weighted content of the addressed row (violet),516 and the read vector feeds back into the controller at the next step517 (dashed). Every operation is differentiable, so the system trains518 end-to-end by gradient descent.}519 \label{fig:em-ntm}520\end{figure}521522\begin{remark}523The Differentiable Neural Computer refines the NTM with dynamic slot524allocation via usage vectors and a temporal link matrix that lets read525heads replay memory in the order it was written. Although superseded in526practice by Transformers — whose attention \emph{is} a form of527content-based addressing \eqref{eq:em-ntm-content} over an internal528memory of past tokens — the NTM/DNC line established the memory-augmented529paradigm that today's retrieval-augmented models inherit.530\end{remark}531532% ============================================================================533\section{Kolmogorov--Arnold Networks}534\label{sec:em-kan}535536\subsection{The representation theorem}537538\begin{theorem}[Kolmogorov--Arnold superposition]\label{thm:em-kart}539Every continuous function $f : [0,1]^n \to \R$ can be written as540\begin{equation}541 f(x_1, \dots, x_n) \;=\;542 \sum_{q=1}^{2n+1} \Phi_q\!\left(543 \sum_{p=1}^{n} \phi_{q,p}(x_p)\right),544 \label{eq:em-kan-kart}545\end{equation}546where the $\Phi_q : \R \to \R$ and $\phi_{q,p} : [0,1] \to \R$ are547continuous \emph{univariate} functions.548\end{theorem}549550Multivariate continuity thus reduces entirely to sums and compositions of551one-dimensional functions — the only truly multivariate operation in552\eqref{eq:em-kan-kart} is addition.553554\subsection{KAN layers}555556Kolmogorov--Arnold networks \cite{liu2024kan} turn this structure into an557architecture. Where an MLP layer computes558$\sigma(\mat{W}\vect{x} + \vect{b})$ — fixed nonlinearities on the nodes,559learnable scalars on the edges — a KAN layer places a \emph{learnable560univariate function on every edge} and reduces nodes to pure summation:561\begin{equation}562 x_{\ell+1,\,j} \;=\; \sum_{i=1}^{n_\ell}563 \phi_{\ell,j,i}\bigl(x_{\ell,\,i}\bigr),564 \label{eq:em-kan-layer}565\end{equation}566and a full network is a composition of such function matrices567$\Phi_\ell = \bigl(\phi_{\ell,j,i}\bigr)_{j,i}$:568\begin{equation}569 \mathrm{KAN}(\vect{x}) \;=\;570 \bigl(\Phi_{L-1} \circ \cdots \circ \Phi_1 \circ571 \Phi_0\bigr)(\vect{x}).572 \label{eq:em-kan-compose}573\end{equation}574Each edge function is parameterized as a B-spline plus a smooth residual575basis that keeps gradients well-behaved:576\begin{equation}577 \phi(x) \;=\; w_b\,\mathrm{silu}(x)578 + w_s \sum_{i} c_i\,B_i(x),579 \label{eq:em-kan-spline}580\end{equation}581with learnable spline coefficients $c_i$ over a grid of knots. The582authors of \cite{liu2024kan} stress that583\eqref{eq:em-kan-layer}--\eqref{eq:em-kan-compose} generalize the exact584depth-2, width-$(2n{+}1)$ form of Theorem~\ref{thm:em-kart} to arbitrary585depths and widths. Figure~\ref{fig:em-kan} draws a small KAN with the586learned univariate function displayed on each edge.587588\begin{figure}[t]589 \centering590 \begin{tikzpicture}[scale=1.0]591 % nodes592 \node[ninput] (x1) at (0, 1.0) {$x_1$};593 \node[ninput] (x2) at (0,-1.0) {$x_2$};594 \node[op, minimum size=7mm] (m1) at (3.4, 1.8) {$+$};595 \node[op, minimum size=7mm] (m2) at (3.4, 0.0) {$+$};596 \node[op, minimum size=7mm] (m3) at (3.4,-1.8) {$+$};597 \node[noutput, minimum size=9mm] (y) at (6.8, 0.0) {$y$};598 % edges first (below boxes)599 \draw[fleche, black!55] (x1) -- (m1);600 \draw[fleche, black!55] (x1) -- (m2);601 \draw[fleche, black!55] (x1) -- (m3);602 \draw[fleche, black!55] (x2) -- (m1);603 \draw[fleche, black!55] (x2) -- (m2);604 \draw[fleche, black!55] (x2) -- (m3);605 \draw[fleche, black!55] (m1) -- (y);606 \draw[fleche, black!55] (m2) -- (y);607 \draw[fleche, black!55] (m3) -- (y);608 % mini function boxes on edges (drawn after edges to sit on top)609 % straight-ish edges: boxes at pos ~0.4 ; crossing edges: pos ~0.72610 \node[draw=black!60, fill=white, minimum width=8mm,611 minimum height=5.5mm, inner sep=0.5pt] at (1.36, 1.32) {};612 \draw[cinput, thick]613 plot[domain=-0.30:0.30, samples=19]614 ({1.36+\x}, {1.32+0.14*sin(600*\x)});615 \node[draw=black!60, fill=white, minimum width=8mm,616 minimum height=5.5mm, inner sep=0.5pt] at (1.36, 0.60) {};617 \draw[cinput, thick]618 plot[domain=-0.30:0.30, samples=19]619 ({1.36+\x}, {0.60+1.1*\x*\x-0.09});620 \node[draw=black!60, fill=white, minimum width=8mm,621 minimum height=5.5mm, inner sep=0.5pt] at (2.448,-1.016) {};622 \draw[cinput, thick]623 plot[domain=-0.30:0.30, samples=19]624 ({2.448+\x}, {-1.016+0.13*tanh(6*\x)});625 \node[draw=black!60, fill=white, minimum width=8mm,626 minimum height=5.5mm, inner sep=0.5pt] at (2.448, 1.016) {};627 \draw[cinput, thick]628 plot[domain=-0.30:0.30, samples=19]629 ({2.448+\x}, {1.016-0.13*tanh(6*\x)});630 \node[draw=black!60, fill=white, minimum width=8mm,631 minimum height=5.5mm, inner sep=0.5pt] at (1.36,-0.60) {};632 \draw[cinput, thick]633 plot[domain=-0.30:0.30, samples=19]634 ({1.36+\x}, {-0.60+0.14*sin(300*\x)});635 \node[draw=black!60, fill=white, minimum width=8mm,636 minimum height=5.5mm, inner sep=0.5pt] at (1.36,-1.32) {};637 \draw[cinput, thick]638 plot[domain=-0.30:0.30, samples=19]639 ({1.36+\x}, {-1.32-1.1*\x*\x+0.09});640 % layer 2 boxes641 \node[draw=black!60, fill=white, minimum width=8mm,642 minimum height=5.5mm, inner sep=0.5pt] at (5.1, 0.90) {};643 \draw[coutput, thick]644 plot[domain=-0.30:0.30, samples=19]645 ({5.1+\x}, {0.90+0.13*tanh(6*\x)});646 \node[draw=black!60, fill=white, minimum width=8mm,647 minimum height=5.5mm, inner sep=0.5pt] at (5.1, 0.0) {};648 \draw[coutput, thick]649 plot[domain=-0.30:0.30, samples=19]650 ({5.1+\x}, {0.0+0.14*sin(600*\x)});651 \node[draw=black!60, fill=white, minimum width=8mm,652 minimum height=5.5mm, inner sep=0.5pt] at (5.1,-0.90) {};653 \draw[coutput, thick]654 plot[domain=-0.30:0.30, samples=19]655 ({5.1+\x}, {-0.90+1.1*\x*\x-0.09});656 % annotations657 \node[etiquette, align=center] at (0,-2.6)658 {inputs};659 \node[etiquette, align=center] at (3.4,-2.6)660 {summation nodes \eqref{eq:em-kan-layer}};661 \node[etiquette, align=center] at (6.8,-2.6)662 {output};663 \node[etiquette, align=center] at (3.4, 2.75)664 {a learnable univariate $\phi_{\ell,j,i}$665 \eqref{eq:em-kan-spline} on \emph{every} edge};666 \end{tikzpicture}667 \caption{A small Kolmogorov--Arnold network with $n_0 = 2$ inputs,668 $n_1 = 3$ hidden summation nodes and one output. Each edge carries669 its own learnable univariate function (inset curves), parameterized670 as a B-spline \eqref{eq:em-kan-spline}; the nodes only add their671 incoming values, exactly as in the superposition672 \eqref{eq:em-kan-kart}.}673 \label{fig:em-kan}674\end{figure}675676\subsection{KANs versus MLPs}677678Table~\ref{tab:em-kan-mlp} summarizes the structural contrast. KANs trade679the hardware-friendliness of dense matrix multiplication for680interpretability: a trained spline can be plotted, pruned, and often681symbolically identified ($\sin$, $x^2$, $\exp$), which has made KANs682attractive for scientific and symbolic-regression tasks.683684\begin{table}[t]685 \centering686 \caption{Multilayer perceptrons versus Kolmogorov--Arnold networks.}687 \label{tab:em-kan-mlp}688 \begin{tabular}{@{}lll@{}}689 \toprule690 & MLP & KAN \\691 \midrule692 Nonlinearity & fixed, on nodes & learnable, on edges \\693 Edge parameters & scalar weights $w_{ij}$ &694 spline functions $\phi_{j,i}$ \\695 Node operation & $\sigma(\mat{W}\vect{x}+\vect{b})$ &696 summation \eqref{eq:em-kan-layer} \\697 Theoretical anchor & universal approximation &698 superposition \eqref{eq:em-kan-kart} \\699 Strengths & fast dense algebra, scales &700 small-scale accuracy, interpretable \\701 Weaknesses & opaque parameters &702 slower training, unproven at scale \\703 \bottomrule704 \end{tabular}705\end{table}706707\begin{remark}[Grid extension]\label{rem:em-grid}708The spline grid in \eqref{eq:em-kan-spline} can be refined during709training: a KAN first fitted on a coarse grid is re-projected onto a710finer one (a small least-squares problem per edge), increasing capacity711exactly where resolution is needed without restarting optimization — a712form of continuation in model space with no analogue in standard MLP713training.714\end{remark}715716% ============================================================================717\section{Outlook}718\label{sec:em-outlook}719720The four families of this chapter relax, in turn, each frozen ingredient721of the classical neural network: the scalar unit (capsules), the discrete722layer (neural ODEs), the fixed-size state (memory-augmented networks) and723the fixed nonlinearity (KANs). None has displaced the Transformer as the724dominant general-purpose architecture, but each has permanently enlarged725the design space — routing lives on in mixture-of-experts gating,726continuous-depth models in diffusion and flow-based generation, external727memory in retrieval-augmented systems, and learnable univariate bases in728scientific machine learning. The history of the field, from the729perceptron onward, suggests that ideas of this kind rarely disappear;730they wait for the scale, the hardware or the objective that lets them731matter.732