spb/artificial-neural-networks-book Public

1% ============================================================================2%  Artificial Neural Networks — Methods, Equations and Graphical3%  Representations4%  Author  : Simon-Pierre Boucher — contact@spboucher.ai5%  Chapter 4 : Convolutional Neural Networks (chapters/04-cnn.tex)6% ============================================================================7\chapter{Convolutional Neural Networks}\label{chap:cnn}89Fully connected networks treat every input coordinate as unrelated to every10other: a pixel in the top-left corner of an image and its immediate11neighbour are, as far as the architecture is concerned, as distant as two12arbitrary pixels. Convolutional neural networks (CNNs) build the spatial13structure of the data directly into the architecture through three14mechanisms: \emph{local connectivity} (each unit sees only a small window15of the input), \emph{weight sharing} (the same filter is applied at every16position, so a feature detector learned in one place is available17everywhere), and \emph{hierarchical composition} (stacked layers assemble18edges into textures, textures into parts, parts into objects). This chapter19develops the convolution and pooling operations, assembles them into the20canonical convolutional architecture of LeCun et21al.~\cite{lecun1998}, analyses the residual connection of He et22al.~\cite{he2016} that made very deep networks trainable, and closes with23the notion of receptive field and a brief genealogy of modern24architectures.2526% ----------------------------------------------------------------------------27\section{From the Visual Cortex to the Convolutional Layer}2829The lineage of the CNN begins in neurophysiology. Recording from the cat's30visual cortex, Hubel and Wiesel identified \emph{simple cells}, which31respond to oriented edges at specific retinal positions, and \emph{complex32cells}, which respond to the same features with tolerance to small shifts.33Fukushima's \emph{Neocognitron} (1980) translated this two-cell hierarchy34into a network that alternates feature-extracting S-cells — local35receptive fields with shared weights, the conceptual ancestor of the36convolutional layer — with position-tolerant C-cells, the ancestor of37pooling. What the Neocognitron lacked was end-to-end supervised training:38that ingredient was supplied when backpropagation was applied to39weight-shared architectures, culminating in LeNet-5~\cite{lecun1998}, the40first widely deployed CNN, which read millions of bank cheques using the41canonical pattern42$[\text{conv} \to \text{pool}] \times N \to \text{dense} \to43\text{output}$ that this chapter formalizes.4445% ----------------------------------------------------------------------------46\section{The Convolution Operation}4748\subsection{Definition}4950\begin{definition}[Discrete two-dimensional convolution]\label{def:cnn-conv}51Let $I$ be an input map and $K \in \R^{k_h \times k_w}$ a kernel (filter).52The discrete convolution of $I$ by $K$ is53\begin{equation}\label{eq:cnn-conv2d}54  S(i,j) \;=\; (I * K)(i,j)55  \;=\; \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} I(i-m,\, j-n)\, K(m,n).56\end{equation}57\end{definition}5859\begin{remark}[Convolution versus cross-correlation]\label{rem:cnn-crosscorr}60Deep learning frameworks implement \emph{cross-correlation} — the same sum61without flipping the kernel, i.e.\ with indices $I(i+m,\,j+n)$. Since the62kernel entries are learned, the two operations are equivalent up to a63re-parameterization of the weights, and the term ``convolution'' is used64for both. All equations below follow the cross-correlation convention.65\end{remark}6667In a convolutional \emph{layer}, the input is a stack of $C_{\mathrm{in}}$68channels $\vect{x} \in \R^{C_{\mathrm{in}} \times H \times W}$, and each of69the $C_{\mathrm{out}}$ output channels is produced by its own70three-dimensional filter plus a bias. With stride $s$ and zero-padding $p$71(writing $\tilde{\vect{x}}$ for the input bordered by $p$ zeros on each72side), the output channel $c_{\mathrm{out}}$ is73\begin{equation}\label{eq:cnn-convlayer}74  y_{c_{\mathrm{out}}}(i,j)75  \;=\; b_{c_{\mathrm{out}}}76  \;+\; \sum_{c=1}^{C_{\mathrm{in}}} \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1}77        W_{c_{\mathrm{out}},c}(m,n)\;78        \tilde{x}_{c}\big(s\,i + m,\; s\,j + n\big),79\end{equation}80where $i,j$ index the output positions. Three properties of81\eqref{eq:cnn-convlayer} explain the effectiveness of the operation. Each82output value depends only on a $k_h \times k_w$ window of the input83(\emph{local connectivity}); the same weights84$W_{c_{\mathrm{out}},c}$ are used at every position $(i,j)$ (\emph{weight85sharing}), which makes the layer \emph{equivariant} to translation — a86shifted input produces a correspondingly shifted output — and reduces the87parameter count dramatically; and stacking such layers composes local88detectors into progressively more global and abstract89ones~\cite{goodfellow2016book}. Algorithm~\ref{alg:cnn-forward} spells out90the layer's forward pass as literal loops over91\eqref{eq:cnn-convlayer} — production implementations replace the loops92with im2col-plus-GEMM or FFT-based routines, but compute exactly the same93map.9495\begin{algorithm}[htbp]96  \caption{Forward pass of a convolutional layer97    (stride $s$, padding $p$)}98  \label{alg:cnn-forward}99  \begin{algorithmic}[1]100    \Require input $\vect{x} \in \R^{C_{\mathrm{in}} \times H \times W}$,101      weights $W \in \R^{C_{\mathrm{out}} \times C_{\mathrm{in}} \times102      k_h \times k_w}$, biases $\vect{b}$, activation $\varphi$103    \State $\tilde{\vect{x}} \gets$ zero-pad $\vect{x}$ with a border of104      $p$ zeros on each spatial side105    \State $H_{\mathrm{out}} \gets \lfloor (H + 2p - k_h)/s \rfloor + 1$;106      \quad $W_{\mathrm{out}} \gets \lfloor (W + 2p - k_w)/s \rfloor + 1$107      \Comment{\eqref{eq:cnn-outsize}}108    \For{$c_{\mathrm{out}} = 1, \dots, C_{\mathrm{out}}$}109      \For{$i = 0, \dots, H_{\mathrm{out}}-1$;\;110           $j = 0, \dots, W_{\mathrm{out}}-1$}111        \State $u \gets b_{c_{\mathrm{out}}}$112        \For{$c = 1, \dots, C_{\mathrm{in}}$;\;113             $m = 0, \dots, k_h - 1$;\;114             $n = 0, \dots, k_w - 1$}115          \State $u \gets u + W_{c_{\mathrm{out}},c}(m,n)\,116            \tilde{x}_c(s\,i + m,\; s\,j + n)$117            \Comment{windowed sum \eqref{eq:cnn-convlayer}}118        \EndFor119        \State $y_{c_{\mathrm{out}}}(i,j) \gets \varphi(u)$120      \EndFor121    \EndFor122    \State \Return $\vect{y} \in \R^{C_{\mathrm{out}} \times123      H_{\mathrm{out}} \times W_{\mathrm{out}}}$124  \end{algorithmic}125\end{algorithm}126127\subsection{Stride, Padding and Output Geometry}128129The \emph{stride} $s$ is the step of the sliding window: $s>1$ downsamples130the output. The \emph{padding} $p$ adds a border of zeros; ``valid''131convolution uses $p=0$, while ``same'' padding, $p = \lfloor k/2 \rfloor$132for odd $k$ and $s=1$, preserves the spatial size. For an input of height133$H$, kernel size $k$, padding $p$ and stride $s$, the output height is134\begin{equation}\label{eq:cnn-outsize}135  H_{\mathrm{out}}136  \;=\; \left\lfloor \frac{H + 2p - k}{s} \right\rfloor + 1,137\end{equation}138and symmetrically for the width. (With \emph{dilation} $d$, which inserts139$d-1$ gaps between kernel taps, $k$ in \eqref{eq:cnn-outsize} is replaced140by the effective size $k_{\mathrm{eff}} = d(k-1)+1$; dilation enlarges141receptive fields without adding parameters.)142143Figure~\ref{fig:cnn-grid} traces \eqref{eq:cnn-convlayer} on a concrete144example: a $5 \times 5$ input, a $3 \times 3$ vertical-edge kernel, stride145$s=1$ and padding $p=0$, so that \eqref{eq:cnn-outsize} gives146$H_{\mathrm{out}} = \lfloor (5 + 0 - 3)/1 \rfloor + 1 = 3$.147148\begin{figure}[t]149  \centering150  \begin{tikzpicture}151    % -------- input grid 5x5 (6 mm cells), top-left corner at (0,0)152    \fill[cinput!30] (0,0) rectangle (1.8,-1.8);153    \draw[black!60] (0,-3) grid[step=6mm] (3,0);154    \draw[cinput!80!black, very thick] (0,0) rectangle (1.8,-1.8);155    \foreach \v [count=\k from 0] in156        {1,2,0,1,3, 0,1,2,3,1, 1,0,2,2,0, 2,1,0,1,2, 0,2,1,0,1}{157      \pgfmathtruncatemacro{\r}{int(\k/5)}158      \pgfmathtruncatemacro{\c}{mod(\k,5)}159      \node[font=\small] at (\c*0.6+0.3, -\r*0.6-0.3) {$\v$};160    }161    \node[etiquette, font=\small] at (1.5,-3.45) {input $5\times5$};162    % -------- operator163    \node[font=\large] at (3.9,-1.5) {$\ast$};164    % -------- kernel 3x3, vertically centred on the input165    \begin{scope}[shift={(4.8,-0.6)}]166      \fill[cgate!15] (0,0) rectangle (1.8,-1.8);167      \draw[black!60] (0,-1.8) grid[step=6mm] (1.8,0);168      \foreach \v [count=\k from 0] in {1,0,-1, 1,0,-1, 1,0,-1}{169        \pgfmathtruncatemacro{\r}{int(\k/3)}170        \pgfmathtruncatemacro{\c}{mod(\k,3)}171        \node[font=\small] at (\c*0.6+0.3, -\r*0.6-0.3) {$\v$};172      }173      \node[etiquette, font=\small] at (0.9,-2.25) {kernel $3\times3$};174    \end{scope}175    % -------- operator176    \node[font=\large] at (7.2,-1.5) {$=$};177    % -------- output grid 3x3178    \begin{scope}[shift={(7.8,-0.6)}]179      \fill[coutput!30] (0,0) rectangle (0.6,-0.6);180      \draw[black!60] (0,-1.8) grid[step=6mm] (1.8,0);181      \draw[coutput!80!black, very thick] (0,0) rectangle (0.6,-0.6);182      \foreach \v [count=\k from 0] in {-2,-3,0, -1,-4,1, 0,0,0}{183        \pgfmathtruncatemacro{\r}{int(\k/3)}184        \pgfmathtruncatemacro{\c}{mod(\k,3)}185        \node[font=\small] at (\c*0.6+0.3, -\r*0.6-0.3) {$\v$};186      }187      \node[etiquette, font=\small] at (0.9,-2.25) {output $3\times3$};188    \end{scope}189    % -------- window -> output cell190    \draw[flechep, cinput!80!black] (0.9,0.12) to[bend left=18] (8.1,-0.5);191  \end{tikzpicture}192  \caption{Convolution (cross-correlation) of a $5\times5$ input with a193    $3\times3$ vertical-edge kernel, stride $s=1$, padding $p=0$. The194    highlighted window produces the highlighted output cell:195    $1\cdot1 + 2\cdot0 + 0\cdot(-1) + 0\cdot1 + 1\cdot0 + 2\cdot(-1)196     + 1\cdot1 + 0\cdot0 + 2\cdot(-1) = -2$. Sliding the window over all197    nine valid positions fills the $3\times3$ output map, in accordance198    with \eqref{eq:cnn-outsize}.}199  \label{fig:cnn-grid}200\end{figure}201202\subsection{Parameter Efficiency}203204The number of parameters of the layer \eqref{eq:cnn-convlayer} is205\begin{equation}\label{eq:cnn-params}206  \#\text{params}207  \;=\; C_{\mathrm{out}} \left( C_{\mathrm{in}}\, k_h\, k_w + 1 \right),208\end{equation}209\emph{independent of the spatial resolution} $H \times W$ — the decisive210advantage over a dense layer, whose parameter count grows with the square211of the image size. A $3\times3$ convolution mapping $64$ channels to $64$212channels costs $36{,}928$ parameters whether the image is $32\times32$ or213$1024\times1024$.214215% ----------------------------------------------------------------------------216\section{Pooling}217218Pooling summarizes each local neighbourhood of a feature map by a single219value, providing a small amount of translation \emph{invariance} (on top of220the equivariance of convolution) and reducing the spatial resolution. Let221$\mathcal{R}_{ij}$ denote the $k \times k$ window of input positions222associated with output position $(i,j)$ (with stride $s$, typically223$k=s=2$). \emph{Max pooling} keeps the strongest activation,224\begin{equation}\label{eq:cnn-maxpool}225  y_c(i,j) \;=\; \max_{(m,n)\,\in\,\mathcal{R}_{ij}} x_c(m,n),226\end{equation}227whereas \emph{average pooling} keeps the mean response,228\begin{equation}\label{eq:cnn-avgpool}229  y_c(i,j) \;=\; \frac{1}{\lvert \mathcal{R}_{ij} \rvert}230                 \sum_{(m,n)\,\in\,\mathcal{R}_{ij}} x_c(m,n).231\end{equation}232The limiting case of \eqref{eq:cnn-avgpool}, \emph{global average pooling},233collapses each channel's entire $H \times W$ map to one scalar,234\begin{equation}\label{eq:cnn-gap}235  y_c \;=\; \frac{1}{HW} \sum_{i=1}^{H} \sum_{j=1}^{W} x_c(i,j),236\end{equation}237and is the modern replacement for the large dense layers that dominated238the parameter budget of early CNNs: it contributes no parameters, acts as239a structural regularizer, and lets the same network accept inputs of any240size.241242% ----------------------------------------------------------------------------243\section{The Canonical Convolutional Architecture}244245Assembling the two operations yields the canonical architecture of246Figure~\ref{fig:cnn-archi}: a \emph{feature-extraction} stage alternating247convolution (+ nonlinearity) and pooling, in which the spatial resolution248decreases while the channel count — the richness of the learned249vocabulary — increases; then a \emph{classification} stage, in which the250final maps are flattened (or globally pooled) into a vector, passed251through dense layers, and mapped by a softmax to class probabilities. This252is exactly the structure of LeNet-5~\cite{lecun1998}, which stacked two253conv--pool pairs and two dense layers ($\sim$60k parameters) and254established that such a pipeline can be trained end to end by gradient255descent on raw pixels.256257\begin{figure}[t]258  \centering259  \begin{tikzpicture}[node distance=0.55cm]260    \node[blocinput,  minimum width=1.05cm, minimum height=2.6cm,261          font=\scriptsize] (in) {input\\image};262    \node[blochidden, minimum width=1.05cm, minimum height=2.6cm,263          font=\scriptsize, right=of in]  (c1) {conv\\ReLU};264    \node[bloc,       minimum width=0.95cm, minimum height=2.0cm,265          font=\scriptsize, right=of c1]  (p1) {pool};266    \node[blochidden, minimum width=1.05cm, minimum height=2.0cm,267          font=\scriptsize, right=of p1]  (c2) {conv\\ReLU};268    \node[bloc,       minimum width=0.95cm, minimum height=1.4cm,269          font=\scriptsize, right=of c2]  (p2) {pool};270    \node[bloc,       minimum width=1.0cm,  minimum height=2.2cm,271          font=\scriptsize, right=of p2]  (fl) {flatten};272    \node[blochidden, minimum width=1.0cm,  minimum height=1.7cm,273          font=\scriptsize, right=of fl]  (fc) {dense\\ReLU};274    \node[blocoutput, minimum width=1.1cm,  minimum height=1.2cm,275          font=\scriptsize, right=of fc]  (sm) {softmax};276    \foreach \a/\b in {in/c1, c1/p1, p1/c2, c2/p2, p2/fl, fl/fc, fc/sm}277      \draw[fleche] (\a) -- (\b);278    % grouping braces279    \draw[decorate, decoration={brace, mirror, amplitude=5pt}]280      (c1.west |- 0,-1.55) -- (p2.east |- 0,-1.55)281      node[midway, below=7pt, etiquette, font=\small]{feature extraction};282    \draw[decorate, decoration={brace, mirror, amplitude=5pt}]283      (fl.west |- 0,-1.55) -- (sm.east |- 0,-1.55)284      node[midway, below=7pt, etiquette, font=\small]{classification};285  \end{tikzpicture}286  \caption{The canonical convolutional architecture in the lineage of287    LeNet-5~\cite{lecun1998}. In the feature-extraction stage the spatial288    resolution shrinks (decreasing block heights) while the channel count289    grows; the classification stage flattens the final maps and applies290    dense layers followed by a softmax.}291  \label{fig:cnn-archi}292\end{figure}293294The historical turning point came when this recipe met large datasets and295GPU computing: AlexNet's 2012 ImageNet victory — five convolutional and296three dense layers, ReLU activations and dropout — cut the top-5 error297from 26.2\% to 15.3\% and opened the modern era of deep298learning~\cite{goodfellow2016book}.299300% ----------------------------------------------------------------------------301\section{Residual Learning}302303Depth is the currency of representational power, yet naively stacking304layers eventually makes even the \emph{training} error worse — the305\emph{degradation problem}. A network of $50$ plain layers underperforms306its $20$-layer counterpart not because of overfitting but because the307optimizer fails to find a solution as good as ``the shallow network plus308identity layers'', although one exists by construction. Residual309learning~\cite{he2016} removes this obstacle by making the identity the310default behaviour of every block: instead of asking a block to learn a311mapping $\mathcal{H}(\vect{x})$ directly, one asks it to learn only the312\emph{residual} $\mathcal{F}(\vect{x}) = \mathcal{H}(\vect{x}) - \vect{x}$313and adds the input back through a shortcut connection,314\begin{equation}\label{eq:cnn-residual}315  \vect{y} \;=\; \mathcal{F}(\vect{x}, \{\mat{W}_i\}) + \vect{x},316\end{equation}317where typically $\mathcal{F}$ consists of two convolution--batch-norm318stages with a ReLU in between (Figure~\ref{fig:cnn-resblock}), and a319learned projection $\mat{W}_s \vect{x}$ replaces the identity when320dimensions change. If the optimal mapping is close to the identity, the321block merely needs to drive $\mathcal{F}$ towards zero — far easier than322reproducing the identity through a stack of nonlinear layers.323324\begin{figure}[t]325  \centering326  \begin{tikzpicture}327    \node (xl) at (0,0) {$\vect{x}$};328    \node[blochidden, right=0.95cm of xl, minimum width=2.3cm]329      (f1) {conv $3{\times}3$\\BN, ReLU};330    \node[blochidden, right=0.7cm of f1, minimum width=2.3cm]331      (f2) {conv $3{\times}3$\\BN};332    \node[op, right=0.95cm of f2] (plus) {$+$};333    \node[bloc, right=0.7cm of plus, minimum width=1.1cm] (relu) {ReLU};334    \node[right=0.7cm of relu] (y) {$\vect{y}$};335    \draw[fleche] (xl) -- (f1);336    \draw[fleche] (f1) -- (f2);337    \draw[fleche] (f2) -- (plus);338    \draw[fleche] (plus) -- (relu);339    \draw[fleche] (relu) -- (y);340    % identity shortcut341    \coordinate (tap) at ($(xl.east)+(0.45,0)$);342    \fill (tap) circle (1.3pt);343    \draw[fleche, cmem!80!black] (tap) -- ++(0,1.35) -| (plus.north);344    \node[etiquette, font=\small, text=cmem!80!black]345      at ($(f1.north)!0.5!(f2.north)+(0,1.15)$) {identity shortcut $\vect{x}$};346    \node[etiquette, font=\small]347      at ($(f1.south)!0.5!(f2.south)+(0,-0.35)$)348      {residual branch $\mathcal{F}(\vect{x})$};349  \end{tikzpicture}350  \caption{The residual block of ResNet~\cite{he2016}. The main path351    computes the residual $\mathcal{F}(\vect{x})$ through two352    convolution--batch-norm stages; the shortcut carries $\vect{x}$353    unchanged to the addition node, implementing354    \eqref{eq:cnn-residual}.}355  \label{fig:cnn-resblock}356\end{figure}357358\begin{property}[Gradient flow through the identity359  shortcut]\label{prop:cnn-resgrad}360Consider a stack of residual blocks361$\vect{x}_{\ell+1} = \vect{x}_\ell + \mathcal{F}(\vect{x}_\ell)$. Unrolling362from layer $\ell$ to any deeper layer $L$ gives363\begin{equation}\label{eq:cnn-unroll}364  \vect{x}_L \;=\; \vect{x}_\ell365    \;+\; \sum_{i=\ell}^{L-1} \mathcal{F}(\vect{x}_i),366\end{equation}367and the chain rule then yields368\begin{equation}\label{eq:cnn-resgrad}369  \frac{\partial \Loss}{\partial \vect{x}_\ell}370  \;=\;371  \frac{\partial \Loss}{\partial \vect{x}_L}372  \left( \mat{I} \;+\;373         \frac{\partial}{\partial \vect{x}_\ell}374         \sum_{i=\ell}^{L-1} \mathcal{F}(\vect{x}_i) \right).375\end{equation}376The additive identity term $\mat{I}$ in \eqref{eq:cnn-resgrad} provides a377direct gradient path from any layer to any shallower layer: the gradient378cannot vanish even when the Jacobian of the residual branch is small,379because it is never forced through a long product of weight matrices.380\end{property}381382This single architectural idea allowed networks of $152$ layers — eight383times deeper than the deepest previous mainstream design yet cheaper in384floating-point operations — to win the 2015 ImageNet competition with a3853.57\% top-5 ensemble error~\cite{he2016}; the same shortcut structure386reappears in virtually every subsequent deep architecture, including the387Transformer.388389% ----------------------------------------------------------------------------390\section{The Receptive Field}391392The \emph{receptive field} of a unit is the region of the input image that393can influence its value. For a stack of layers where layer $\ell$ has394kernel size $k_\ell$ and stride $s_\ell$, the receptive field $r_\ell$395grows according to the recursion396\begin{equation}\label{eq:cnn-receptive}397  r_\ell \;=\; r_{\ell-1} \;+\; (k_\ell - 1) \prod_{i=1}^{\ell-1} s_i,398  \qquad r_0 = 1,399\end{equation}400where the product accounts for the cumulative downsampling in front of401layer $\ell$. Two consequences guide architecture design. First, small402kernels compose efficiently: two stacked $3\times3$ convolutions reach the403same $5\times5$ receptive field as one $5\times5$ kernel with fewer404parameters ($18C^2$ versus $25C^2$) and one extra nonlinearity — the405observation that underlies VGG's uniform $3\times3$ design. Second,406strides and pooling are multiplicative in \eqref{eq:cnn-receptive}: early407downsampling is the cheapest way to grow the receptive field, which is why408deep units eventually see the whole image and can encode global,409object-level structure.410411% ----------------------------------------------------------------------------412\section{A Brief Genealogy of Architectures}413414Table~\ref{tab:cnn-genealogy} summarizes the milestones that separate415LeNet-5 from today's networks. Beyond ResNet, three ideas deserve mention.416\emph{Multi-branch design} (GoogLeNet's Inception module) processes each417position at several kernel sizes in parallel and concatenates the results,418using $1\times1$ ``bottleneck'' convolutions to keep the cost low —41922 layers with only $\sim$7M parameters. \emph{Dense connectivity}420(DenseNet) generalizes the shortcut of \eqref{eq:cnn-residual} by421concatenating, rather than adding, the features of \emph{all} preceding422layers, maximizing feature reuse and gradient flow. \emph{Factorized423convolution} (MobileNet) splits a standard convolution into a per-channel424(depthwise) spatial filter followed by a $1\times1$ (pointwise) channel425mixer; on a feature map with $M$ input channels, $N$ output channels and a426$D_K \times D_K$ kernel, the cost ratio relative to the standard layer is427\begin{equation}\label{eq:cnn-separable}428  \frac{\text{separable}}{\text{standard}}429  \;=\; \frac{1}{N} + \frac{1}{D_K^2},430\end{equation}431about an $8$--$9\times$ saving for $3\times3$ kernels — the enabling432arithmetic of mobile and embedded vision.433434\begin{table}[t]435  \centering436  \caption{Milestones of convolutional architecture design.}437  \label{tab:cnn-genealogy}438  \begin{tabular}{@{}llll@{}}439    \toprule440    Year & Architecture & Key idea & Scale \\441    \midrule442    1980 & Neocognitron & S/C-cell hierarchy, weight sharing & --- \\443    1998 & LeNet-5~\cite{lecun1998} & end-to-end conv--pool--dense & 60k \\444    2012 & AlexNet & ReLU, dropout, GPU training & 60M \\445    2014 & VGG & uniform $3\times3$ depth & 138M \\446    2014 & GoogLeNet & Inception multi-branch, $1\times1$ bottlenecks & 7M \\447    2015 & ResNet~\cite{he2016} & identity shortcut \eqref{eq:cnn-residual} & 25M \\448    2017 & DenseNet & concatenated dense connectivity & 8M \\449    2017 & MobileNet & depthwise separable conv.\ \eqref{eq:cnn-separable} & 4M \\450    2019 & EfficientNet & compound depth/width/resolution scaling & 5--66M \\451    2022 & ConvNeXt & modernized ResNet, $7\times7$ depthwise & 29M+ \\452    \bottomrule453  \end{tabular}454\end{table}455456The through-line of this genealogy is that every leap either improved457\emph{gradient flow} (ReLU, batch normalization, the shortcut of458\eqref{eq:cnn-resgrad}) or improved the \emph{allocation of computation}459(bottlenecks, separable filters, compound scaling). The convolutional460prior itself — locality, weight sharing, hierarchy — has remained intact461from the Neocognitron to ConvNeXt, and it transfers beyond images: 1D462convolutions over sequences and 3D convolutions over videos and volumes463are the direct analogues of \eqref{eq:cnn-convlayer} with one fewer or one464more spatial index~\cite{goodfellow2016book}.465