# Convolutional Neural Networks (CNNs): History, Mathematics, and Architectures

# 1. The Neocognitron (Fukushima, 1980): The Precursor

The direct ancestor of modern CNNs is the Neocognitron, proposed by Kunihiko Fukushima (Fukushima, K., 1980, "Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position", Biological Cybernetics, 36, 193–202). Invented in 1979 at NHK Science & Technical Research Laboratories, it was directly inspired by the neurophysiological work of Hubel and Wiesel (1959, 1962) on the cat's visual cortex, which identified simple cells (responding to oriented edges at specific positions) and complex cells (responding to the same features with positional tolerance).

The Neocognitron alternates two layer types in a hierarchy:

• S-cells (simple): extract local features via receptive fields with shared, learnable weights — the conceptual ancestor of the convolutional layer;
• C-cells (complex): pool responses of S-cells over a local neighborhood to gain invariance to small shifts — the ancestor of the pooling layer.

The network was trained by unsupervised, competitive self-organization ("learning without a teacher") and achieved shift-invariant pattern recognition. It lacked two ingredients of modern CNNs: end-to-end supervised training by backpropagation (introduced to CNNs by LeCun et al., 1989) and large-scale data/compute. Nevertheless, its S/C alternation is exactly the convolution/pooling alternation of LeNet and its successors.

# 2. The Convolution Operation

# 2.1 Discrete 2D equation

For an input image (or feature map) $I$ and a kernel (filter) $K$ of size $k_h \times k_w$, the 2D discrete convolution is:

$$ S(i, j) = (I * K)(i, j) = \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} I(i - m,; j - n), K(m, n) $$

In practice, deep learning frameworks implement cross-correlation (no kernel flip), which is equivalent up to a re-parameterization of learned weights:

$$ S(i, j) = (I \star K)(i, j) = \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} I(i + m,; j + n), K(m, n) $$

For a multi-channel input $x \in \mathbb{R}^{C_{in} \times H \times W}$ producing output channel $c_{out}$, with stride $s$ and bias $b$:

$$ y_{c_{out}}(i, j) = b_{c_{out}} + \sum_{c=1}^{C_{in}} \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} W_{c_{out}, c}(m, n); x_c(s,i + m,; s,j + n) $$

Key properties: local connectivity (each output depends only on a small receptive field), weight sharing (the same kernel slides over the whole image, giving translation equivariance and drastically reducing parameters), and hierarchical composition (stacked layers grow the receptive field, building edge → texture → part → object features).

# 2.2 Stride, padding, dilation

• Stride $s$: the step of the sliding window; $s > 1$ downsamples the output.
• Padding $p$: zeros (typically) added around the border. "Valid" = no padding; "same" padding ($p = \lfloor k/2 \rfloor$ for odd $k$, $s=1$) preserves spatial size.
• Dilation $d$: inserts $d - 1$ gaps between kernel taps (à trous convolution; Yu & Koltun, 2016, "Multi-Scale Context Aggregation by Dilated Convolutions", ICLR). The effective kernel size becomes

$$ k_{\text{eff}} = d,(k - 1) + 1 $$

which enlarges the receptive field exponentially when stacked, without adding parameters — central to segmentation networks such as DeepLab.

# 2.3 Output size formula

For input size $W_{in}$, kernel $k$, padding $p$, stride $s$, dilation $d$:

$$ W_{out} = \left\lfloor \frac{W_{in} + 2p - d,(k - 1) - 1}{s} \right\rfloor + 1 $$

which reduces to the classic formula when $d = 1$:

$$ W_{out} = \left\lfloor \frac{W_{in} - k + 2p}{s} \right\rfloor + 1 $$

The parameter count of a layer is $C_{out} \times (C_{in} \times k_h \times k_w + 1)$, independent of the spatial resolution — the essential advantage over fully connected layers.

# 3. Pooling

Pooling summarizes local neighborhoods, providing small translation invariance and downsampling. For a pooling window $\mathcal{R}_{ij}$ of size $k \times k$ with stride $s$:

Max pooling:

$$ y_{c}(i, j) = \max_{(m, n) \in \mathcal{R}{ij}} x{c}(m, n) $$

Average pooling:

$$ y_{c}(i, j) = \frac{1}{|\mathcal{R}{ij}|} \sum{(m, n) \in \mathcal{R}{ij}} x{c}(m, n) $$

Global average pooling (GAP) (Lin, Chen & Yan, 2014, "Network in Network", ICLR) collapses each channel's entire $H \times W$ map to one scalar:

$$ y_c = \frac{1}{H W} \sum_{i=1}^{H} \sum_{j=1}^{W} x_c(i, j) $$

GAP replaces the huge fully connected layers of AlexNet/VGG (which held most of their parameters), acts as a structural regularizer, and makes the network accept variable input sizes; it is standard from GoogLeNet and ResNet onward.

# 4. LeNet-5 (LeCun et al., 1998)

LeNet-5 (LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998, "Gradient-Based Learning Applied to Document Recognition", Proceedings of the IEEE, 86(11), 2278–2324) was the first widely deployed CNN, reading millions of bank checks. It takes $32 \times 32$ grayscale inputs and stacks 7 trainable layers:

Total: ~60k parameters. LeNet-5 established the canonical pattern [conv → pool] × N → FC → output and demonstrated end-to-end gradient-based training on raw pixels (MNIST error ~0.95%, ~0.8% with augmentation).

# 5. AlexNet (2012): The Deep Learning Detonator

AlexNet (Krizhevsky, A., Sutskever, I., Hinton, G. E., 2012, "ImageNet Classification with Deep Convolutional Neural Networks", NeurIPS) won ILSVRC-2012 with 15.3% top-5 error versus 26.2% for the runner-up — the gap that ignited the deep learning revolution.

Architecture: 8 learned layers — 5 convolutional (kernels $11\times11$ stride 4, then $5\times5$, then three $3\times3$) + 3 fully connected (4096, 4096, 1000), ~60M parameters, trained on 1.2M ImageNet images.

Key innovations:

• ReLU activation, $f(x) = \max(0, x)$: non-saturating, it trains ~6× faster than tanh and mitigates gradient saturation in deep stacks;
• Dropout (rate 0.5 in FC layers): randomly zeroing units at training time to prevent co-adaptation and overfitting (Hinton et al., 2012; Srivastava et al., 2014);
• Dual-GPU training (two GTX 580, 3 GB each): the model was split across GPUs, pioneering large-scale GPU training;
• Data augmentation (random crops, horizontal flips, PCA color jitter), overlapping max pooling ($3\times3$, stride 2), and local response normalization (LRN, later abandoned in favor of batch norm).

# 6. The Golden Age: VGG, GoogLeNet, ResNet

# 6.1 VGG (Simonyan & Zisserman, 2014)

VGG (Simonyan, K., Zisserman, A., 2015, "Very Deep Convolutional Networks for Large-Scale Image Recognition", ICLR; arXiv 2014) systematized depth using only $3\times3$ convolutions. Two stacked $3\times3$ layers have the receptive field of one $5\times5$; three match a $7\times7$ — with fewer parameters ($3 \cdot 9C^2 = 27C^2$ vs $49C^2$) and more nonlinearities. VGG-16/VGG-19 (16/19 weight layers, ~138M parameters, channels doubling 64→128→256→512 after each max pool) took 2nd place in ILSVRC-2014 classification and 1st in localization; its uniform design made it the default feature-extraction backbone for years.

# 6.2 GoogLeNet / Inception (Szegedy et al., 2014)

GoogLeNet (Szegedy, C., et al., 2015, "Going Deeper with Convolutions", CVPR; ILSVRC-2014 classification winner, 6.7% top-5) is a 22-layer network built from Inception modules: parallel branches of $1\times1$, $3\times3$, $5\times5$ convolutions and $3\times3$ max pooling, concatenated along the channel axis:

$$ y = \big[, f_{1\times1}(x); |; f_{3\times3}(f^{r}{1\times1}(x)); |; f{5\times5}(f^{r}{1\times1}(x)); |; f{1\times1}(\text{pool}(x)) ,\big] $$

The $1\times1$ "bottleneck" convolutions ($f^r_{1\times1}$) reduce channel dimension before the expensive $3\times3/5\times5$ operations, so the network captures multi-scale features cheaply: only ~7M parameters (vs 60M for AlexNet, 138M for VGG). Auxiliary classifiers injected gradient mid-network during training. Successors: Inception-v2/v3 (Szegedy et al., 2016; factorized convolutions, batch norm), Inception-v4 / Inception-ResNet (2017).

# 6.3 ResNet (He et al., 2015) and the residual connection

ResNet (He, K., Zhang, X., Ren, S., Sun, J., 2016, "Deep Residual Learning for Image Recognition", CVPR; arXiv:1512.03385, Dec 2015) solved the degradation problem: naively stacking more layers made even training error worse. The fix is to have each block learn a residual function with respect to its input via an identity shortcut:

$$ y = \mathcal{F}(x, {W_i}) + x $$

where typically $\mathcal{F}(x) = W_2, \sigma(\text{BN}(W_1 x))$ (two or three conv-BN-ReLU stages), followed by $\sigma(y)$. When dimensions change, a projection is used: $y = \mathcal{F}(x) + W_s x$. Deep ResNets use a bottleneck block ($1\times1$ reduce → $3\times3$ → $1\times1$ expand).

Why it fixes vanishing gradients / degradation. Consider stacked residual blocks $x_{l+1} = x_l + \mathcal{F}(x_l)$. Unrolling to any deeper layer $L$:

$$ x_L = x_l + \sum_{i=l}^{L-1} \mathcal{F}(x_i) $$

and by the chain rule the gradient of the loss $\mathcal{L}$ is:

$$ \frac{\partial \mathcal{L}}{\partial x_l} = \frac{\partial \mathcal{L}}{\partial x_L}\left(1 + \frac{\partial}{\partial x_l} \sum_{i=l}^{L-1} \mathcal{F}(x_i)\right) $$

The additive "$1$" term means the gradient flows directly from any layer to any shallower layer without being multiplied through dozens of weight matrices; it cannot vanish even if the residual branch's Jacobian is small (He et al., 2016, "Identity Mappings in Deep Residual Networks", ECCV). Moreover, learning $\mathcal{F} \approx 0$ (an identity mapping) is trivial — the network can only improve on shallower counterparts. ResNet-152 (8× deeper than VGG-19, yet cheaper in FLOPs) achieved 3.57% top-5 error as an ensemble, winning ILSVRC-2015 classification, detection, and localization, plus COCO detection and segmentation. The residual connection is arguably the most influential architectural idea in deep learning, adopted by Transformers as well.

# 7. Efficient and Modern Architectures

# 7.1 DenseNet (Huang et al., 2017)

DenseNet (Huang, G., Liu, Z., van der Maaten, L., Weinberger, K. Q., 2017, "Densely Connected Convolutional Networks", CVPR Best Paper) generalizes shortcuts: within a dense block, layer $\ell$ receives the concatenation of all preceding feature maps:

$$ x_\ell = H_\ell\big([,x_0, x_1, \ldots, x_{\ell-1},]\big) $$

where $H_\ell$ is BN → ReLU → conv and $[\cdot]$ denotes channel-wise concatenation (vs ResNet's addition). Each layer adds only $k$ channels (the growth rate, e.g. $k = 32$), so features are reused rather than recomputed, yielding strong parameter efficiency, implicit deep supervision, and excellent gradient flow. Transition layers ($1\times1$ conv + $2\times2$ average pooling) compress channels between blocks.

# 7.2 MobileNet (Howard et al., 2017): depthwise separable convolution

MobileNet (Howard, A. G., et al., 2017, "MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications", arXiv:1704.04861) factorizes a standard convolution into:

1. Depthwise convolution — one $D_K \times D_K$ filter per input channel (no cross-channel mixing): $$ \hat{y}m(i, j) = \sum{u,v} \hat{K}_m(u, v); x_m(i + u,; j + v) $$
2. Pointwise convolution — a $1\times1$ convolution mixing channels: $$ y_n(i, j) = \sum_{m=1}^{M} W_{n,m}; \hat{y}_m(i, j) $$

Cost comparison on a $D_F \times D_F$ feature map with $M$ input and $N$ output channels:

$$ \text{Standard: } D_K^2 \cdot M \cdot N \cdot D_F^2 \qquad \text{Separable: } D_K^2 \cdot M \cdot D_F^2 + M \cdot N \cdot D_F^2 $$

Reduction ratio:

$$ \frac{D_K^2 , M , D_F^2 + M N D_F^2}{D_K^2 , M , N , D_F^2} = \frac{1}{N} + \frac{1}{D_K^2} $$

For $3\times3$ kernels this is an ~8–9× reduction in computation with a small accuracy loss. MobileNetV2 (Sandler et al., 2018) added inverted residuals with linear bottlenecks; MobileNetV3 (Howard et al., 2019) added squeeze-and-excitation and neural architecture search.

# 7.3 EfficientNet (Tan & Le, 2019): compound scaling

EfficientNet (Tan, M., Le, Q. V., 2019, "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks", ICML) observed that scaling depth, width, or resolution in isolation saturates. Compound scaling scales all three jointly with one coefficient $\phi$:

$$ \text{depth } d = \alpha^{\phi}, \qquad \text{width } w = \beta^{\phi}, \qquad \text{resolution } r = \gamma^{\phi} $$

$$ \text{subject to } \alpha \cdot \beta^2 \cdot \gamma^2 \approx 2, \quad \alpha, \beta, \gamma \geq 1 $$

Since FLOPs scale as $d \cdot w^2 \cdot r^2$, the constraint makes total FLOPs grow as $\approx 2^{\phi}$. From a NAS-found baseline (EfficientNet-B0, built on MBConv blocks with squeeze-and-excitation; grid search gave $\alpha = 1.2$, $\beta = 1.1$, $\gamma = 1.15$), scaling produced the B1–B7 family; B7 reached 84.3% ImageNet top-1 with 8.4× fewer parameters than the best prior CNN. EfficientNetV2 (Tan & Le, 2021) improved training speed.

# 7.4 ConvNeXt (Liu et al., 2022)

ConvNeXt (Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., Xie, S., 2022, "A ConvNet for the 2020s", CVPR) answered the Vision Transformer wave by "modernizing" a ResNet step by step with Transformer-era design choices, while remaining a pure ConvNet: stage compute ratio 3:3:9:3 (like Swin), a patchify stem ($4\times4$ conv, stride 4), depthwise convolutions enlarged to $7\times7$, inverted bottlenecks, GELU instead of ReLU (and fewer activations), LayerNorm instead of BatchNorm (and fewer norms), separate downsampling layers, and modern training recipes (AdamW, 300 epochs, heavy augmentation). ConvNeXt matches or beats Swin Transformer (up to 87.8% ImageNet top-1, and superior COCO/ADE20K transfer), proving that much of ViT's advantage was training methodology and design details, not attention per se.

# 8. Batch Normalization in CNNs

Batch Normalization (Ioffe, S., Szegedy, C., 2015, "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift", ICML) normalizes each activation over the mini-batch, then rescales with learnable parameters:

$$ \mu_{\mathcal{B}} = \frac{1}{m}\sum_{i=1}^{m} x_i, \qquad \sigma_{\mathcal{B}}^2 = \frac{1}{m}\sum_{i=1}^{m} (x_i - \mu_{\mathcal{B}})^2 $$

$$ \hat{x}i = \frac{x_i - \mu{\mathcal{B}}}{\sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}}, \qquad y_i = \gamma, \hat{x}_i + \beta $$

CNN specificity: to respect convolutional weight sharing, normalization statistics are computed per channel, over the batch and all spatial positions jointly — the effective "batch" for channel $c$ has $m \cdot H \cdot W$ elements, and there is one pair $(\gamma_c, \beta_c)$ per channel, not per pixel. At inference, running (moving-average) estimates of $\mu$ and $\sigma^2$ replace batch statistics, allowing BN to be folded into the preceding convolution.

Benefits: much higher learning rates, faster convergence, reduced sensitivity to initialization, regularization (reducing the need for dropout), and smoothing of the optimization landscape (Santurkar et al., 2018, showed the original "internal covariate shift" explanation is incomplete). BN is integral to Inception-v2+, ResNet, DenseNet, MobileNet, EfficientNet. Alternatives for small batches or other modalities: Layer Norm (Ba et al., 2016), Instance Norm, Group Norm (Wu & He, 2018).

# 9. Object Detection Architectures

# 9.1 The R-CNN family (two-stage detectors)

• R-CNN (Girshick, R., Donahue, J., Darrell, T., Malik, J., 2014, "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation", CVPR): ~2000 region proposals from selective search, each warped and passed through a CNN, classified by per-class SVMs, with bounding-box regression. Accurate (mAP 58.5% on VOC07) but extremely slow (~47 s/image) since the CNN runs once per region.
• Fast R-CNN (Girshick, R., 2015, ICCV): runs the CNN once on the whole image; an RoI Pooling layer extracts a fixed-size feature vector per proposal; a single network jointly predicts class (softmax) and box offsets, trained with a multi-task loss $\mathcal{L} = \mathcal{L}{cls} + \lambda [u \geq 1], \mathcal{L}{loc}$ (smooth-$L_1$ for boxes). mAP 70.0% on VOC07, >200× faster inference than R-CNN.
• Faster R-CNN (Ren, S., He, K., Girshick, R., Sun, J., 2015, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks", NeurIPS): replaces selective search with a Region Proposal Network (RPN) — a small fully convolutional head sliding over shared feature maps, predicting objectness and box offsets relative to $k$ anchors (multi-scale, multi-aspect-ratio) at each position. Proposals become nearly free; the whole detector is end-to-end trainable at ~5 fps, and it remains the template for two-stage detection.

# 9.2 YOLO (one-stage) and its loss

YOLO (Redmon, J., Divvala, S., Girshick, R., Farhadi, A., 2016, "You Only Look Once: Unified, Real-Time Object Detection", CVPR) reframes detection as a single regression: the image is divided into an $S \times S$ grid ($S = 7$); each cell predicts $B$ boxes ($B = 2$) with confidence, plus $C$ class probabilities — one forward pass, 45 fps (155 fps for Fast YOLO). The sum-squared-error loss:

$$ \begin{aligned} \mathcal{L} = ;& \lambda_{\text{coord}} \sum_{i=0}^{S^2} \sum_{j=0}^{B} \mathbb{1}{ij}^{\text{obj}} \left[ (x_i - \hat{x}i)^2 + (y_i - \hat{y}i)^2 \right] \ +;& \lambda{\text{coord}} \sum{i=0}^{S^2} \sum{j=0}^{B} \mathbb{1}{ij}^{\text{obj}} \left[ \left(\sqrt{w_i} - \sqrt{\hat{w}i}\right)^2 + \left(\sqrt{h_i} - \sqrt{\hat{h}i}\right)^2 \right] \ +;& \sum{i=0}^{S^2} \sum{j=0}^{B} \mathbb{1}{ij}^{\text{obj}} \left( C_i - \hat{C}i \right)^2 ;+; \lambda{\text{noobj}} \sum_{i=0}^{S^2} \sum_{j=0}^{B} \mathbb{1}{ij}^{\text{noobj}} \left( C_i - \hat{C}i \right)^2 \ +;& \sum{i=0}^{S^2} \mathbb{1}{i}^{\text{obj}} \sum_{c \in \text{classes}} \left( p_i(c) - \hat{p}_i(c) \right)^2 \end{aligned} $$

with $\lambda_{\text{coord}} = 5$ (emphasize localization) and $\lambda_{\text{noobj}} = 0.5$ (most cells are background — prevents confidence collapse); $\mathbb{1}_{ij}^{\text{obj}}$ selects the predictor "responsible" for the object (highest IoU); square roots on $w, h$ make errors on small boxes matter more. Successors: YOLOv2/9000 (2017, anchors), YOLOv3 (2018, multi-scale FPN-style heads), then YOLOv4–v11+ by other teams.

# 9.3 SSD

SSD (Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., Berg, A. C., 2016, "SSD: Single Shot MultiBox Detector", ECCV) is a one-stage detector predicting class scores and offsets for default boxes of multiple aspect ratios on feature maps at several scales (early layers detect small objects, deep layers large ones). Loss: $\mathcal{L} = \frac{1}{N}(\mathcal{L}{conf} + \alpha, \mathcal{L}{loc})$ with hard negative mining (3:1 negative:positive). SSD300 matched Faster R-CNN accuracy at real-time speed (59 fps). The one-stage class-imbalance problem was later addressed by Focal Loss in RetinaNet (Lin et al., 2017): $\mathcal{L}_{FL} = -\alpha_t (1 - p_t)^{\gamma} \log(p_t)$.

# 10. Segmentation Architectures

# 10.1 FCN

FCN (Long, J., Shelhamer, E., Darrell, T., 2015, "Fully Convolutional Networks for Semantic Segmentation", CVPR) is the founding work of dense prediction: replace the fully connected layers of a classification CNN with $1\times1$ convolutions, so the network outputs a spatial class map for arbitrary input sizes; upsample with learned transposed convolutions ("deconvolutions"); and fuse coarse deep predictions with shallow, fine-grained features via skip fusions (FCN-32s → FCN-16s → FCN-8s), trained end-to-end with per-pixel cross-entropy.

# 10.2 U-Net

U-Net (Ronneberger, O., Fischer, P., Brox, T., 2015, "U-Net: Convolutional Networks for Biomedical Image Segmentation", MICCAI) is a symmetric encoder–decoder:

• Contracting path (encoder): repeated [two $3\times3$ convs + ReLU] → $2\times2$ max pool, doubling channels at each of 4 levels (64→128→256→512→1024) — captures context;
• Expanding path (decoder): $2\times2$ up-convolution halving channels, concatenation with the corresponding encoder feature map (skip connection), then two $3\times3$ convs — recovers localization;
• final $1\times1$ conv maps to class scores.

The skip connections reinject high-resolution spatial detail lost to pooling, enabling pixel-accurate boundaries, and provide short gradient paths. Trained with heavy elastic augmentation and a weighted cross-entropy emphasizing boundaries between touching cells, U-Net excels with very few annotated images and is the dominant architecture in medical imaging — and, notably, the standard denoising backbone of diffusion models. Variants: 3D U-Net (Çiçek et al., 2016), V-Net (Milletari et al., 2016, with Dice loss), U-Net++ (2018), nnU-Net (Isensee et al., 2021).

# 10.3 Mask R-CNN

Mask R-CNN (He, K., Gkioxari, G., Dollár, P., Girshick, R., 2017, "Mask R-CNN", ICCV, Marr Prize) extends Faster R-CNN for instance segmentation with a third, FCN-based branch predicting a binary mask per RoI, alongside classification and box regression:

$$ \mathcal{L} = \mathcal{L}{cls} + \mathcal{L}{box} + \mathcal{L}_{mask} $$

where $\mathcal{L}_{mask}$ is the average per-pixel binary cross-entropy applied only to the mask of the ground-truth class — decoupling mask and class prediction (no inter-class competition). Its key technical contribution is RoIAlign, which replaces RoI Pooling's harsh coordinate quantization with bilinear interpolation at exactly computed sampling points, preserving pixel-level spatial alignment — essential for masks and for keypoint estimation.

# 11. 1D and 3D CNNs

# 11.1 1D CNNs (signals, audio, text)

The 1D convolution over a sequence $x$ with kernel of size $k$:

$$ y(i) = \sum_{m=0}^{k-1} \sum_{c=1}^{C_{in}} K_c(m); x_c(i + m) $$

Applications:

• Text: Kim, Y. (2014, "Convolutional Neural Networks for Sentence Classification", EMNLP) convolves filters of widths 3/4/5 over word-embedding sequences (each filter an n-gram detector), followed by max-over-time pooling — a strong, simple sentence classifier. See also character-level CNNs (Zhang et al., 2015).
• Audio / time series: WaveNet (van den Oord et al., 2016) generates raw audio with stacked dilated causal 1D convolutions (dilations 1, 2, 4, …, 512) for exponentially large receptive fields; Temporal Convolutional Networks (TCN; Bai et al., 2018) apply the same recipe to generic sequence modeling and often beat RNNs. 1D CNNs are standard for ECG/EEG analysis, fault detection, and sensor data (Kiranyaz et al., 2021, survey).

# 11.2 3D CNNs (video, medical imaging)

3D convolution adds a depth/time axis; for a spatiotemporal kernel $k_t \times k_h \times k_w$:

$$ y(t, i, j) = \sum_{l=0}^{k_t-1} \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} K(l, m, n); x(t + l,; i + m,; j + n) $$

so features capture motion as well as appearance. Landmarks:

• Ji et al. (2013, TPAMI), "3D Convolutional Neural Networks for Human Action Recognition" — first 3D CNN for video;
• C3D (Tran, D., et al., 2015, "Learning Spatiotemporal Features with 3D Convolutional Networks", ICCV): homogeneous $3\times3\times3$ kernels shown to be the best choice; generic video features;
• I3D (Carreira, J., Zisserman, A., 2017, CVPR): "inflates" 2D ImageNet-pretrained kernels into 3D ($k\times k \to t \times k \times k$), two-stream RGB+flow, state of the art on Kinetics;
• Factorized variants — P3D (Qiu et al., 2017), R(2+1)D (Tran et al., 2018): decompose $3\times3\times3$ into a $1\times3\times3$ spatial plus $3\times1\times1$ temporal convolution, cheaper and often more accurate; SlowFast (Feichtenhofer et al., 2019) uses dual pathways at different frame rates.
• Medical imaging: 3D U-Net (Çiçek et al., 2016) and V-Net (Milletari et al., 2016) segment volumetric CT/MRI data directly, exploiting full 3D context at the cost of cubic memory growth — hence patch-based training and hybrid 2.5D approaches.

# Summary Timeline

Sources: Fukushima 1980 (Springer), output-size formula (Baeldung), LeNet-5 architecture, AlexNet paper (PDF), ResNet arXiv:1512.03385, DenseNet journal version, EfficientNet (PMLR), ConvNeXt arXiv:2201.03545, GoogLeNet overview, YOLOv1 loss walkthrough, Faster R-CNN (NeurIPS 2015), Fast R-CNN (ICCV 2015), SSD (Springer), FCN (CVPR 2015), U-Net guide, Mask R-CNN / RoIAlign, BatchNorm (PMLR), C3D arXiv:1412.0767, Kim 2014 arXiv:1408.5882.