% ============================================================================ % Artificial Neural Networks — Methods, Equations and Graphical % Representations % Author : Simon-Pierre Boucher — contact@spboucher.ai % Chapter 3 : Regularization (chapters/03-regularization.tex) % ============================================================================ \chapter{Regularization}\label{chap:regularization} A network that fits its training set perfectly is not, in general, a good network: what we actually care about is its behaviour on data it has never seen. Deep networks are typically over-parameterized — they possess enough capacity to memorize their training set outright — so the gap between training error and test error must be controlled deliberately. \emph{Regularization} is the collective name for the techniques that control this gap. Following Goodfellow, Bengio and Courville~\cite{goodfellow2016book}, we use the term broadly: a regularizer is any modification of the model, the objective or the training procedure whose purpose is to reduce generalization error, even at the price of a higher training error. \begin{definition}[Regularization]\label{def:reg} Let $\Loss_{\mathrm{data}}(\vect{\theta})$ denote the empirical loss of a model with parameters $\vect{\theta}$. A regularization technique is any alteration of the learning problem — an additive penalty $\Loss_{\mathrm{data}} + \lambda\,\Omega(\vect{\theta})$, a stochastic perturbation of the architecture, a normalization of intermediate activations, or a constraint on the training trajectory — designed to decrease the expected loss on unseen data drawn from the same distribution. \end{definition} \section{The Bias--Variance Decomposition} Why should reducing capacity ever help? The classical answer is the bias--variance decomposition. Consider a regression setting with targets generated as $y = f(\vect{x}) + \epsilon$, where $\epsilon$ is zero-mean noise of variance $\sigma^2$, and let $\hat{f}(\vect{x}; \mathcal{D})$ be the predictor learned from a random training set $\mathcal{D}$. The expected squared error at a point $\vect{x}$, averaged over draws of the training set and of the noise, splits into three terms: \begin{equation}\label{eq:reg-biasvariance} \E_{\mathcal{D},\epsilon}\!\left[\bigl(y - \hat{f}(\vect{x};\mathcal{D})\bigr)^2\right] = \underbrace{\bigl(f(\vect{x}) - \E_{\mathcal{D}}[\hat{f}(\vect{x};\mathcal{D})]\bigr)^2}_{\text{bias}^2} + \underbrace{\E_{\mathcal{D}}\!\left[\bigl(\hat{f}(\vect{x};\mathcal{D}) - \E_{\mathcal{D}}[\hat{f}(\vect{x};\mathcal{D})]\bigr)^2\right]}_{\text{variance}} + \underbrace{\sigma^2}_{\text{noise}} . \end{equation} The last term is irreducible: no learner can predict the noise. The first two trade off against each other. A rigid model (high bias) misses structure in the data; a flexible model fitted to a finite sample (high variance) tracks the accidents of that particular sample. Every technique in this chapter can be read as a way of purchasing a large reduction in variance at the cost of a small increase in bias, so that the sum in~\eqref{eq:reg-biasvariance} decreases. \section{Norm Penalties: $L_2$ and $L_1$} \subsection{$L_2$ regularization (weight decay)} The most venerable regularizer adds the squared Frobenius norm of the weight matrices to the data loss: \begin{equation}\label{eq:reg-l2loss} \Loss_{\mathrm{total}}(\vect{\theta}) = \Loss_{\mathrm{data}}(\vect{\theta}) + \frac{\lambda}{2} \sum_{\ell} \bigl\lVert \mat{W}^{(\ell)} \bigr\rVert_F^2 , \qquad \frac{\partial \Loss_{\mathrm{total}}}{\partial \mat{W}^{(\ell)}} = \frac{\partial \Loss_{\mathrm{data}}}{\partial \mat{W}^{(\ell)}} + \lambda \mat{W}^{(\ell)} . \end{equation} Substituting the penalized gradient into the SGD step with learning rate $\eta$ reveals why the method is called \emph{weight decay}: \begin{equation}\label{eq:reg-l2update} \mat{W}^{(\ell)} \;\leftarrow\; (1 - \eta\lambda)\,\mat{W}^{(\ell)} \;-\; \eta\, \frac{\partial \Loss_{\mathrm{data}}}{\partial \mat{W}^{(\ell)}} . \end{equation} Each update first shrinks every weight multiplicatively by the factor $(1-\eta\lambda) < 1$, then applies the usual data-driven correction. Weights that the data does not actively support are steadily pulled toward zero. Biases are conventionally excluded from the penalty: they do not multiply inputs, so penalizing them adds bias without reducing variance. \begin{remark}[Bayesian reading]\label{rem:reg-bayes} Minimizing~\eqref{eq:reg-l2loss} is maximum a posteriori estimation under a zero-mean Gaussian prior on the weights; the $L_1$ penalty of~\eqref{eq:reg-l1loss} corresponds in the same way to a Laplace prior. The regularization strength $\lambda$ plays the role of the inverse prior variance: the stronger our prior belief that weights are small, the harder we shrink. \end{remark} \subsection{$L_1$ regularization and sparsity} Replacing the squared norm by the absolute-value norm changes the character of the solution: \begin{equation}\label{eq:reg-l1loss} \Loss_{\mathrm{total}}(\vect{\theta}) = \Loss_{\mathrm{data}}(\vect{\theta}) + \lambda \sum_{\ell} \bigl\lVert \mat{W}^{(\ell)} \bigr\rVert_1 , \qquad \frac{\partial \Loss_{\mathrm{total}}}{\partial \mat{W}^{(\ell)}} = \frac{\partial \Loss_{\mathrm{data}}}{\partial \mat{W}^{(\ell)}} + \lambda \operatorname{sign}\bigl(\mat{W}^{(\ell)}\bigr) . \end{equation} The penalty gradient has \emph{constant magnitude} $\lambda$ regardless of how small a weight already is, so weights whose data gradient cannot sustain them are driven exactly to zero rather than merely toward it. The result is a sparse network — an implicit form of feature selection. The two penalties are often combined (the \emph{elastic net}), $\lambda_1 \lVert \mat{W} \rVert_1 + \tfrac{\lambda_2}{2}\lVert \mat{W} \rVert_2^2$, retaining the sparsity of $L_1$ and the grouping stability of $L_2$. \section{Dropout} Norm penalties act on parameters; \emph{dropout}, introduced by Srivastava et al.~\cite{srivastava2014}, acts on the architecture itself. During training, each unit of a layer is deleted independently at random, so that no unit can rely on the presence of any particular other unit — the phenomenon the authors call \emph{co-adaptation} is thereby suppressed. Formally, let $\vect{a}^{(\ell)}$ be the activation vector of layer $\ell$ and let $p$ be the dropout rate (the probability of deletion). A binary mask is sampled anew for every training example, and the surviving activations are rescaled — the \emph{inverted dropout} convention: \begin{equation}\label{eq:reg-dropout-mask} m_j^{(\ell)} \sim \mathrm{Bernoulli}(1-p), \qquad \tilde{\vect{a}}^{(\ell)} = \frac{\vect{m}^{(\ell)} \odot \vect{a}^{(\ell)}}{1-p} , \end{equation} and the next layer consumes $\tilde{\vect{a}}^{(\ell)}$ in place of $\vect{a}^{(\ell)}$. The division by $1-p$ keeps the expected input to each downstream unit unchanged, $\E[\tilde{a}_j^{(\ell)}] = a_j^{(\ell)}$, so that at test time the network is used \emph{without any modification}: \begin{equation}\label{eq:reg-dropout-test} \tilde{\vect{a}}^{(\ell)} = \vect{a}^{(\ell)} \qquad \text{(inference: no mask, no rescaling).} \end{equation} Typical rates are $p = 0.5$ for hidden layers and $p \approx 0.2$ for inputs. Figure~\ref{fig:reg-dropout} contrasts a standard fully connected network with one realization of its dropout-thinned counterpart. \begin{figure}[htbp] \centering \begin{tikzpicture}[scale=0.82, transform shape] % ---------- (a) standard network ---------- \begin{scope} \foreach \i in {1,2,3} \node[ninput] (ai\i) at (0, 1.0-\i*1.0) {}; \foreach \j in {1,...,4} \node[nhidden] (ah1\j) at (1.9, 1.5-\j*1.0) {}; \foreach \j in {1,...,4} \node[nhidden] (ah2\j) at (3.8, 1.5-\j*1.0) {}; \foreach \k in {1,2} \node[noutput] (ao\k) at (5.7, 0.5-\k*1.0) {}; \foreach \i in {1,2,3} \foreach \j in {1,...,4} \draw[black!40, semithick] (ai\i) -- (ah1\j); \foreach \i in {1,...,4} \foreach \j in {1,...,4} \draw[black!40, semithick] (ah1\i) -- (ah2\j); \foreach \i in {1,...,4} \foreach \k in {1,2} \draw[black!40, semithick] (ah2\i) -- (ao\k); \node[etiquette, align=center] at (2.85, -3.6) {(a) standard network}; \end{scope} % ---------- (b) after dropout ---------- \begin{scope}[xshift=8.6cm] % surviving units: all inputs, h1 = {1,3}, h2 = {2,4}, all outputs \foreach \i in {1,2,3} \node[ninput] (bi\i) at (0, 1.0-\i*1.0) {}; \foreach \j in {1,3} \node[nhidden] (bh1\j) at (1.9, 1.5-\j*1.0) {}; \foreach \j in {2,4} \node[nhidden] (bh2\j) at (3.8, 1.5-\j*1.0) {}; % dropped units: dashed outline, pale fill \foreach \j in {2,4} \node[neuron, dashed, draw=black!45, fill=black!5] (bh1\j) at (1.9, 1.5-\j*1.0) {}; \foreach \j in {1,3} \node[neuron, dashed, draw=black!45, fill=black!5] (bh2\j) at (3.8, 1.5-\j*1.0) {}; \foreach \k in {1,2} \node[noutput] (bo\k) at (5.7, 0.5-\k*1.0) {}; % edges touching a dropped unit: faint and dashed \foreach \i in {1,2,3} \foreach \j in {2,4} \draw[black!20, dashed] (bi\i) -- (bh1\j); \foreach \i in {1,3} \foreach \j in {1,3} \draw[black!20, dashed] (bh1\i) -- (bh2\j); \foreach \i in {2,4} \foreach \j in {1,...,4} \draw[black!20, dashed] (bh1\i) -- (bh2\j); \foreach \i in {1,3} \foreach \k in {1,2} \draw[black!20, dashed] (bh2\i) -- (bo\k); % active edges between surviving units \foreach \i in {1,2,3} \foreach \j in {1,3} \draw[black!40, semithick] (bi\i) -- (bh1\j); \foreach \i in {1,3} \foreach \j in {2,4} \draw[black!40, semithick] (bh1\i) -- (bh2\j); \foreach \i in {2,4} \foreach \k in {1,2} \draw[black!40, semithick] (bh2\i) -- (bo\k); % crosses on dropped units \foreach \n in {bh12, bh14, bh21, bh23}{ \draw[black!60, thick] (\n.north east) -- (\n.south west); \draw[black!60, thick] (\n.north west) -- (\n.south east); } \node[etiquette, align=center] at (2.85, -3.6) {(b) after dropout ($p = 0.5$ on hidden layers)}; \end{scope} \end{tikzpicture} \caption{Dropout as stochastic architecture perturbation. (a)~The full network. (b)~One training-time realization: each hidden unit is deleted independently with probability $p$ (crossed out, dashed), together with all of its incoming and outgoing connections; surviving activations are rescaled by $1/(1-p)$ as in~\eqref{eq:reg-dropout-mask}. A different subnetwork is sampled for every example.} \label{fig:reg-dropout} \end{figure} \begin{remark}[Ensemble interpretation]\label{rem:reg-ensemble} A network with $N$ droppable units defines $2^N$ thinned subnetworks sharing one set of weights. Dropout training optimizes the expected loss over this exponential ensemble, and inference with~\eqref{eq:reg-dropout-test} approximates the ensemble's geometric-mean prediction with a single forward pass~\cite{srivastava2014}. The original paper recommends pairing dropout with a max-norm constraint $\lVert \vect{w}_j \rVert_2 \le c$ on incoming weight vectors. \end{remark} \section{Normalization Layers} \subsection{Batch Normalization} Batch Normalization (BN), due to Ioffe and Szegedy~\cite{ioffe2015}, standardizes each pre-activation over the current mini-batch and then restores expressive freedom through two learned parameters. For a given unit, let $\mathcal{B} = \{z_1, \dots, z_m\}$ be the values it takes over a mini-batch of size $m$. The transform is defined by four equations: \begin{align} \mu_{\mathcal{B}} &= \frac{1}{m} \sum_{i=1}^{m} z_i && \text{(mini-batch mean)} \label{eq:reg-bn-mean}\\ \sigma_{\mathcal{B}}^2 &= \frac{1}{m} \sum_{i=1}^{m} \bigl(z_i - \mu_{\mathcal{B}}\bigr)^2 && \text{(mini-batch variance)} \label{eq:reg-bn-var}\\ \hat{z}_i &= \frac{z_i - \mu_{\mathcal{B}}} {\sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}} && \text{(normalize)} \label{eq:reg-bn-norm}\\ y_i &= \gamma\, \hat{z}_i + \beta && \text{(scale and shift)} \label{eq:reg-bn-scale} \end{align} where $\epsilon > 0$ is a small constant for numerical stability. The learned pair $(\gamma, \beta)$ ensures BN never destroys capacity: setting $\gamma = \sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}$ and $\beta = \mu_{\mathcal{B}}$ recovers the identity map. In convolutional networks the statistics are computed per channel, jointly over the batch and all spatial positions, with one pair $(\gamma_c, \beta_c)$ per channel, so as to respect the weight sharing of the convolution. At \emph{inference} no mini-batch is available. Exponential moving averages of the training statistics are accumulated, $\hat{\mu} \leftarrow \mathrm{EMA}(\mu_{\mathcal{B}})$ and $\hat{\sigma}^2 \leftarrow \mathrm{EMA}(\sigma_{\mathcal{B}}^2)$, and the whole layer becomes a fixed affine map, \begin{equation}\label{eq:reg-bn-inference} y = \gamma\, \frac{z - \hat{\mu}}{\sqrt{\hat{\sigma}^2 + \epsilon}} + \beta , \end{equation} which can be folded into the preceding linear or convolutional layer at no runtime cost. In practice BN permits substantially larger learning rates, reduces sensitivity to initialization, and contributes a mild regularizing effect through the noise of mini-batch statistics — which is why it appears in this chapter. Its main weaknesses are its dependence on a sufficiently large batch size and its awkwardness in recurrent or online settings. \subsection{Layer Normalization} Layer Normalization (LN) transposes the computation: the statistics are taken over the \emph{features of a single example} rather than over the batch. For a vector $\vect{z} \in \R^d$, \begin{equation}\label{eq:reg-ln} \mu = \frac{1}{d} \sum_{j=1}^{d} z_j , \qquad \sigma^2 = \frac{1}{d} \sum_{j=1}^{d} (z_j - \mu)^2 , \qquad \mathrm{LN}(\vect{z}) = \vect{\gamma} \odot \frac{\vect{z} - \mu}{\sqrt{\sigma^2 + \epsilon}} + \vect{\beta} , \end{equation} with learned vectors $\vect{\gamma}, \vect{\beta} \in \R^d$. \begin{remark}[BN versus LN: the normalization axis]\label{rem:reg-bnln} BN normalizes each feature across examples (equations~\eqref{eq:reg-bn-mean}--\eqref{eq:reg-bn-scale} run over the batch index $i$); LN normalizes each example across features (equation~\eqref{eq:reg-ln} runs over the feature index $j$). Because LN is independent of the batch, it behaves identically at training and test time, works at batch size one, and handles variable-length sequences — the reasons it, rather than BN, is the standard choice in recurrent networks and Transformers. \end{remark} \section{Regularizing the Training Procedure} \subsection{Early stopping} The simplest regularizer costs nothing: monitor the loss on a held-out validation set and stop training when it ceases to improve. The training loss decreases essentially monotonically, but the validation loss typically traces a U-shape — beyond its minimum, further optimization fits sampling noise rather than structure. Figure~\ref{fig:reg-earlystop} illustrates the regime change; in practice one checkpoints the parameters at each validation improvement and restores the best checkpoint when a \emph{patience} budget of non-improving epochs is exhausted. \begin{figure}[htbp] \centering \begin{tikzpicture} \begin{axis}[ width=0.78\textwidth, height=6.2cm, xlabel={epoch}, ylabel={loss}, xmin=0, xmax=100, ymin=0, ymax=1.15, axis lines=left, legend style={draw=none, fill=none, at={(0.97,0.95)}, anchor=north east, font=\small}, domain=0:100, samples=200, ] \addplot[cinput, thick] {0.10 + 0.90*exp(-x/15)}; \addlegendentry{training loss} \addplot[coutput, thick] {0.21 + 0.79*exp(-x/15) + 0.004*x}; \addlegendentry{validation loss} \draw[black!60, dashed, thick] (axis cs:37.9,0) -- (axis cs:37.9,1.08); \node[etiquette, anchor=south, rotate=90, text=black!75] at (axis cs:37.9,0.72) {early stopping point}; \addplot[coutput, only marks, mark=*, mark size=1.8pt] coordinates {(37.9,0.425)}; \end{axis} \end{tikzpicture} \caption{Early stopping. The training loss (blue) decreases throughout, while the validation loss (red) reaches a minimum and then rises as the network begins to overfit. Training is halted at the dashed line and the parameters of the best validation checkpoint are retained.} \label{fig:reg-earlystop} \end{figure} \begin{property}[Early stopping as implicit weight decay]\label{prop:reg-es} For a quadratic loss optimized by gradient descent from $\vect{\theta}_0 = \vect{0}$, halting after $\tau$ steps with learning rate $\eta$ yields a solution equivalent to fully optimizing the $L_2$-penalized objective~\eqref{eq:reg-l2loss} with regularization strength $\lambda \approx 1/(\eta\tau)$: a trajectory of limited length cannot travel far along low-curvature directions, exactly as weight decay suppresses them~\cite{goodfellow2016book}. \end{property} \subsection{Data augmentation} Rather than constraining the model, one can enlarge the data. If $\mathcal{T}$ is a family of label-preserving transformations — translations, crops, horizontal flips, small rotations, photometric jitter for images; time-stretching or noise injection for audio — training on transformed samples replaces the empirical loss by $\E_{t \sim \mathcal{T}}\bigl[\Loss\bigl(f(t(\vect{x})), \vect{y}\bigr)\bigr]$, encoding the invariances of the task directly into the training distribution. Augmentation is often the single most effective regularizer available for perception tasks, because it injects genuine prior knowledge rather than generic smoothness. \subsection{Label smoothing} Hard one-hot targets push the softmax toward infinite logit gaps and overconfident predictions. Label smoothing blends the one-hot vector $\vect{y}$ with the uniform distribution over the $K$ classes: \begin{equation}\label{eq:reg-labelsmooth} \vect{y}^{\mathrm{smooth}} = (1 - \varepsilon)\, \vect{y} + \frac{\varepsilon}{K} \, \vect{1} , \qquad \varepsilon \in [0, 1), \end{equation} with $\varepsilon = 0.1$ a common choice. The correct class keeps probability $1 - \varepsilon + \varepsilon/K$; every other class receives $\varepsilon/K$. The cross-entropy gradient then stops rewarding unbounded confidence, which improves calibration and often test accuracy. \subsection{Gradient clipping} Finally, the optimization trajectory itself can be regularized. When a loss surface contains cliffs — as in recurrent networks — a single large gradient can catapult the parameters out of a good basin. Gradient clipping rescales the gradient whenever its norm exceeds a threshold $\tau$: \begin{equation}\label{eq:reg-clip} \vect{g} \;\leftarrow\; \begin{cases} \vect{g} & \text{if } \lVert \vect{g} \rVert_2 \le \tau, \\[2pt] \dfrac{\tau}{\lVert \vect{g} \rVert_2}\, \vect{g} & \text{otherwise,} \end{cases} \end{equation} preserving the direction of the update while bounding its size. Clipping does not change the location of minima; it changes which minima the trajectory can reach, and is standard practice whenever exploding gradients are a risk. \medskip Taken together, the techniques of this chapter form a layered defence: penalties shape the hypothesis space, dropout and augmentation randomize what the network sees, normalization conditions the optimization, and early stopping bounds how far the fit is allowed to proceed. Modern practice combines several of them almost by default.