spb/artificial-neural-networks-book Public

1% ============================================================================2%  Artificial Neural Networks — Methods, Equations and Graphical3%  Representations4%  Author  : Simon-Pierre Boucher — contact@spboucher.ai5%  Chapter 3 : Regularization (chapters/03-regularization.tex)6% ============================================================================7\chapter{Regularization}\label{chap:regularization}89A network that fits its training set perfectly is not, in general, a good10network: what we actually care about is its behaviour on data it has never11seen. Deep networks are typically over-parameterized — they possess enough12capacity to memorize their training set outright — so the gap between13training error and test error must be controlled deliberately.14\emph{Regularization} is the collective name for the techniques that15control this gap. Following Goodfellow, Bengio and16Courville~\cite{goodfellow2016book}, we use the term broadly: a17regularizer is any modification of the model, the objective or the18training procedure whose purpose is to reduce generalization error, even19at the price of a higher training error.2021\begin{definition}[Regularization]\label{def:reg}22Let $\Loss_{\mathrm{data}}(\vect{\theta})$ denote the empirical loss of a23model with parameters $\vect{\theta}$. A regularization technique is any24alteration of the learning problem — an additive penalty25$\Loss_{\mathrm{data}} + \lambda\,\Omega(\vect{\theta})$, a stochastic26perturbation of the architecture, a normalization of intermediate27activations, or a constraint on the training trajectory — designed to28decrease the expected loss on unseen data drawn from the same29distribution.30\end{definition}3132\section{The Bias--Variance Decomposition}3334Why should reducing capacity ever help? The classical answer is the35bias--variance decomposition. Consider a regression setting with targets36generated as $y = f(\vect{x}) + \epsilon$, where $\epsilon$ is zero-mean37noise of variance $\sigma^2$, and let $\hat{f}(\vect{x}; \mathcal{D})$ be38the predictor learned from a random training set $\mathcal{D}$. The39expected squared error at a point $\vect{x}$, averaged over draws of the40training set and of the noise, splits into three terms:41\begin{equation}\label{eq:reg-biasvariance}42\E_{\mathcal{D},\epsilon}\!\left[\bigl(y - \hat{f}(\vect{x};\mathcal{D})\bigr)^2\right]43=44\underbrace{\bigl(f(\vect{x}) - \E_{\mathcal{D}}[\hat{f}(\vect{x};\mathcal{D})]\bigr)^2}_{\text{bias}^2}45+46\underbrace{\E_{\mathcal{D}}\!\left[\bigl(\hat{f}(\vect{x};\mathcal{D}) - \E_{\mathcal{D}}[\hat{f}(\vect{x};\mathcal{D})]\bigr)^2\right]}_{\text{variance}}47+48\underbrace{\sigma^2}_{\text{noise}} .49\end{equation}50The last term is irreducible: no learner can predict the noise. The first51two trade off against each other. A rigid model (high bias) misses52structure in the data; a flexible model fitted to a finite sample (high53variance) tracks the accidents of that particular sample. Every technique54in this chapter can be read as a way of purchasing a large reduction in55variance at the cost of a small increase in bias, so that the sum56in~\eqref{eq:reg-biasvariance} decreases.5758\section{Norm Penalties: $L_2$ and $L_1$}5960\subsection{$L_2$ regularization (weight decay)}6162The most venerable regularizer adds the squared Frobenius norm of the63weight matrices to the data loss:64\begin{equation}\label{eq:reg-l2loss}65\Loss_{\mathrm{total}}(\vect{\theta})66= \Loss_{\mathrm{data}}(\vect{\theta})67+ \frac{\lambda}{2} \sum_{\ell} \bigl\lVert \mat{W}^{(\ell)} \bigr\rVert_F^2 ,68\qquad69\frac{\partial \Loss_{\mathrm{total}}}{\partial \mat{W}^{(\ell)}}70= \frac{\partial \Loss_{\mathrm{data}}}{\partial \mat{W}^{(\ell)}}71+ \lambda \mat{W}^{(\ell)} .72\end{equation}73Substituting the penalized gradient into the SGD step with learning rate74$\eta$ reveals why the method is called \emph{weight decay}:75\begin{equation}\label{eq:reg-l2update}76\mat{W}^{(\ell)} \;\leftarrow\; (1 - \eta\lambda)\,\mat{W}^{(\ell)}77\;-\; \eta\, \frac{\partial \Loss_{\mathrm{data}}}{\partial \mat{W}^{(\ell)}} .78\end{equation}79Each update first shrinks every weight multiplicatively by the factor80$(1-\eta\lambda) < 1$, then applies the usual data-driven correction.81Weights that the data does not actively support are steadily pulled toward82zero. Biases are conventionally excluded from the penalty: they do not83multiply inputs, so penalizing them adds bias without reducing variance.8485\begin{remark}[Bayesian reading]\label{rem:reg-bayes}86Minimizing~\eqref{eq:reg-l2loss} is maximum a posteriori estimation under87a zero-mean Gaussian prior on the weights; the $L_1$ penalty88of~\eqref{eq:reg-l1loss} corresponds in the same way to a Laplace prior.89The regularization strength $\lambda$ plays the role of the inverse prior90variance: the stronger our prior belief that weights are small, the harder91we shrink.92\end{remark}9394\subsection{$L_1$ regularization and sparsity}9596Replacing the squared norm by the absolute-value norm changes the97character of the solution:98\begin{equation}\label{eq:reg-l1loss}99\Loss_{\mathrm{total}}(\vect{\theta})100= \Loss_{\mathrm{data}}(\vect{\theta})101+ \lambda \sum_{\ell} \bigl\lVert \mat{W}^{(\ell)} \bigr\rVert_1 ,102\qquad103\frac{\partial \Loss_{\mathrm{total}}}{\partial \mat{W}^{(\ell)}}104= \frac{\partial \Loss_{\mathrm{data}}}{\partial \mat{W}^{(\ell)}}105+ \lambda \operatorname{sign}\bigl(\mat{W}^{(\ell)}\bigr) .106\end{equation}107The penalty gradient has \emph{constant magnitude} $\lambda$ regardless of108how small a weight already is, so weights whose data gradient cannot109sustain them are driven exactly to zero rather than merely toward it. The110result is a sparse network — an implicit form of feature selection. The111two penalties are often combined (the \emph{elastic net}),112$\lambda_1 \lVert \mat{W} \rVert_1 + \tfrac{\lambda_2}{2}\lVert \mat{W} \rVert_2^2$,113retaining the sparsity of $L_1$ and the grouping stability of $L_2$.114115\section{Dropout}116117Norm penalties act on parameters; \emph{dropout}, introduced by Srivastava118et al.~\cite{srivastava2014}, acts on the architecture itself. During119training, each unit of a layer is deleted independently at random, so that120no unit can rely on the presence of any particular other unit — the121phenomenon the authors call \emph{co-adaptation} is thereby suppressed.122123Formally, let $\vect{a}^{(\ell)}$ be the activation vector of layer $\ell$124and let $p$ be the dropout rate (the probability of deletion). A binary125mask is sampled anew for every training example, and the surviving126activations are rescaled — the \emph{inverted dropout} convention:127\begin{equation}\label{eq:reg-dropout-mask}128m_j^{(\ell)} \sim \mathrm{Bernoulli}(1-p),129\qquad130\tilde{\vect{a}}^{(\ell)}131= \frac{\vect{m}^{(\ell)} \odot \vect{a}^{(\ell)}}{1-p} ,132\end{equation}133and the next layer consumes $\tilde{\vect{a}}^{(\ell)}$ in place of134$\vect{a}^{(\ell)}$. The division by $1-p$ keeps the expected input to135each downstream unit unchanged,136$\E[\tilde{a}_j^{(\ell)}] = a_j^{(\ell)}$, so that at test time the137network is used \emph{without any modification}:138\begin{equation}\label{eq:reg-dropout-test}139\tilde{\vect{a}}^{(\ell)} = \vect{a}^{(\ell)}140\qquad \text{(inference: no mask, no rescaling).}141\end{equation}142Typical rates are $p = 0.5$ for hidden layers and $p \approx 0.2$ for143inputs. Figure~\ref{fig:reg-dropout} contrasts a standard fully connected144network with one realization of its dropout-thinned counterpart.145146\begin{figure}[htbp]147  \centering148  \begin{tikzpicture}[scale=0.82, transform shape]149    % ---------- (a) standard network ----------150    \begin{scope}151      \foreach \i in {1,2,3}152        \node[ninput]  (ai\i) at (0, 1.0-\i*1.0) {};153      \foreach \j in {1,...,4}154        \node[nhidden] (ah1\j) at (1.9, 1.5-\j*1.0) {};155      \foreach \j in {1,...,4}156        \node[nhidden] (ah2\j) at (3.8, 1.5-\j*1.0) {};157      \foreach \k in {1,2}158        \node[noutput] (ao\k) at (5.7, 0.5-\k*1.0) {};159      \foreach \i in {1,2,3} \foreach \j in {1,...,4}160        \draw[black!40, semithick] (ai\i) -- (ah1\j);161      \foreach \i in {1,...,4} \foreach \j in {1,...,4}162        \draw[black!40, semithick] (ah1\i) -- (ah2\j);163      \foreach \i in {1,...,4} \foreach \k in {1,2}164        \draw[black!40, semithick] (ah2\i) -- (ao\k);165      \node[etiquette, align=center] at (2.85, -3.6)166        {(a) standard network};167    \end{scope}168    % ---------- (b) after dropout ----------169    \begin{scope}[xshift=8.6cm]170      % surviving units: all inputs, h1 = {1,3}, h2 = {2,4}, all outputs171      \foreach \i in {1,2,3}172        \node[ninput]  (bi\i) at (0, 1.0-\i*1.0) {};173      \foreach \j in {1,3}174        \node[nhidden] (bh1\j) at (1.9, 1.5-\j*1.0) {};175      \foreach \j in {2,4}176        \node[nhidden] (bh2\j) at (3.8, 1.5-\j*1.0) {};177      % dropped units: dashed outline, pale fill178      \foreach \j in {2,4}179        \node[neuron, dashed, draw=black!45, fill=black!5]180          (bh1\j) at (1.9, 1.5-\j*1.0) {};181      \foreach \j in {1,3}182        \node[neuron, dashed, draw=black!45, fill=black!5]183          (bh2\j) at (3.8, 1.5-\j*1.0) {};184      \foreach \k in {1,2}185        \node[noutput] (bo\k) at (5.7, 0.5-\k*1.0) {};186      % edges touching a dropped unit: faint and dashed187      \foreach \i in {1,2,3} \foreach \j in {2,4}188        \draw[black!20, dashed] (bi\i) -- (bh1\j);189      \foreach \i in {1,3} \foreach \j in {1,3}190        \draw[black!20, dashed] (bh1\i) -- (bh2\j);191      \foreach \i in {2,4} \foreach \j in {1,...,4}192        \draw[black!20, dashed] (bh1\i) -- (bh2\j);193      \foreach \i in {1,3} \foreach \k in {1,2}194        \draw[black!20, dashed] (bh2\i) -- (bo\k);195      % active edges between surviving units196      \foreach \i in {1,2,3} \foreach \j in {1,3}197        \draw[black!40, semithick] (bi\i) -- (bh1\j);198      \foreach \i in {1,3} \foreach \j in {2,4}199        \draw[black!40, semithick] (bh1\i) -- (bh2\j);200      \foreach \i in {2,4} \foreach \k in {1,2}201        \draw[black!40, semithick] (bh2\i) -- (bo\k);202      % crosses on dropped units203      \foreach \n in {bh12, bh14, bh21, bh23}{204        \draw[black!60, thick] (\n.north east) -- (\n.south west);205        \draw[black!60, thick] (\n.north west) -- (\n.south east);206      }207      \node[etiquette, align=center] at (2.85, -3.6)208        {(b) after dropout ($p = 0.5$ on hidden layers)};209    \end{scope}210  \end{tikzpicture}211  \caption{Dropout as stochastic architecture perturbation. (a)~The full212  network. (b)~One training-time realization: each hidden unit is deleted213  independently with probability $p$ (crossed out, dashed), together with214  all of its incoming and outgoing connections; surviving activations are215  rescaled by $1/(1-p)$ as in~\eqref{eq:reg-dropout-mask}. A different216  subnetwork is sampled for every example.}217  \label{fig:reg-dropout}218\end{figure}219220\begin{remark}[Ensemble interpretation]\label{rem:reg-ensemble}221A network with $N$ droppable units defines $2^N$ thinned subnetworks222sharing one set of weights. Dropout training optimizes the expected loss223over this exponential ensemble, and inference224with~\eqref{eq:reg-dropout-test} approximates the ensemble's geometric-mean225prediction with a single forward pass~\cite{srivastava2014}. The original226paper recommends pairing dropout with a max-norm constraint227$\lVert \vect{w}_j \rVert_2 \le c$ on incoming weight vectors.228\end{remark}229230\section{Normalization Layers}231232\subsection{Batch Normalization}233234Batch Normalization (BN), due to Ioffe and Szegedy~\cite{ioffe2015},235standardizes each pre-activation over the current mini-batch and then236restores expressive freedom through two learned parameters. For a given237unit, let $\mathcal{B} = \{z_1, \dots, z_m\}$ be the values it takes over238a mini-batch of size $m$. The transform is defined by four equations:239\begin{align}240\mu_{\mathcal{B}} &= \frac{1}{m} \sum_{i=1}^{m} z_i241  && \text{(mini-batch mean)} \label{eq:reg-bn-mean}\\242\sigma_{\mathcal{B}}^2 &= \frac{1}{m} \sum_{i=1}^{m}243  \bigl(z_i - \mu_{\mathcal{B}}\bigr)^2244  && \text{(mini-batch variance)} \label{eq:reg-bn-var}\\245\hat{z}_i &= \frac{z_i - \mu_{\mathcal{B}}}246  {\sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}}247  && \text{(normalize)} \label{eq:reg-bn-norm}\\248y_i &= \gamma\, \hat{z}_i + \beta249  && \text{(scale and shift)} \label{eq:reg-bn-scale}250\end{align}251where $\epsilon > 0$ is a small constant for numerical stability. The252learned pair $(\gamma, \beta)$ ensures BN never destroys capacity:253setting $\gamma = \sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}$ and254$\beta = \mu_{\mathcal{B}}$ recovers the identity map. In convolutional255networks the statistics are computed per channel, jointly over the batch256and all spatial positions, with one pair $(\gamma_c, \beta_c)$ per257channel, so as to respect the weight sharing of the convolution.258259At \emph{inference} no mini-batch is available. Exponential moving260averages of the training statistics are accumulated,261$\hat{\mu} \leftarrow \mathrm{EMA}(\mu_{\mathcal{B}})$ and262$\hat{\sigma}^2 \leftarrow \mathrm{EMA}(\sigma_{\mathcal{B}}^2)$, and the263whole layer becomes a fixed affine map,264\begin{equation}\label{eq:reg-bn-inference}265y = \gamma\, \frac{z - \hat{\mu}}{\sqrt{\hat{\sigma}^2 + \epsilon}} + \beta ,266\end{equation}267which can be folded into the preceding linear or convolutional layer at268no runtime cost. In practice BN permits substantially larger learning269rates, reduces sensitivity to initialization, and contributes a mild270regularizing effect through the noise of mini-batch statistics — which is271why it appears in this chapter. Its main weaknesses are its dependence on272a sufficiently large batch size and its awkwardness in recurrent or273online settings.274275\subsection{Layer Normalization}276277Layer Normalization (LN) transposes the computation: the statistics are278taken over the \emph{features of a single example} rather than over the279batch. For a vector $\vect{z} \in \R^d$,280\begin{equation}\label{eq:reg-ln}281\mu = \frac{1}{d} \sum_{j=1}^{d} z_j ,282\qquad283\sigma^2 = \frac{1}{d} \sum_{j=1}^{d} (z_j - \mu)^2 ,284\qquad285\mathrm{LN}(\vect{z})286= \vect{\gamma} \odot \frac{\vect{z} - \mu}{\sqrt{\sigma^2 + \epsilon}}287+ \vect{\beta} ,288\end{equation}289with learned vectors $\vect{\gamma}, \vect{\beta} \in \R^d$.290291\begin{remark}[BN versus LN: the normalization axis]\label{rem:reg-bnln}292BN normalizes each feature across examples293(equations~\eqref{eq:reg-bn-mean}--\eqref{eq:reg-bn-scale} run over the294batch index $i$); LN normalizes each example across features295(equation~\eqref{eq:reg-ln} runs over the feature index $j$). Because LN296is independent of the batch, it behaves identically at training and test297time, works at batch size one, and handles variable-length sequences —298the reasons it, rather than BN, is the standard choice in recurrent299networks and Transformers.300\end{remark}301302\section{Regularizing the Training Procedure}303304\subsection{Early stopping}305306The simplest regularizer costs nothing: monitor the loss on a held-out307validation set and stop training when it ceases to improve. The training308loss decreases essentially monotonically, but the validation loss309typically traces a U-shape — beyond its minimum, further optimization310fits sampling noise rather than structure. Figure~\ref{fig:reg-earlystop}311illustrates the regime change; in practice one checkpoints the parameters312at each validation improvement and restores the best checkpoint when a313\emph{patience} budget of non-improving epochs is exhausted.314315\begin{figure}[htbp]316  \centering317  \begin{tikzpicture}318    \begin{axis}[319        width=0.78\textwidth, height=6.2cm,320        xlabel={epoch}, ylabel={loss},321        xmin=0, xmax=100, ymin=0, ymax=1.15,322        axis lines=left,323        legend style={draw=none, fill=none, at={(0.97,0.95)},324                      anchor=north east, font=\small},325        domain=0:100, samples=200,326      ]327      \addplot[cinput, thick] {0.10 + 0.90*exp(-x/15)};328      \addlegendentry{training loss}329      \addplot[coutput, thick] {0.21 + 0.79*exp(-x/15) + 0.004*x};330      \addlegendentry{validation loss}331      \draw[black!60, dashed, thick] (axis cs:37.9,0) -- (axis cs:37.9,1.08);332      \node[etiquette, anchor=south, rotate=90, text=black!75]333        at (axis cs:37.9,0.72) {early stopping point};334      \addplot[coutput, only marks, mark=*, mark size=1.8pt]335        coordinates {(37.9,0.425)};336    \end{axis}337  \end{tikzpicture}338  \caption{Early stopping. The training loss (blue) decreases throughout,339  while the validation loss (red) reaches a minimum and then rises as the340  network begins to overfit. Training is halted at the dashed line and341  the parameters of the best validation checkpoint are retained.}342  \label{fig:reg-earlystop}343\end{figure}344345\begin{property}[Early stopping as implicit weight decay]\label{prop:reg-es}346For a quadratic loss optimized by gradient descent from347$\vect{\theta}_0 = \vect{0}$, halting after $\tau$ steps with learning348rate $\eta$ yields a solution equivalent to fully optimizing the349$L_2$-penalized objective~\eqref{eq:reg-l2loss} with regularization350strength $\lambda \approx 1/(\eta\tau)$: a trajectory of limited length351cannot travel far along low-curvature directions, exactly as weight decay352suppresses them~\cite{goodfellow2016book}.353\end{property}354355\subsection{Data augmentation}356357Rather than constraining the model, one can enlarge the data. If358$\mathcal{T}$ is a family of label-preserving transformations —359translations, crops, horizontal flips, small rotations, photometric360jitter for images; time-stretching or noise injection for audio —361training on transformed samples replaces the empirical loss by362$\E_{t \sim \mathcal{T}}\bigl[\Loss\bigl(f(t(\vect{x})), \vect{y}\bigr)\bigr]$,363encoding the invariances of the task directly into the training364distribution. Augmentation is often the single most effective365regularizer available for perception tasks, because it injects genuine366prior knowledge rather than generic smoothness.367368\subsection{Label smoothing}369370Hard one-hot targets push the softmax toward infinite logit gaps and371overconfident predictions. Label smoothing blends the one-hot vector372$\vect{y}$ with the uniform distribution over the $K$ classes:373\begin{equation}\label{eq:reg-labelsmooth}374\vect{y}^{\mathrm{smooth}} = (1 - \varepsilon)\, \vect{y}375+ \frac{\varepsilon}{K} \, \vect{1} ,376\qquad \varepsilon \in [0, 1),377\end{equation}378with $\varepsilon = 0.1$ a common choice. The correct class keeps379probability $1 - \varepsilon + \varepsilon/K$; every other class receives380$\varepsilon/K$. The cross-entropy gradient then stops rewarding381unbounded confidence, which improves calibration and often test accuracy.382383\subsection{Gradient clipping}384385Finally, the optimization trajectory itself can be regularized. When a386loss surface contains cliffs — as in recurrent networks — a single large387gradient can catapult the parameters out of a good basin. Gradient388clipping rescales the gradient whenever its norm exceeds a threshold389$\tau$:390\begin{equation}\label{eq:reg-clip}391\vect{g} \;\leftarrow\;392\begin{cases}393\vect{g} & \text{if } \lVert \vect{g} \rVert_2 \le \tau, \\[2pt]394\dfrac{\tau}{\lVert \vect{g} \rVert_2}\, \vect{g} & \text{otherwise,}395\end{cases}396\end{equation}397preserving the direction of the update while bounding its size. Clipping398does not change the location of minima; it changes which minima the399trajectory can reach, and is standard practice whenever exploding400gradients are a risk.401402\medskip403Taken together, the techniques of this chapter form a layered defence:404penalties shape the hypothesis space, dropout and augmentation randomize405what the network sees, normalization conditions the optimization, and406early stopping bounds how far the fit is allowed to proceed. Modern407practice combines several of them almost by default.408