% ============================================================================
%  Artificial Neural Networks — Methods, Equations and Graphical
%  Representations
%  Author  : Simon-Pierre Boucher — contact@spboucher.ai
%  Chapter 3 : Regularization (chapters/03-regularization.tex)
% ============================================================================
\chapter{Regularization}\label{chap:regularization}

A network that fits its training set perfectly is not, in general, a good
network: what we actually care about is its behaviour on data it has never
seen. Deep networks are typically over-parameterized — they possess enough
capacity to memorize their training set outright — so the gap between
training error and test error must be controlled deliberately.
\emph{Regularization} is the collective name for the techniques that
control this gap. Following Goodfellow, Bengio and
Courville~\cite{goodfellow2016book}, we use the term broadly: a
regularizer is any modification of the model, the objective or the
training procedure whose purpose is to reduce generalization error, even
at the price of a higher training error.

\begin{definition}[Regularization]\label{def:reg}
Let $\Loss_{\mathrm{data}}(\vect{\theta})$ denote the empirical loss of a
model with parameters $\vect{\theta}$. A regularization technique is any
alteration of the learning problem — an additive penalty
$\Loss_{\mathrm{data}} + \lambda\,\Omega(\vect{\theta})$, a stochastic
perturbation of the architecture, a normalization of intermediate
activations, or a constraint on the training trajectory — designed to
decrease the expected loss on unseen data drawn from the same
distribution.
\end{definition}

\section{The Bias--Variance Decomposition}

Why should reducing capacity ever help? The classical answer is the
bias--variance decomposition. Consider a regression setting with targets
generated as $y = f(\vect{x}) + \epsilon$, where $\epsilon$ is zero-mean
noise of variance $\sigma^2$, and let $\hat{f}(\vect{x}; \mathcal{D})$ be
the predictor learned from a random training set $\mathcal{D}$. The
expected squared error at a point $\vect{x}$, averaged over draws of the
training set and of the noise, splits into three terms:
\begin{equation}\label{eq:reg-biasvariance}
\E_{\mathcal{D},\epsilon}\!\left[\bigl(y - \hat{f}(\vect{x};\mathcal{D})\bigr)^2\right]
=
\underbrace{\bigl(f(\vect{x}) - \E_{\mathcal{D}}[\hat{f}(\vect{x};\mathcal{D})]\bigr)^2}_{\text{bias}^2}
+
\underbrace{\E_{\mathcal{D}}\!\left[\bigl(\hat{f}(\vect{x};\mathcal{D}) - \E_{\mathcal{D}}[\hat{f}(\vect{x};\mathcal{D})]\bigr)^2\right]}_{\text{variance}}
+
\underbrace{\sigma^2}_{\text{noise}} .
\end{equation}
The last term is irreducible: no learner can predict the noise. The first
two trade off against each other. A rigid model (high bias) misses
structure in the data; a flexible model fitted to a finite sample (high
variance) tracks the accidents of that particular sample. Every technique
in this chapter can be read as a way of purchasing a large reduction in
variance at the cost of a small increase in bias, so that the sum
in~\eqref{eq:reg-biasvariance} decreases.

\section{Norm Penalties: $L_2$ and $L_1$}

\subsection{$L_2$ regularization (weight decay)}

The most venerable regularizer adds the squared Frobenius norm of the
weight matrices to the data loss:
\begin{equation}\label{eq:reg-l2loss}
\Loss_{\mathrm{total}}(\vect{\theta})
= \Loss_{\mathrm{data}}(\vect{\theta})
+ \frac{\lambda}{2} \sum_{\ell} \bigl\lVert \mat{W}^{(\ell)} \bigr\rVert_F^2 ,
\qquad
\frac{\partial \Loss_{\mathrm{total}}}{\partial \mat{W}^{(\ell)}}
= \frac{\partial \Loss_{\mathrm{data}}}{\partial \mat{W}^{(\ell)}}
+ \lambda \mat{W}^{(\ell)} .
\end{equation}
Substituting the penalized gradient into the SGD step with learning rate
$\eta$ reveals why the method is called \emph{weight decay}:
\begin{equation}\label{eq:reg-l2update}
\mat{W}^{(\ell)} \;\leftarrow\; (1 - \eta\lambda)\,\mat{W}^{(\ell)}
\;-\; \eta\, \frac{\partial \Loss_{\mathrm{data}}}{\partial \mat{W}^{(\ell)}} .
\end{equation}
Each update first shrinks every weight multiplicatively by the factor
$(1-\eta\lambda) < 1$, then applies the usual data-driven correction.
Weights that the data does not actively support are steadily pulled toward
zero. Biases are conventionally excluded from the penalty: they do not
multiply inputs, so penalizing them adds bias without reducing variance.

\begin{remark}[Bayesian reading]\label{rem:reg-bayes}
Minimizing~\eqref{eq:reg-l2loss} is maximum a posteriori estimation under
a zero-mean Gaussian prior on the weights; the $L_1$ penalty
of~\eqref{eq:reg-l1loss} corresponds in the same way to a Laplace prior.
The regularization strength $\lambda$ plays the role of the inverse prior
variance: the stronger our prior belief that weights are small, the harder
we shrink.
\end{remark}

\subsection{$L_1$ regularization and sparsity}

Replacing the squared norm by the absolute-value norm changes the
character of the solution:
\begin{equation}\label{eq:reg-l1loss}
\Loss_{\mathrm{total}}(\vect{\theta})
= \Loss_{\mathrm{data}}(\vect{\theta})
+ \lambda \sum_{\ell} \bigl\lVert \mat{W}^{(\ell)} \bigr\rVert_1 ,
\qquad
\frac{\partial \Loss_{\mathrm{total}}}{\partial \mat{W}^{(\ell)}}
= \frac{\partial \Loss_{\mathrm{data}}}{\partial \mat{W}^{(\ell)}}
+ \lambda \operatorname{sign}\bigl(\mat{W}^{(\ell)}\bigr) .
\end{equation}
The penalty gradient has \emph{constant magnitude} $\lambda$ regardless of
how small a weight already is, so weights whose data gradient cannot
sustain them are driven exactly to zero rather than merely toward it. The
result is a sparse network — an implicit form of feature selection. The
two penalties are often combined (the \emph{elastic net}),
$\lambda_1 \lVert \mat{W} \rVert_1 + \tfrac{\lambda_2}{2}\lVert \mat{W} \rVert_2^2$,
retaining the sparsity of $L_1$ and the grouping stability of $L_2$.

\section{Dropout}

Norm penalties act on parameters; \emph{dropout}, introduced by Srivastava
et al.~\cite{srivastava2014}, acts on the architecture itself. During
training, each unit of a layer is deleted independently at random, so that
no unit can rely on the presence of any particular other unit — the
phenomenon the authors call \emph{co-adaptation} is thereby suppressed.

Formally, let $\vect{a}^{(\ell)}$ be the activation vector of layer $\ell$
and let $p$ be the dropout rate (the probability of deletion). A binary
mask is sampled anew for every training example, and the surviving
activations are rescaled — the \emph{inverted dropout} convention:
\begin{equation}\label{eq:reg-dropout-mask}
m_j^{(\ell)} \sim \mathrm{Bernoulli}(1-p),
\qquad
\tilde{\vect{a}}^{(\ell)}
= \frac{\vect{m}^{(\ell)} \odot \vect{a}^{(\ell)}}{1-p} ,
\end{equation}
and the next layer consumes $\tilde{\vect{a}}^{(\ell)}$ in place of
$\vect{a}^{(\ell)}$. The division by $1-p$ keeps the expected input to
each downstream unit unchanged,
$\E[\tilde{a}_j^{(\ell)}] = a_j^{(\ell)}$, so that at test time the
network is used \emph{without any modification}:
\begin{equation}\label{eq:reg-dropout-test}
\tilde{\vect{a}}^{(\ell)} = \vect{a}^{(\ell)}
\qquad \text{(inference: no mask, no rescaling).}
\end{equation}
Typical rates are $p = 0.5$ for hidden layers and $p \approx 0.2$ for
inputs. Figure~\ref{fig:reg-dropout} contrasts a standard fully connected
network with one realization of its dropout-thinned counterpart.

\begin{figure}[htbp]
  \centering
  \begin{tikzpicture}[scale=0.82, transform shape]
    % ---------- (a) standard network ----------
    \begin{scope}
      \foreach \i in {1,2,3}
        \node[ninput]  (ai\i) at (0, 1.0-\i*1.0) {};
      \foreach \j in {1,...,4}
        \node[nhidden] (ah1\j) at (1.9, 1.5-\j*1.0) {};
      \foreach \j in {1,...,4}
        \node[nhidden] (ah2\j) at (3.8, 1.5-\j*1.0) {};
      \foreach \k in {1,2}
        \node[noutput] (ao\k) at (5.7, 0.5-\k*1.0) {};
      \foreach \i in {1,2,3} \foreach \j in {1,...,4}
        \draw[black!40, semithick] (ai\i) -- (ah1\j);
      \foreach \i in {1,...,4} \foreach \j in {1,...,4}
        \draw[black!40, semithick] (ah1\i) -- (ah2\j);
      \foreach \i in {1,...,4} \foreach \k in {1,2}
        \draw[black!40, semithick] (ah2\i) -- (ao\k);
      \node[etiquette, align=center] at (2.85, -3.6)
        {(a) standard network};
    \end{scope}
    % ---------- (b) after dropout ----------
    \begin{scope}[xshift=8.6cm]
      % surviving units: all inputs, h1 = {1,3}, h2 = {2,4}, all outputs
      \foreach \i in {1,2,3}
        \node[ninput]  (bi\i) at (0, 1.0-\i*1.0) {};
      \foreach \j in {1,3}
        \node[nhidden] (bh1\j) at (1.9, 1.5-\j*1.0) {};
      \foreach \j in {2,4}
        \node[nhidden] (bh2\j) at (3.8, 1.5-\j*1.0) {};
      % dropped units: dashed outline, pale fill
      \foreach \j in {2,4}
        \node[neuron, dashed, draw=black!45, fill=black!5]
          (bh1\j) at (1.9, 1.5-\j*1.0) {};
      \foreach \j in {1,3}
        \node[neuron, dashed, draw=black!45, fill=black!5]
          (bh2\j) at (3.8, 1.5-\j*1.0) {};
      \foreach \k in {1,2}
        \node[noutput] (bo\k) at (5.7, 0.5-\k*1.0) {};
      % edges touching a dropped unit: faint and dashed
      \foreach \i in {1,2,3} \foreach \j in {2,4}
        \draw[black!20, dashed] (bi\i) -- (bh1\j);
      \foreach \i in {1,3} \foreach \j in {1,3}
        \draw[black!20, dashed] (bh1\i) -- (bh2\j);
      \foreach \i in {2,4} \foreach \j in {1,...,4}
        \draw[black!20, dashed] (bh1\i) -- (bh2\j);
      \foreach \i in {1,3} \foreach \k in {1,2}
        \draw[black!20, dashed] (bh2\i) -- (bo\k);
      % active edges between surviving units
      \foreach \i in {1,2,3} \foreach \j in {1,3}
        \draw[black!40, semithick] (bi\i) -- (bh1\j);
      \foreach \i in {1,3} \foreach \j in {2,4}
        \draw[black!40, semithick] (bh1\i) -- (bh2\j);
      \foreach \i in {2,4} \foreach \k in {1,2}
        \draw[black!40, semithick] (bh2\i) -- (bo\k);
      % crosses on dropped units
      \foreach \n in {bh12, bh14, bh21, bh23}{
        \draw[black!60, thick] (\n.north east) -- (\n.south west);
        \draw[black!60, thick] (\n.north west) -- (\n.south east);
      }
      \node[etiquette, align=center] at (2.85, -3.6)
        {(b) after dropout ($p = 0.5$ on hidden layers)};
    \end{scope}
  \end{tikzpicture}
  \caption{Dropout as stochastic architecture perturbation. (a)~The full
  network. (b)~One training-time realization: each hidden unit is deleted
  independently with probability $p$ (crossed out, dashed), together with
  all of its incoming and outgoing connections; surviving activations are
  rescaled by $1/(1-p)$ as in~\eqref{eq:reg-dropout-mask}. A different
  subnetwork is sampled for every example.}
  \label{fig:reg-dropout}
\end{figure}

\begin{remark}[Ensemble interpretation]\label{rem:reg-ensemble}
A network with $N$ droppable units defines $2^N$ thinned subnetworks
sharing one set of weights. Dropout training optimizes the expected loss
over this exponential ensemble, and inference
with~\eqref{eq:reg-dropout-test} approximates the ensemble's geometric-mean
prediction with a single forward pass~\cite{srivastava2014}. The original
paper recommends pairing dropout with a max-norm constraint
$\lVert \vect{w}_j \rVert_2 \le c$ on incoming weight vectors.
\end{remark}

\section{Normalization Layers}

\subsection{Batch Normalization}

Batch Normalization (BN), due to Ioffe and Szegedy~\cite{ioffe2015},
standardizes each pre-activation over the current mini-batch and then
restores expressive freedom through two learned parameters. For a given
unit, let $\mathcal{B} = \{z_1, \dots, z_m\}$ be the values it takes over
a mini-batch of size $m$. The transform is defined by four equations:
\begin{align}
\mu_{\mathcal{B}} &= \frac{1}{m} \sum_{i=1}^{m} z_i
  && \text{(mini-batch mean)} \label{eq:reg-bn-mean}\\
\sigma_{\mathcal{B}}^2 &= \frac{1}{m} \sum_{i=1}^{m}
  \bigl(z_i - \mu_{\mathcal{B}}\bigr)^2
  && \text{(mini-batch variance)} \label{eq:reg-bn-var}\\
\hat{z}_i &= \frac{z_i - \mu_{\mathcal{B}}}
  {\sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}}
  && \text{(normalize)} \label{eq:reg-bn-norm}\\
y_i &= \gamma\, \hat{z}_i + \beta
  && \text{(scale and shift)} \label{eq:reg-bn-scale}
\end{align}
where $\epsilon > 0$ is a small constant for numerical stability. The
learned pair $(\gamma, \beta)$ ensures BN never destroys capacity:
setting $\gamma = \sqrt{\sigma_{\mathcal{B}}^2 + \epsilon}$ and
$\beta = \mu_{\mathcal{B}}$ recovers the identity map. In convolutional
networks the statistics are computed per channel, jointly over the batch
and all spatial positions, with one pair $(\gamma_c, \beta_c)$ per
channel, so as to respect the weight sharing of the convolution.

At \emph{inference} no mini-batch is available. Exponential moving
averages of the training statistics are accumulated,
$\hat{\mu} \leftarrow \mathrm{EMA}(\mu_{\mathcal{B}})$ and
$\hat{\sigma}^2 \leftarrow \mathrm{EMA}(\sigma_{\mathcal{B}}^2)$, and the
whole layer becomes a fixed affine map,
\begin{equation}\label{eq:reg-bn-inference}
y = \gamma\, \frac{z - \hat{\mu}}{\sqrt{\hat{\sigma}^2 + \epsilon}} + \beta ,
\end{equation}
which can be folded into the preceding linear or convolutional layer at
no runtime cost. In practice BN permits substantially larger learning
rates, reduces sensitivity to initialization, and contributes a mild
regularizing effect through the noise of mini-batch statistics — which is
why it appears in this chapter. Its main weaknesses are its dependence on
a sufficiently large batch size and its awkwardness in recurrent or
online settings.

\subsection{Layer Normalization}

Layer Normalization (LN) transposes the computation: the statistics are
taken over the \emph{features of a single example} rather than over the
batch. For a vector $\vect{z} \in \R^d$,
\begin{equation}\label{eq:reg-ln}
\mu = \frac{1}{d} \sum_{j=1}^{d} z_j ,
\qquad
\sigma^2 = \frac{1}{d} \sum_{j=1}^{d} (z_j - \mu)^2 ,
\qquad
\mathrm{LN}(\vect{z})
= \vect{\gamma} \odot \frac{\vect{z} - \mu}{\sqrt{\sigma^2 + \epsilon}}
+ \vect{\beta} ,
\end{equation}
with learned vectors $\vect{\gamma}, \vect{\beta} \in \R^d$.

\begin{remark}[BN versus LN: the normalization axis]\label{rem:reg-bnln}
BN normalizes each feature across examples
(equations~\eqref{eq:reg-bn-mean}--\eqref{eq:reg-bn-scale} run over the
batch index $i$); LN normalizes each example across features
(equation~\eqref{eq:reg-ln} runs over the feature index $j$). Because LN
is independent of the batch, it behaves identically at training and test
time, works at batch size one, and handles variable-length sequences —
the reasons it, rather than BN, is the standard choice in recurrent
networks and Transformers.
\end{remark}

\section{Regularizing the Training Procedure}

\subsection{Early stopping}

The simplest regularizer costs nothing: monitor the loss on a held-out
validation set and stop training when it ceases to improve. The training
loss decreases essentially monotonically, but the validation loss
typically traces a U-shape — beyond its minimum, further optimization
fits sampling noise rather than structure. Figure~\ref{fig:reg-earlystop}
illustrates the regime change; in practice one checkpoints the parameters
at each validation improvement and restores the best checkpoint when a
\emph{patience} budget of non-improving epochs is exhausted.

\begin{figure}[htbp]
  \centering
  \begin{tikzpicture}
    \begin{axis}[
        width=0.78\textwidth, height=6.2cm,
        xlabel={epoch}, ylabel={loss},
        xmin=0, xmax=100, ymin=0, ymax=1.15,
        axis lines=left,
        legend style={draw=none, fill=none, at={(0.97,0.95)},
                      anchor=north east, font=\small},
        domain=0:100, samples=200,
      ]
      \addplot[cinput, thick] {0.10 + 0.90*exp(-x/15)};
      \addlegendentry{training loss}
      \addplot[coutput, thick] {0.21 + 0.79*exp(-x/15) + 0.004*x};
      \addlegendentry{validation loss}
      \draw[black!60, dashed, thick] (axis cs:37.9,0) -- (axis cs:37.9,1.08);
      \node[etiquette, anchor=south, rotate=90, text=black!75]
        at (axis cs:37.9,0.72) {early stopping point};
      \addplot[coutput, only marks, mark=*, mark size=1.8pt]
        coordinates {(37.9,0.425)};
    \end{axis}
  \end{tikzpicture}
  \caption{Early stopping. The training loss (blue) decreases throughout,
  while the validation loss (red) reaches a minimum and then rises as the
  network begins to overfit. Training is halted at the dashed line and
  the parameters of the best validation checkpoint are retained.}
  \label{fig:reg-earlystop}
\end{figure}

\begin{property}[Early stopping as implicit weight decay]\label{prop:reg-es}
For a quadratic loss optimized by gradient descent from
$\vect{\theta}_0 = \vect{0}$, halting after $\tau$ steps with learning
rate $\eta$ yields a solution equivalent to fully optimizing the
$L_2$-penalized objective~\eqref{eq:reg-l2loss} with regularization
strength $\lambda \approx 1/(\eta\tau)$: a trajectory of limited length
cannot travel far along low-curvature directions, exactly as weight decay
suppresses them~\cite{goodfellow2016book}.
\end{property}

\subsection{Data augmentation}

Rather than constraining the model, one can enlarge the data. If
$\mathcal{T}$ is a family of label-preserving transformations —
translations, crops, horizontal flips, small rotations, photometric
jitter for images; time-stretching or noise injection for audio —
training on transformed samples replaces the empirical loss by
$\E_{t \sim \mathcal{T}}\bigl[\Loss\bigl(f(t(\vect{x})), \vect{y}\bigr)\bigr]$,
encoding the invariances of the task directly into the training
distribution. Augmentation is often the single most effective
regularizer available for perception tasks, because it injects genuine
prior knowledge rather than generic smoothness.

\subsection{Label smoothing}

Hard one-hot targets push the softmax toward infinite logit gaps and
overconfident predictions. Label smoothing blends the one-hot vector
$\vect{y}$ with the uniform distribution over the $K$ classes:
\begin{equation}\label{eq:reg-labelsmooth}
\vect{y}^{\mathrm{smooth}} = (1 - \varepsilon)\, \vect{y}
+ \frac{\varepsilon}{K} \, \vect{1} ,
\qquad \varepsilon \in [0, 1),
\end{equation}
with $\varepsilon = 0.1$ a common choice. The correct class keeps
probability $1 - \varepsilon + \varepsilon/K$; every other class receives
$\varepsilon/K$. The cross-entropy gradient then stops rewarding
unbounded confidence, which improves calibration and often test accuracy.

\subsection{Gradient clipping}

Finally, the optimization trajectory itself can be regularized. When a
loss surface contains cliffs — as in recurrent networks — a single large
gradient can catapult the parameters out of a good basin. Gradient
clipping rescales the gradient whenever its norm exceeds a threshold
$\tau$:
\begin{equation}\label{eq:reg-clip}
\vect{g} \;\leftarrow\;
\begin{cases}
\vect{g} & \text{if } \lVert \vect{g} \rVert_2 \le \tau, \\[2pt]
\dfrac{\tau}{\lVert \vect{g} \rVert_2}\, \vect{g} & \text{otherwise,}
\end{cases}
\end{equation}
preserving the direction of the update while bounding its size. Clipping
does not change the location of minima; it changes which minima the
trajectory can reach, and is standard practice whenever exploding
gradients are a risk.

\medskip
Taken together, the techniques of this chapter form a layered defence:
penalties shape the hypothesis space, dropout and augmentation randomize
what the network sees, normalization conditions the optimization, and
early stopping bounds how far the fit is allowed to proceed. Modern
practice combines several of them almost by default.
