spb/artificial-neural-networks-book Public

1% ============================================================================2%  Artificial Neural Networks — Methods, Equations and Graphical3%  Representations4%  Author  : Simon-Pierre Boucher — contact@spboucher.ai5%  Chapter 2 : Training Neural Networks (chapters/02-training.tex)6% ============================================================================7\chapter{Training Neural Networks}\label{chap:training}89The previous chapter established what a feed-forward network computes; this10chapter establishes how its parameters are found. Training is cast as the11minimization of an empirical risk, and virtually every modern network is12trained by the same recipe: compute the gradient of the loss with respect to13every parameter by \emph{backpropagation}~\cite{rumelhart1986}, then descend14along that gradient with a first-order update rule. We derive the15backpropagation equations in full, present the algorithm in pseudocode, and16then survey the family of optimizers — from plain stochastic gradient17descent to Adam~\cite{kingma2015adam} — together with the learning-rate18schedules and weight-initialization schemes that make deep training stable19in practice.2021% ============================================================================22\section{The Learning Problem}2324\begin{definition}[Empirical risk minimization]\label{def:train-erm}25Let $f(\cdot\,;\vect{\theta})$ be a network with parameters $\vect{\theta}$,26let $\Loss(\hat{\vect{y}},\vect{y})$ be a per-example loss, and let27$\mathcal{D}$ be the data distribution. The \emph{(true) risk} and its28Monte-Carlo estimate on a training set29$\{(\vect{x}_i,\vect{y}_i)\}_{i=1}^{n}$, the \emph{empirical risk}, are30\begin{equation}\label{eq:train-risk}31  R(\vect{\theta})32    = \E_{(\vect{x},\vect{y})\sim\mathcal{D}}33      \bigl[\Loss\bigl(f(\vect{x};\vect{\theta}),\vect{y}\bigr)\bigr],34  \qquad35  \widehat{R}(\vect{\theta})36    = \frac{1}{n}\sum_{i=1}^{n}37      \Loss\bigl(f(\vect{x}_i;\vect{\theta}),\vect{y}_i\bigr).38\end{equation}39Training solves40$\vect{\theta}^{\star} \in \argmin_{\vect{\theta}} \widehat{R}(\vect{\theta})$.41\end{definition}4243The choice of $\Loss$ encodes the task. We review the two workhorses —44squared error for regression, cross-entropy for classification — and the45single most useful gradient identity in deep learning.4647\subsection{Regression: mean squared error}4849For scalar targets $y_i \in \R$ and predictions $\hat{y}_i$, the50\emph{mean squared error} and its gradient are51\begin{equation}\label{eq:train-mse}52  \Loss_{\mathrm{MSE}}53    = \frac{1}{n}\sum_{i=1}^{n}\bigl(y_i - \hat{y}_i\bigr)^{2},54  \qquad55  \frac{\partial \Loss_{\mathrm{MSE}}}{\partial \hat{y}_i}56    = -\frac{2}{n}\,\bigl(y_i - \hat{y}_i\bigr).57\end{equation}58Minimizing~\eqref{eq:train-mse} is maximum-likelihood estimation under59Gaussian observation noise; the quadratic growth makes it sensitive to60outliers, which motivates robust alternatives such as the mean absolute61error $\frac{1}{n}\sum_i |y_i-\hat y_i|$ (Laplace likelihood, whose62minimizer is the conditional median) and the Huber loss, quadratic for63small residuals and linear in the tails.6465\subsection{Classification: cross-entropy}6667For binary labels $y\in\{0,1\}$ with $\hat{y}=\sigma(z)$ produced by a68sigmoid over the logit $z$, the \emph{binary cross-entropy} is69\begin{equation}\label{eq:train-bce}70  \Loss_{\mathrm{BCE}}71    = -\frac{1}{n}\sum_{i=1}^{n}72      \Bigl[\,y_i\ln\hat{y}_i + (1-y_i)\ln\bigl(1-\hat{y}_i\bigr)\Bigr],73  \qquad74  \frac{\partial \Loss_{\mathrm{BCE}}}{\partial z} = \hat{y}-y .75\end{equation}76For $K$-way classification with one-hot target $\vect{y}$ and77$\hat{\vect{y}}=\softmax(\vect{z})$, the \emph{categorical cross-entropy}78is79\begin{equation}\label{eq:train-ce}80  \Loss_{\mathrm{CE}}81    = -\sum_{k=1}^{K} y_k \ln \hat{y}_k ,82\end{equation}83which equals, up to the (constant) entropy of $\vect{y}$, the84Kullback--Leibler divergence $\KL(\vect{y}\,\|\,\hat{\vect{y}})$.8586\begin{property}[Softmax--cross-entropy gradient]\label{prop:train-ce-grad}87With $\hat{\vect{y}}=\softmax(\vect{z})$ and $\Loss_{\mathrm{CE}}$ as88in~\eqref{eq:train-ce},89\begin{equation}\label{eq:train-ce-grad}90  \frac{\partial \Loss_{\mathrm{CE}}}{\partial \vect{z}}91    \;=\; \hat{\vect{y}} - \vect{y}.92\end{equation}93\end{property}9495\begin{proof}[Sketch]96The softmax Jacobian is97$\partial\hat{y}_i/\partial z_j = \hat{y}_i(\delta_{ij}-\hat{y}_j)$ with98$\delta_{ij}$ the Kronecker delta. Chaining it against99$\partial\Loss/\partial\hat{y}_i = -y_i/\hat{y}_i$ gives100$\partial\Loss/\partial z_j101 = \sum_i (-y_i/\hat{y}_i)\,\hat{y}_i(\delta_{ij}-\hat{y}_j)102 = -y_j + \hat{y}_j\sum_i y_i = \hat{y}_j - y_j$,103since $\sum_i y_i = 1$.104\end{proof}105106\begin{remark}107The cancellation in~\eqref{eq:train-ce-grad} is not a coincidence: it holds108for every matched pair of a canonical link and its exponential-family109negative log-likelihood (sigmoid with binary cross-entropy110in~\eqref{eq:train-bce}, identity with MSE). The practical consequence is111the absence of saturating $\sigma'$ factors at the output layer: a112confidently wrong prediction still receives a large gradient.113\end{remark}114115% ============================================================================116\section{Backpropagation}117118Consider the $L$-layer network of Chapter~\ref{chap:foundations},119\begin{equation}\label{eq:train-forward}120  \vect{z}^{(\ell)} = \mat{W}^{(\ell)}\vect{a}^{(\ell-1)}+\vect{b}^{(\ell)},121  \qquad122  \vect{a}^{(\ell)} = \varphi\bigl(\vect{z}^{(\ell)}\bigr),123  \qquad \ell = 1,\dots,L,124\end{equation}125with $\vect{a}^{(0)}=\vect{x}$ and a scalar loss126$\Loss\bigl(\vect{a}^{(L)},\vect{y}\bigr)$. Backpropagation is reverse-mode127automatic differentiation applied to this composition: it computes128$\partial\Loss/\partial\vect{\theta}$ for \emph{all} parameters at the cost129of a constant number of forward passes. The central object is the130\emph{error signal} of layer $\ell$,131\begin{equation}\label{eq:train-delta-def}132  \vect{\delta}^{(\ell)}133    \;\equiv\; \frac{\partial \Loss}{\partial \vect{z}^{(\ell)}}134    \;\in\; \R^{n_\ell}.135\end{equation}136137\begin{theorem}[Backpropagation equations~\cite{rumelhart1986}]138\label{thm:train-backprop}139For the network~\eqref{eq:train-forward}, the error signals and parameter140gradients satisfy141\begin{align}142  \vect{\delta}^{(L)}143    &= \nabla_{\vect{a}^{(L)}}\Loss144       \,\odot\, \varphi'\bigl(\vect{z}^{(L)}\bigr),145  \label{eq:train-bp-out}\\[2pt]146  \vect{\delta}^{(\ell)}147    &= \Bigl(\mat{W}^{(\ell+1)\transp}\,\vect{\delta}^{(\ell+1)}\Bigr)148       \odot \varphi'\bigl(\vect{z}^{(\ell)}\bigr),149  \quad \ell = L-1,\dots,1,150  \label{eq:train-bp-rec}\\[2pt]151  \frac{\partial \Loss}{\partial \mat{W}^{(\ell)}}152    &= \vect{\delta}^{(\ell)}\,\vect{a}^{(\ell-1)\transp},153  \qquad154  \frac{\partial \Loss}{\partial \vect{b}^{(\ell)}}155    = \vect{\delta}^{(\ell)} .156  \label{eq:train-bp-w}157\end{align}158\end{theorem}159160\begin{proof}[Sketch]161Equation~\eqref{eq:train-bp-out} is the chain rule through162$\vect{a}^{(L)}=\varphi(\vect{z}^{(L)})$. For~\eqref{eq:train-bp-rec},163each $z^{(\ell+1)}_k$ depends on $z^{(\ell)}_j$ through164$W^{(\ell+1)}_{kj}\varphi'(z^{(\ell)}_j)$, so summing over downstream165paths,166\[167\delta^{(\ell)}_j168  = \sum_k \delta^{(\ell+1)}_k W^{(\ell+1)}_{kj}\,169    \varphi'\bigl(z^{(\ell)}_j\bigr),170\]171which is~\eqref{eq:train-bp-rec} in matrix form.172For~\eqref{eq:train-bp-w}, note173$\partial z^{(\ell)}_j / \partial W^{(\ell)}_{ji} = a^{(\ell-1)}_i$ and174$\partial z^{(\ell)}_j / \partial b^{(\ell)}_j = 1$.175\end{proof}176177The name of the algorithm is visible in~\eqref{eq:train-bp-rec}: the error178is propagated \emph{backwards} through the transposes of the forward weight179matrices. With softmax output and cross-entropy loss,180Property~\ref{prop:train-ce-grad} replaces~\eqref{eq:train-bp-out} directly181by $\vect{\delta}^{(L)} = \hat{\vect{y}}-\vect{y}$.182Figure~\ref{fig:train-compgraph} shows the flow of both passes on the183computational graph, and Algorithm~\ref{alg:train-backprop} states the full184procedure for one mini-batch.185186\begin{figure}[htbp]187  \centering188  \begin{tikzpicture}189    % ---- forward row -------------------------------------------------190    \node[blocinput,  minimum width=1.15cm] (x)  at (0,0)    {$\vect{x}$};191    \node[blochidden, minimum width=1.15cm] (z1) at (2.35,0) {$\vect{z}^{(1)}$};192    \node[blochidden, minimum width=1.15cm] (a1) at (4.70,0) {$\vect{a}^{(1)}$};193    \node[blochidden, minimum width=1.15cm] (z2) at (7.05,0) {$\vect{z}^{(2)}$};194    \node[blocoutput, minimum width=1.15cm] (a2) at (9.40,0) {$\hat{\vect{y}}$};195    \node[bloc,       minimum width=1.15cm] (Ls) at (11.75,0){$\Loss$};196    % ---- parameter and target nodes ------------------------------------197    \node[mem, minimum width=1.5cm] (W1) at (2.35,1.9)198      {$\mat{W}^{(1)},\vect{b}^{(1)}$};199    \node[mem, minimum width=1.5cm] (W2) at (7.05,1.9)200      {$\mat{W}^{(2)},\vect{b}^{(2)}$};201    \node[blocinput, minimum width=1.15cm] (y) at (11.75,1.9) {$\vect{y}$};202    % ---- forward arrows ------------------------------------------------203    \draw[fleche] (x)  -- (z1);204    \draw[fleche] (z1) -- node[above, etiquette] {$\varphi$} (a1);205    \draw[fleche] (a1) -- (z2);206    \draw[fleche] (z2) -- node[above, etiquette] {$\varphi$} (a2);207    \draw[fleche] (a2) -- (Ls);208    \draw[fleche] (W1) -- (z1);209    \draw[fleche] (W2) -- (z2);210    \draw[fleche] (y)  -- (Ls);211    % ---- backward (dashed) arrows, routed below ------------------------212    \draw[flechep, coutput] (Ls.south) to[bend left=35]213      node[below, etiquette] {$\nabla_{\hat{\vect{y}}}\Loss$} (a2.south);214    \draw[flechep, coutput] (a2.south) to[bend left=35]215      node[below, etiquette] {$\vect{\delta}^{(2)}$} (z2.south);216    \draw[flechep, coutput] (z2.south) to[bend left=35]217      node[below, etiquette] {$\mat{W}^{(2)\transp}\vect{\delta}^{(2)}$}218      (a1.south);219    \draw[flechep, coutput] (a1.south) to[bend left=35]220      node[below, etiquette] {$\vect{\delta}^{(1)}$} (z1.south);221    % ---- gradients to parameters ---------------------------------------222    \draw[flechep, coutput] (z1.north east) to[bend right=30]223      node[right, etiquette, xshift=2pt]224      {$\vect{\delta}^{(1)}\vect{a}^{(0)\transp}$} (W1.east);225    \draw[flechep, coutput] (z2.north east) to[bend right=30]226      node[right, etiquette, xshift=2pt]227      {$\vect{\delta}^{(2)}\vect{a}^{(1)\transp}$} (W2.east);228  \end{tikzpicture}229  \caption{Computational graph of a two-layer network. Solid arrows:230    forward pass~\eqref{eq:train-forward}. Dashed red arrows: backward231    pass — the error signals $\vect{\delta}^{(\ell)}$ of232    Theorem~\ref{thm:train-backprop} flow from the loss back through the233    layers, branching off to the parameter234    gradients~\eqref{eq:train-bp-w}.}235  \label{fig:train-compgraph}236\end{figure}237238\begin{algorithm}[htbp]239  \caption{Backpropagation with mini-batch gradient descent}240  \label{alg:train-backprop}241  \begin{algorithmic}[1]242    \Require mini-batch $\{(\vect{x}_i,\vect{y}_i)\}_{i=1}^{m}$, parameters243      $\{\mat{W}^{(\ell)},\vect{b}^{(\ell)}\}_{\ell=1}^{L}$, learning rate244      $\eta$245    \For{$i = 1$ \textbf{to} $m$} \Comment{forward pass}246      \State $\vect{a}^{(0)} \gets \vect{x}_i$247      \For{$\ell = 1$ \textbf{to} $L$}248        \State $\vect{z}^{(\ell)} \gets249          \mat{W}^{(\ell)}\vect{a}^{(\ell-1)} + \vect{b}^{(\ell)}$;250          \quad $\vect{a}^{(\ell)} \gets \varphi(\vect{z}^{(\ell)})$251      \EndFor252      \State $\vect{\delta}^{(L)} \gets253        \nabla_{\vect{a}^{(L)}}\Loss \odot \varphi'(\vect{z}^{(L)})$254        \Comment{backward pass; $\hat{\vect{y}}-\vect{y}$ for softmax+CE}255      \For{$\ell = L-1$ \textbf{down to} $1$}256        \State $\vect{\delta}^{(\ell)} \gets257          \bigl(\mat{W}^{(\ell+1)\transp}\vect{\delta}^{(\ell+1)}\bigr)258          \odot \varphi'(\vect{z}^{(\ell)})$259      \EndFor260      \State accumulate261        $\Delta\mat{W}^{(\ell)} \mathrel{+}=262           \vect{\delta}^{(\ell)}\vect{a}^{(\ell-1)\transp}$,263        \; $\Delta\vect{b}^{(\ell)} \mathrel{+}= \vect{\delta}^{(\ell)}$264        \textbf{for all} $\ell$265    \EndFor266    \For{$\ell = 1$ \textbf{to} $L$} \Comment{gradient step}267      \State $\mat{W}^{(\ell)} \gets268        \mat{W}^{(\ell)} - \dfrac{\eta}{m}\,\Delta\mat{W}^{(\ell)}$;269        \quad270        $\vect{b}^{(\ell)} \gets271        \vect{b}^{(\ell)} - \dfrac{\eta}{m}\,\Delta\vect{b}^{(\ell)}$272    \EndFor273  \end{algorithmic}274\end{algorithm}275276\begin{remark}[Cost]277One backward pass costs the same order as one forward pass,278$O\bigl(\sum_\ell n_\ell n_{\ell-1}\bigr)$: the full gradient of a scalar279with respect to $P$ parameters is obtained for $O(1)$ — not $O(P)$ —280forward-pass equivalents. This efficiency of reverse-mode differentiation281is what makes deep learning computationally feasible.282\end{remark}283284\begin{remark}[Vanishing and exploding gradients]\label{rem:train-vanish}285The recursion~\eqref{eq:train-bp-rec} multiplies a $\varphi'$ factor and a286weight matrix at every layer. With sigmoid activations,287$\sigma'(z)\le 1/4$, so error signals shrink at least geometrically with288depth; with large weights they can instead grow without bound. This289vanishing/exploding behaviour motivates ReLU-family activations, careful290initialization (Section~\ref{sec:train-init}), normalization layers and291residual connections, treated in later chapters.292\end{remark}293294% ============================================================================295\section{First-Order Optimizers}296297Throughout this section $\vect{\theta}_t$ denotes the parameters at step298$t$, $\vect{g}_t = \nabla_{\vect{\theta}}\Loss(\vect{\theta}_t)$ the299mini-batch gradient, $\eta$ the learning rate, and all operations on300vectors are elementwise.301302\subsection{Stochastic gradient descent and momentum}303304\emph{Stochastic gradient descent} (SGD) applies the elementary update305\begin{equation}\label{eq:train-sgd}306  \vect{\theta}_{t+1} = \vect{\theta}_t - \eta\,\vect{g}_t .307\end{equation}308Classical stochastic-approximation theory guarantees convergence when the309step sizes satisfy $\sum_t \eta_t = \infty$ and $\sum_t \eta_t^2 < \infty$310\cite{goodfellow2016book}.311312\emph{Momentum} (Polyak's heavy ball) accumulates an exponentially weighted313velocity,314\begin{equation}\label{eq:train-momentum}315  \vect{v}_t = \beta\,\vect{v}_{t-1} + \vect{g}_t ,316  \qquad317  \vect{\theta}_{t+1} = \vect{\theta}_t - \eta\,\vect{v}_t ,318\end{equation}319with $\beta \approx 0.9$. Directions in which successive gradients agree320are amplified by up to $1/(1-\beta)$, while oscillating components cancel —321precisely the geometry of Figure~\ref{fig:train-trajectories}.322\emph{Nesterov's accelerated gradient} evaluates the gradient at a323look-ahead point,324\begin{equation}\label{eq:train-nesterov}325  \vect{v}_t = \beta\,\vect{v}_{t-1}326    + \nabla_{\vect{\theta}}327      \Loss\bigl(\vect{\theta}_t - \eta\beta\,\vect{v}_{t-1}\bigr),328  \qquad329  \vect{\theta}_{t+1} = \vect{\theta}_t - \eta\,\vect{v}_t ,330\end{equation}331letting the update ``see'' where it is heading and correct in advance; for332smooth convex objectives it attains the optimal $O(1/t^2)$ convergence333rate.334335\begin{figure}[htbp]336  \centering337  \begin{tikzpicture}338    \begin{axis}[339        width=0.88\textwidth, height=6.2cm,340        xmin=-10, xmax=1.5, ymin=-2.4, ymax=2.4,341        xlabel={$\theta_1$}, ylabel={$\theta_2$},342        xlabel near ticks, ylabel near ticks,343        tick label style={font=\scriptsize},344        label style={font=\small},345        legend style={font=\scriptsize, at={(0.98,0.04)},346                      anchor=south east, draw=black!30},347        legend cell align=left,348      ]349      % loss contours of f = x^2/20 + y^2 (anisotropic bowl)350      \addplot[domain=0:360, samples=91, smooth, black!25, forget plot]351        ({2.828*cos(x)}, {0.632*sin(x)});352      \addplot[domain=0:360, samples=91, smooth, black!25, forget plot]353        ({4.899*cos(x)}, {1.095*sin(x)});354      \addplot[domain=0:360, samples=91, smooth, black!25, forget plot]355        ({6.928*cos(x)}, {1.549*sin(x)});356      \addplot[domain=0:360, samples=91, smooth, black!25, forget plot]357        ({8.944*cos(x)}, {2.000*sin(x)});358      \addplot[domain=0:360, samples=91, smooth, black!25, forget plot]359        ({11.662*cos(x)}, {2.608*sin(x)});360      % SGD trajectory (zigzag)361      \addplot[coutput, thick, mark=*, mark size=1.1pt] coordinates {362        (-8.500,1.800) (-7.777,-1.260) (-7.116,0.882) (-6.512,-0.617)363        (-5.958,0.432) (-5.452,-0.303) (-4.988,0.212) (-4.564,-0.148)364        (-4.176,0.104) (-3.821,-0.073) (-3.496,0.051) (-3.199,-0.036)365        (-2.927,0.025) (-2.679,-0.017) (-2.451,0.012) (-2.243,-0.009)366        (-2.052,0.006)367      };368      \addlegendentry{SGD~\eqref{eq:train-sgd}}369      % Momentum trajectory (smooth)370      \addplot[cinput, thick, mark=*, mark size=1.1pt] coordinates {371        (-8.500,1.800) (-8.381,1.296) (-8.157,0.480) (-7.840,-0.390)372        (-7.446,-1.063) (-6.987,-1.371) (-6.476,-1.264) (-5.925,-0.815)373        (-5.347,-0.182) (-4.751,0.439) (-4.149,0.874) (-3.548,1.022)374        (-2.959,0.868) (-2.386,0.487) (-1.838,0.007) (-1.318,-0.426)375        (-0.832,-0.697)376      };377      \addlegendentry{Momentum~\eqref{eq:train-momentum}}378      % optimum379      \addplot[only marks, mark=star, mark size=3.2pt, black]380        coordinates {(0,0)};381      \addlegendentry{minimum $\vect{\theta}^{\star}$}382    \end{axis}383  \end{tikzpicture}384  \caption{Sixteen steps of SGD and momentum on the anisotropic quadratic385    $\Loss(\vect{\theta}) = \theta_1^2/20 + \theta_2^2$ (grey level sets),386    both computed numerically from the same starting point. SGD (red)387    oscillates across the narrow valley while creeping along the shallow388    direction; momentum (blue) damps the oscillation and accelerates along389    the valley floor.}390  \label{fig:train-trajectories}391\end{figure}392393\subsection{Adaptive methods: AdaGrad, RMSProp, Adam}394395Adaptive methods give every coordinate its own effective learning rate,396scaled by the history of gradient magnitudes. \emph{AdaGrad} accumulates397the squared gradients,398\begin{equation}\label{eq:train-adagrad}399  \vect{G}_t = \vect{G}_{t-1} + \vect{g}_t^{2},400  \qquad401  \vect{\theta}_{t+1}402    = \vect{\theta}_t403      - \frac{\eta}{\sqrt{\vect{G}_t}+\epsilon}\odot\vect{g}_t ,404\end{equation}405so rarely active (sparse) coordinates receive large steps. Because406$\vect{G}_t$ grows monotonically, however, the effective step size decays407to zero. \emph{RMSProp} repairs this by replacing the sum with an408exponential moving average,409\begin{equation}\label{eq:train-rmsprop}410  \E[\vect{g}^2]_t411    = \rho\,\E[\vect{g}^2]_{t-1} + (1-\rho)\,\vect{g}_t^{2},412  \qquad413  \vect{\theta}_{t+1}414    = \vect{\theta}_t415      - \frac{\eta}{\sqrt{\E[\vect{g}^2]_t}+\epsilon}\odot\vect{g}_t ,416\end{equation}417with $\rho\approx 0.9$, so that stale gradients are forgotten.418419\emph{Adam}~\cite{kingma2015adam} (\emph{adaptive moment estimation})420combines the momentum idea~\eqref{eq:train-momentum} — an exponential421moving average of the gradient, the first moment — with the RMSProp422idea~\eqref{eq:train-rmsprop} — an exponential moving average of its423square, the second moment — and corrects the initialization bias of both:424\begin{align}425  \vect{m}_t &= \beta_1\,\vect{m}_{t-1} + (1-\beta_1)\,\vect{g}_t ,426  \qquad427  \vect{v}_t = \beta_2\,\vect{v}_{t-1} + (1-\beta_2)\,\vect{g}_t^{2},428  \label{eq:train-adam-moments}\\[2pt]429  \hat{\vect{m}}_t &= \frac{\vect{m}_t}{1-\beta_1^{\,t}} ,430  \qquad431  \hat{\vect{v}}_t = \frac{\vect{v}_t}{1-\beta_2^{\,t}} ,432  \label{eq:train-adam-bias}\\[2pt]433  \vect{\theta}_{t+1}434    &= \vect{\theta}_t435       - \eta\,\frac{\hat{\vect{m}}_t}{\sqrt{\hat{\vect{v}}_t}+\epsilon} .436  \label{eq:train-adam-update}437\end{align}438The standard defaults are $\eta = 10^{-3}$, $\beta_1 = 0.9$,439$\beta_2 = 0.999$ and $\epsilon = 10^{-8}$.440441\begin{remark}[Why the bias correction matters]442Since $\vect{m}_0=\vect{v}_0=\vect{0}$, the raw averages443in~\eqref{eq:train-adam-moments} are biased toward zero for small $t$:444$\E[\vect{m}_t]\approx(1-\beta_1^{\,t})\,\E[\vect{g}_t]$. Dividing by445$(1-\beta_1^{\,t})$ and $(1-\beta_2^{\,t})$446in~\eqref{eq:train-adam-bias} removes this bias exactly; without it the447first updates would be far too small — dramatically so for448$\vect{v}_t$, whose decay rate $\beta_2 = 0.999$ makes the bias persist449for roughly a thousand steps. The ratio450$\hat{\vect{m}}_t/\sqrt{\hat{\vect{v}}_t}$ acts as a per-coordinate451signal-to-noise estimate, and the magnitude of each update is bounded by452approximately $\eta$ regardless of the gradient scale, making453Adam invariant to gradient rescaling. The decoupled-weight-decay variant454AdamW, which applies the $L_2$ shrinkage outside the adaptive rescaling,455is the default optimizer for modern Transformer models.456\end{remark}457458% ============================================================================459\section{Learning-Rate Schedules}460461The learning rate is the single most important hyperparameter of the462updates above, and it is rarely held constant. Writing $\eta_t$ for the463rate at step $t$ over a horizon of $T$ steps, the three standard schedules464are465\begin{align}466  \eta_t &= \eta_0\,\gamma^{\lfloor t/s \rfloor},467  \qquad 0<\gamma<1468  &&\text{(step decay, factor $\gamma$ every $s$ steps)},469  \label{eq:train-step-decay}\\[2pt]470  \eta_t &= \frac{\eta_0}{2}471    \Bigl(1+\cos\frac{\pi t}{T}\Bigr)472  &&\text{(cosine annealing)},473  \label{eq:train-cosine}\\[2pt]474  \eta_t &=475  \begin{cases}476    \eta_0\, t/T_w & t \le T_w\\[2pt]477    \dfrac{\eta_0}{2}478      \Bigl(1+\cos\dfrac{\pi (t-T_w)}{T-T_w}\Bigr) & t > T_w479  \end{cases}480  &&\text{(linear warmup, then cosine)}.481  \label{eq:train-warmup}482\end{align}483Warmup~\eqref{eq:train-warmup} protects the early phase of training — when484Adam's second-moment estimate $\hat{\vect{v}}_t$ is still noisy — from485destructively large steps, and is standard practice for Transformers; the486three profiles are compared in Figure~\ref{fig:train-schedules}.487488\begin{figure}[htbp]489  \centering490  \begin{tikzpicture}491    \begin{axis}[492        width=0.8\textwidth, height=5.2cm,493        xmin=0, xmax=100, ymin=0, ymax=0.115,494        xlabel={training step $t$ (\% of horizon $T$)},495        ylabel={$\eta_t$},496        xlabel near ticks, ylabel near ticks,497        tick label style={font=\scriptsize},498        label style={font=\small},499        legend style={font=\scriptsize, draw=black!30},500        legend cell align=left,501      ]502      \addplot[coutput, thick, const plot, domain=0:100, samples=201]503        {0.1 * 0.5^(floor(x/30))};504      \addlegendentry{step decay~\eqref{eq:train-step-decay}}505      \addplot[cinput, thick, domain=0:100, samples=201]506        {0.05*(1+cos(deg(pi*x/100)))};507      \addlegendentry{cosine annealing~\eqref{eq:train-cosine}}508      \addplot[cgate, thick, domain=0:10, samples=21, forget plot]509        {0.1*x/10};510      \addplot[cgate, thick, domain=10:100, samples=181]511        {0.05*(1+cos(deg(pi*(x-10)/90)))};512      \addlegendentry{warmup + cosine~\eqref{eq:train-warmup}}513    \end{axis}514  \end{tikzpicture}515  \caption{Learning-rate schedules with $\eta_0=0.1$: step decay516    ($\gamma=0.5$, $s=0.3\,T$), cosine annealing, and linear warmup over517    the first $10\%$ of training followed by cosine annealing.}518  \label{fig:train-schedules}519\end{figure}520521% ============================================================================522\section{Weight Initialization}\label{sec:train-init}523524Remark~\ref{rem:train-vanish} showed that signals are multiplied by a525weight matrix at every layer; initialization must therefore keep the526variance of activations and of backpropagated gradients approximately527constant with depth~\cite{goodfellow2016book}. For a layer with528$n_{\mathrm{in}}$ inputs and $n_{\mathrm{out}}$ outputs,529\emph{Xavier/Glorot} initialization — appropriate for symmetric, roughly530linear-around-zero activations such as $\tanh$ — balances both passes:531\begin{equation}\label{eq:train-xavier}532  \operatorname{Var}\bigl(W_{ij}\bigr)533    = \frac{2}{n_{\mathrm{in}}+n_{\mathrm{out}}},534  \qquad\text{e.g.}\quad535  W_{ij} \sim536  \mathcal{U}\!\left[537    -\sqrt{\tfrac{6}{n_{\mathrm{in}}+n_{\mathrm{out}}}},\;538    \sqrt{\tfrac{6}{n_{\mathrm{in}}+n_{\mathrm{out}}}}539  \right].540\end{equation}541\emph{He/Kaiming} initialization corrects for the fact that ReLU zeroes542half of its inputs, which halves the activation variance at each layer:543\begin{equation}\label{eq:train-he}544  \operatorname{Var}\bigl(W_{ij}\bigr) = \frac{2}{n_{\mathrm{in}}},545  \qquad546  W_{ij} \sim547  \mathcal{N}\!\Bigl(0,\; \tfrac{2}{n_{\mathrm{in}}}\Bigr),548\end{equation}549and is the default for ReLU-family networks. Biases are initialized to550zero in both schemes. Together with the schedules of551Figure~\ref{fig:train-schedules} and an adaptive optimizer such as552Adam~\eqref{eq:train-adam-moments}--\eqref{eq:train-adam-update}, these553choices form the standard modern training recipe on which the554regularization techniques of the next chapter are layered.555