spb/artificial-neural-networks-book Public
Artificial Neural Networks — Methods, Equations and Graphical Representations: a complete book, every method with rigorous equations, pseudocode and native TikZ figures.
TeX 100%
1% ============================================================================2% Artificial Neural Networks — Methods, Equations and Graphical3% Representations4% Author : Simon-Pierre Boucher — contact@spboucher.ai5% Chapter 9 : Energy-Based and Associative-Memory Networks6% (chapters/09-energy.tex)7% ============================================================================8\chapter{Energy-Based and Associative-Memory Networks}\label{chap:energy}910The architectures of the preceding chapters compute by \emph{propagating}11signals through a directed graph. This chapter studies a different12computational metaphor, imported from statistical physics: a network is13assigned a scalar \emph{energy} for every joint configuration of its units,14and computation proceeds by \emph{relaxation} — the state slides downhill15on the energy surface until it settles in a minimum. Memories are not16stored at addresses; they are sculpted into the landscape as attractors,17and retrieval is descent. The deterministic version of this idea is the18Hopfield network \cite{hopfield1982}; its stochastic generalization, the19Boltzmann machine \cite{ackley1985}, turns the energy into a probability20distribution and thereby becomes a trainable generative model. The21restricted Boltzmann machine and its stack, the deep belief network22\cite{hinton2006}, converted these physical ideas into the first practical23recipe for training deep architectures — the spark of the modern deep24learning era.2526\section{The Classical Hopfield Network}2728\begin{definition}[Hopfield network]\label{def:en-hopfield}29A Hopfield network is a fully connected recurrent network of $N$ binary30units with states $s_i \in \{-1,+1\}$, symmetric coupling weights31$w_{ij} = w_{ji}$, no self-connections ($w_{ii}=0$), and thresholds32$\theta_i$. The network state is the vector33$\vect{s} = (s_1,\dots,s_N)\transp \in \{-1,+1\}^N$.34\end{definition}3536Every configuration $\vect{s}$ is scored by the energy function37\begin{equation}38 E(\vect{s})39 \;=\;40 -\frac{1}{2}\sum_{i,j} w_{ij}\, s_i s_j \;+\; \sum_i \theta_i s_i .41 \label{eq:en-hopfield-energy}42\end{equation}43The dynamics are \emph{asynchronous}: at each step a single unit $i$ is44selected (in random or cyclic order) and set to the sign of its local45field,46\begin{equation}47 s_i \;\leftarrow\;48 \operatorname{sign}\!\Bigl(\sum_j w_{ij}\, s_j - \theta_i\Bigr).49 \label{eq:en-hopfield-update}50\end{equation}51The central fact — and the reason the network computes anything at all —52is that this update can only lower the energy.5354\begin{theorem}[Energy descent]\label{thm:en-descent}55Under the asynchronous update \eqref{eq:en-hopfield-update} with symmetric56weights and $w_{ii}=0$, the energy \eqref{eq:en-hopfield-energy} never57increases. Since $E$ is bounded below on the finite state space, the58dynamics converge in finitely many steps to a fixed point — a local59minimum of $E$.60\end{theorem}6162\begin{proof}63Suppose unit $i$ flips from $s_i$ to $s_i' = -s_i$; all other units are64frozen. Writing $h_i = \sum_j w_{ij}s_j - \theta_i$ for the local field,65the energy change collects exactly the terms involving $s_i$ (symmetry66counts each pair once, and $w_{ii}=0$ removes the diagonal):67$\Delta E = E(\vect{s}') - E(\vect{s}) = -(s_i' - s_i)\,h_i$.68The rule \eqref{eq:en-hopfield-update} flips $s_i$ only when $s_i$ and69$h_i$ disagree, i.e.\ when $s_i' = \operatorname{sign}(h_i)$, so70$(s_i' - s_i)\,h_i = 2\,|h_i| \ge 0$ and $\Delta E \le 0$. A strict flip71with $h_i \neq 0$ strictly decreases $E$; as the state space is finite,72only finitely many strict decreases can occur.73\end{proof}7475\subsection{Hebbian storage and capacity}7677To turn descent into \emph{recall}, one shapes the landscape so that the78desired memories $\{\vect{\xi}^\mu\}_{\mu=1}^{P}$,79$\vect{\xi}^\mu \in \{-1,+1\}^N$, sit at the bottoms of basins. The80Hebbian (outer-product) prescription is81\begin{equation}82 w_{ij} \;=\; \frac{1}{N}\sum_{\mu=1}^{P} \xi_i^\mu\, \xi_j^\mu ,83 \qquad w_{ii}=0,84 \label{eq:en-hebb}85\end{equation}86which makes each $\vect{\xi}^\mu$ (approximately) a fixed point of87\eqref{eq:en-hopfield-update}: units that fire together are wired88together, so a corrupted probe within a basin is pulled back to the89stored pattern. This is \emph{content-addressable} memory — the query is90a partial or noisy version of the content itself.9192\begin{remark}[Capacity and spurious states]\label{rem:en-capacity}93Statistical-mechanics analysis of \eqref{eq:en-hebb} shows that reliable94retrieval survives only up to95\begin{equation}96 P_{\max} \;\approx\; 0.138\, N97 \label{eq:en-capacity}98\end{equation}99stored patterns; for essentially error-free recall the bound tightens to100$N/(2\ln N)$. Beyond capacity, crosstalk between patterns proliferates101\emph{spurious attractors} — mixture states and inverted patterns that102were never stored — and retrieval degrades abruptly (catastrophic103interference).104\end{remark}105106\begin{algorithm}[htbp]107\caption{Hopfield network: Hebbian storage and asynchronous recall}108\label{alg:en-hopfield}109\begin{algorithmic}[1]110\Require patterns $\vect{\xi}^1,\dots,\vect{\xi}^P \in \{-1,+1\}^N$;111 probe $\vect{x} \in \{-1,+1\}^N$112\State $w_{ij} \gets \frac{1}{N}\sum_{\mu=1}^{P}\xi_i^\mu \xi_j^\mu$113 for all $i \neq j$; \quad $w_{ii} \gets 0$114 \Comment{storage \eqref{eq:en-hebb}}115\State $\vect{s} \gets \vect{x}$ \Comment{initialize at the probe}116\Repeat117 \State draw a unit index $i$ (random or cyclic sweep)118 \State $h_i \gets \sum_j w_{ij} s_j - \theta_i$119 \State $s_i \gets \operatorname{sign}(h_i)$120 \Comment{energy never increases (Theorem~\ref{thm:en-descent})}121\Until{no unit changed during a full sweep}122\State \Return $\vect{s}$ \Comment{an attractor: ideally the stored123 pattern nearest $\vect{x}$}124\end{algorithmic}125\end{algorithm}126127Figure~\ref{fig:en-hopfield} shows the two complementary views of the same128object: the fully connected graph of units, and the energy landscape over129the state space in which the stored patterns are the basins.130131\begin{figure}[htbp]132 \centering133 \begin{tikzpicture}134 % ---- left: fully connected 6-unit Hopfield graph (hexagon) ----135 \begin{scope}[local bounding box=graph]136 \foreach \i in {1,...,6}137 \node[nhidden] (s\i) at ({90-(\i-1)*60}:1.9) {$s_{\i}$};138 \foreach \i [evaluate=\i as \nexti using int(\i+1)] in {1,...,5}139 \foreach \j in {\nexti,...,6}140 \draw[black!45, semithick] (s\i) -- (s\j);141 \node[etiquette, sloped, above] at ($(s1)!0.5!(s2)$) {$w_{12}$};142 \node[etiquette, sloped, above] at ($(s2)!0.5!(s3)$) {$w_{23}$};143 \node[etiquette, sloped, below] at ($(s4)!0.5!(s5)$) {$w_{45}$};144 \end{scope}145 \node[etiquette, align=center] at ($(graph.south)+(0,-0.5)$)146 {fully connected, symmetric $w_{ij}=w_{ji}$,\\ no self-loops};147148 % ---- right: 1D energy landscape with two basins ----149 \begin{scope}[xshift=6.4cm, yshift=-2.2cm]150 \begin{axis}[151 width=7.2cm, height=5.6cm,152 axis lines=left,153 xlabel={state space (schematic)},154 ylabel={$E(\vect{s})$},155 xtick=\empty, ytick=\empty,156 domain=-2.35:2.45, samples=120,157 ymin=-1.9, ymax=1.4,158 clip=false,159 ]160 \addplot[cmem, very thick] {0.25*x^4 - x^2 + 0.15*x};161 % basin minima near x=-1.49 and x=+1.34162 \node[etiquette, align=center] at (axis cs:-1.49,-1.55)163 {stored pattern $\vect{\xi}^1$};164 \node[etiquette, align=center] at (axis cs:1.34,-1.35)165 {stored pattern $\vect{\xi}^2$};166 % descending ball167 \fill[coutput] (axis cs:-2.18,0.57) circle (2.2pt);168 \node[etiquette, anchor=west] at (axis cs:-1.98,0.85)169 {probe $\vect{x}$};170 \draw[fleche, coutput]171 (axis cs:-2.1,0.32) to[bend left=25] (axis cs:-1.62,-0.85);172 \end{axis}173 \end{scope}174 \end{tikzpicture}175 \caption{The Hopfield network \cite{hopfield1982}. Left: six binary176 units with symmetric all-to-all couplings (a few weights labeled).177 Right: the energy \eqref{eq:en-hopfield-energy} as a landscape over178 the state space; Hebbian storage \eqref{eq:en-hebb} carves one basin179 per memory, and asynchronous updates180 \eqref{eq:en-hopfield-update} roll a noisy probe downhill into the181 nearest attractor.}182 \label{fig:en-hopfield}183\end{figure}184185\section{Modern Hopfield Networks}186187The classical construction is limited by its $O(N)$ capacity188\eqref{eq:en-capacity}. Modern Hopfield networks \cite{ramsauer2021}189replace binary states by continuous queries $\vect{q} \in \R^{d}$ and the190quadratic energy by a \emph{log-sum-exp} energy over stored patterns191$\mat{X} = [\vect{x}_1,\dots,\vect{x}_P] \in \R^{d\times P}$:192\begin{equation}193 E(\vect{q})194 \;=\;195 -\frac{1}{\beta}\,196 \log \sum_{i=1}^{P} \exp\bigl(\beta\, \vect{x}_i\transp \vect{q}\bigr)197 \;+\; \frac{1}{2}\,\lVert \vect{q}\rVert^2 \;+\; \text{const},198 \label{eq:en-mhn-energy}199\end{equation}200where $\beta > 0$ is an inverse temperature. Minimizing201\eqref{eq:en-mhn-energy} by a concave--convex procedure yields the202strikingly simple fixed-point iteration203\begin{equation}204 \vect{q}^{\mathrm{new}}205 \;=\;206 \mat{X}\,\softmax\bigl(\beta\, \mat{X}\transp \vect{q}\bigr),207 \label{eq:en-mhn-update}208\end{equation}209a convex combination of the stored patterns weighted by their similarity210to the query.211212\begin{property}[Exponential capacity, one-step retrieval]213\label{prop:en-mhn}214For patterns placed on a sphere in $\R^d$, the energy215\eqref{eq:en-mhn-energy} admits on the order of $\exp(c\,d)$216well-separated attractors (for a constant $c$ depending on the required217separation), and for well-separated patterns the update218\eqref{eq:en-mhn-update} retrieves the correct pattern to exponentially219small error in a \emph{single} step \cite{ramsauer2021}.220\end{property}221222\begin{remark}[Associative memory is attention]\label{rem:en-attention}223Read \eqref{eq:en-mhn-update} with $\vect{q}$ as a query and the columns224of $\mat{X}$ as keys and values: it is exactly the scaled dot-product225attention update of the Transformer, with $\beta = 1/\sqrt{d_k}$.226Retrieval from an associative memory and an attention head are the same227computation — a result that unifies one of the oldest ideas in the field228with one of the newest.229\end{remark}230231\section{Boltzmann Machines}232233The Hopfield network is deterministic: it can only descend, so it settles234in whichever basin it starts in, and it has no principled notion of235\emph{learning} a data distribution. The Boltzmann machine236\cite{ackley1985} makes two moves. First, units become stochastic, and237the network is augmented with \emph{hidden} units $\vect{h}$ alongside238the \emph{visible} units $\vect{v}$ that carry the data; the energy239$E(\vect{v},\vect{h})$ keeps the quadratic form of240\eqref{eq:en-hopfield-energy} over the concatenated state. Second, the241energy is exponentiated into a Gibbs (Boltzmann) distribution at242temperature $T$:243\begin{equation}244 p(\vect{v},\vect{h})245 \;=\;246 \frac{e^{-E(\vect{v},\vect{h})/T}}{Z},247 \qquad248 Z \;=\; \sum_{\vect{v}',\vect{h}'} e^{-E(\vect{v}',\vect{h}')/T},249 \label{eq:en-bm-gibbs}250\end{equation}251so that low-energy configurations are exponentially more probable. The252network is now a latent-variable generative model, and maximum-likelihood253learning has a beautifully symmetric gradient: for a coupling $w_{ij}$254between any two units $s_i$ and $s_j$,255\begin{equation}256 \frac{\partial \log p(\vect{v})}{\partial w_{ij}}257 \;=\;258 \E\bigl[s_i s_j\bigr]_{\text{data}}259 \;-\;260 \E\bigl[s_i s_j\bigr]_{\text{model}},261 \label{eq:en-bm-grad}262\end{equation}263a \emph{positive phase} that measures correlations with the visibles264clamped to data, minus a \emph{negative phase} that measures the same265correlations under the model's own equilibrium distribution266\eqref{eq:en-bm-gibbs}. Learning stops when the model dreams what it267sees. The catch is the negative phase: equilibrium expectations require268Markov chains run to convergence under an intractable $Z$, which made269general Boltzmann machines impractical for two decades.270271\section{Restricted Boltzmann Machines and Contrastive Divergence}272273\begin{definition}[Restricted Boltzmann machine]\label{def:en-rbm}274An RBM is a Boltzmann machine whose connectivity graph is275\emph{bipartite}: $n_v$ visible units $\vect{v} \in \{0,1\}^{n_v}$ and276$n_h$ hidden units $\vect{h} \in \{0,1\}^{n_h}$ interact through277$\mat{W} \in \R^{n_v \times n_h}$, with biases $\vect{a}, \vect{b}$ and278\emph{no} intra-layer connections. Its energy is279\begin{equation}280 E(\vect{v},\vect{h})281 \;=\;282 -\vect{a}\transp \vect{v}283 \;-\; \vect{b}\transp \vect{h}284 \;-\; \vect{v}\transp \mat{W}\, \vect{h},285 \label{eq:en-rbm-energy}286\end{equation}287with joint distribution288$p(\vect{v},\vect{h}) = e^{-E(\vect{v},\vect{h})}/Z$ as in289\eqref{eq:en-bm-gibbs} (absorbing $T$ into the parameters).290\end{definition}291292Bipartiteness (Figure~\ref{fig:en-rbm}) is precisely what makes the model293usable: conditioned on one layer, the units of the other layer are294independent, and the conditionals factorize into sigmoids,295\begin{align}296 p(h_j = 1 \mid \vect{v})297 &\;=\;298 \sigma\Bigl(b_j + \sum_i v_i\, w_{ij}\Bigr),299 \label{eq:en-rbm-condh}\\300 p(v_i = 1 \mid \vect{h})301 &\;=\;302 \sigma\Bigl(a_i + \sum_j w_{ij}\, h_j\Bigr),303 \label{eq:en-rbm-condv}304\end{align}305with $\sigma(x) = 1/(1+e^{-x})$. An entire layer can therefore be sampled306in one parallel \emph{block Gibbs} step. Marginalizing the hiddens also307gives a closed-form \emph{free energy} for a visible vector,308\begin{equation}309 F(\vect{v})310 \;=\;311 -\vect{a}\transp\vect{v}312 \;-\;313 \sum_{j=1}^{n_h}314 \log\Bigl(1 + e^{\,b_j + \vect{v}\transp \mat{W}_{:j}}\Bigr),315 \qquad316 p(\vect{v}) = \frac{e^{-F(\vect{v})}}{Z},317 \label{eq:en-rbm-free}318\end{equation}319useful for monitoring training. The exact likelihood gradient specializes320\eqref{eq:en-bm-grad} to321\begin{equation}322 \frac{\partial \log p(\vect{v})}{\partial w_{ij}}323 \;=\;324 \E\bigl[v_i h_j\bigr]_{\text{data}}325 \;-\;326 \E\bigl[v_i h_j\bigr]_{\text{model}} .327 \label{eq:en-rbm-grad}328\end{equation}329330\begin{figure}[htbp]331 \centering332 \begin{tikzpicture}333 \foreach \j in {1,...,3}334 \node[nhidden] (h\j) at (1.1+\j*1.7, 2.1) {$h_{\j}$};335 \foreach \i in {1,...,4}336 \node[ninput] (v\i) at (0.25+\i*1.7, 0) {$v_{\i}$};337 \foreach \i in {1,...,4}338 \foreach \j in {1,...,3}339 \draw[black!45, semithick] (v\i) -- (h\j);340 \node[etiquette, left=3mm of h1] {hidden layer $\vect{h}$};341 \node[etiquette, left=3mm of v1] {visible layer $\vect{v}$};342 \node[etiquette, right=2mm of h3, yshift=-9mm] {$\mat{W}$};343 \node[etiquette, align=center] at (4.2,-0.95)344 {bipartite: no intra-layer connections};345 \end{tikzpicture}346 \caption{The restricted Boltzmann machine347 (Definition~\ref{def:en-rbm}). Visible and hidden units interact348 through $\mat{W}$ only across layers; the missing intra-layer edges349 are what make the conditionals350 \eqref{eq:en-rbm-condh}--\eqref{eq:en-rbm-condv} factorize and block351 Gibbs sampling efficient.}352 \label{fig:en-rbm}353\end{figure}354355\subsection{Contrastive divergence}356357The negative phase of \eqref{eq:en-rbm-grad} still asks for equilibrium358samples. Contrastive divergence \cite{hinton2002} replaces equilibrium359with \emph{proximity}: run the block Gibbs chain360\begin{equation}361 \vect{v}^{(0)} \xrightarrow{\;p(\vect{h}\mid\vect{v})\;} \vect{h}^{(0)}362 \xrightarrow{\;p(\vect{v}\mid\vect{h})\;} \vect{v}^{(1)}363 \xrightarrow{\;\cdots\;} \vect{v}^{(k)}364 \xrightarrow{\;p(\vect{h}\mid\vect{v})\;} \vect{h}^{(k)}365 \label{eq:en-gibbs-chain}366\end{equation}367for only $k$ steps (typically $k=1$), \emph{started at the data}368$\vect{v}^{(0)}$, and contrast the correlations at both ends:369\begin{equation}370 \Delta w_{ij}371 \;\propto\;372 \bigl\langle v_i h_j \bigr\rangle_{0}373 \;-\;374 \bigl\langle v_i h_j \bigr\rangle_{k}375 \qquad \text{(CD-}k\text{)}.376 \label{eq:en-cdk}377\end{equation}378The estimator is biased — the chain has not mixed — but the bias is small379near the data manifold and the signal is strong enough to train excellent380features; persistent CD further improves the negative samples by never381restarting the chain. Algorithm~\ref{alg:en-cdk} is the chapter's382estimation centerpiece, and Figure~\ref{fig:en-cdk} traces the chain383\eqref{eq:en-gibbs-chain}.384385\begin{algorithm}[htbp]386\caption{Contrastive divergence CD-$k$ for one mini-batch (RBM)}387\label{alg:en-cdk}388\begin{algorithmic}[1]389\Require mini-batch $\{\vect{v}_1,\dots,\vect{v}_m\}$, parameters390 $(\mat{W},\vect{a},\vect{b})$, learning rate $\eta$,391 Gibbs steps $k$392\For{$n = 1,\dots,m$}393 \State $\vect{v}^{(0)} \gets \vect{v}_n$394 \State $\hat{\vect{h}}^{(0)} \gets395 \sigma\bigl(\vect{b} + \mat{W}\transp \vect{v}^{(0)}\bigr)$396 \Comment{positive phase, \eqref{eq:en-rbm-condh}}397 \For{$t = 0,\dots,k-1$}398 \State sample $\vect{h}^{(t)} \sim399 p(\vect{h}\mid \vect{v}^{(t)})$400 \Comment{block Gibbs step, \eqref{eq:en-rbm-condh}}401 \State sample $\vect{v}^{(t+1)} \sim402 p(\vect{v}\mid \vect{h}^{(t)})$403 \Comment{block Gibbs step, \eqref{eq:en-rbm-condv}}404 \EndFor405 \State $\hat{\vect{h}}^{(k)} \gets406 \sigma\bigl(\vect{b} + \mat{W}\transp \vect{v}^{(k)}\bigr)$407 \State $\Delta\mat{W}_n \gets408 \vect{v}^{(0)}\hat{\vect{h}}^{(0)\top}409 - \vect{v}^{(k)}\hat{\vect{h}}^{(k)\top}$410 \Comment{contrast \eqref{eq:en-cdk}}411 \State $\Delta\vect{a}_n \gets \vect{v}^{(0)} - \vect{v}^{(k)}$;412 \quad413 $\Delta\vect{b}_n \gets414 \hat{\vect{h}}^{(0)} - \hat{\vect{h}}^{(k)}$415\EndFor416\State $\mat{W} \gets \mat{W} + \frac{\eta}{m}\sum_n \Delta\mat{W}_n$;417 \quad418 $\vect{a} \gets \vect{a} + \frac{\eta}{m}\sum_n \Delta\vect{a}_n$;419 \quad420 $\vect{b} \gets \vect{b} + \frac{\eta}{m}\sum_n \Delta\vect{b}_n$421\end{algorithmic}422\end{algorithm}423424\begin{figure}[htbp]425 \centering426 \begin{tikzpicture}[node distance=6mm and 13mm]427 \node[blocinput, minimum width=1.35cm] (v0) {$\vect{v}^{(0)}$};428 \node[blochidden, minimum width=1.35cm, right=of v0] (h0)429 {$\vect{h}^{(0)}$};430 \node[blocinput, minimum width=1.35cm, right=of h0] (v1)431 {$\vect{v}^{(1)}$};432 \node[blochidden, minimum width=1.35cm, right=9mm of v1] (h1)433 {$\vect{h}^{(1)}$};434 \node[right=5mm of h1] (dots) {$\cdots$};435 \node[blocinput, minimum width=1.35cm, right=5mm of dots] (vk)436 {$\vect{v}^{(k)}$};437 \node[blochidden, minimum width=1.35cm, right=of vk] (hk)438 {$\vect{h}^{(k)}$};439 \draw[fleche] (v0) -- node[etiquette, above]440 {$p(\vect{h}\!\mid\!\vect{v})$} (h0);441 \draw[fleche] (h0) -- node[etiquette, above]442 {$p(\vect{v}\!\mid\!\vect{h})$} (v1);443 \draw[fleche] (v1) -- (h1);444 \draw[fleche] (h1) -- (dots);445 \draw[fleche] (dots) -- (vk);446 \draw[fleche] (vk) -- node[etiquette, above]447 {$p(\vect{h}\!\mid\!\vect{v})$} (hk);448 \node[etiquette, below=2.5mm of v0] {data (clamped)};449 \node[etiquette, below=2.5mm of hk] {negative phase};450 \draw[flechep, coutput]451 ($(v0.south)+(0,-9mm)$) -- node[etiquette, below]452 {contrast453 $\langle v_i h_j\rangle_0 - \langle v_i h_j\rangle_k$454 \ \eqref{eq:en-cdk}}455 ($(hk.south)+(0,-9mm)$);456 \end{tikzpicture}457 \caption{The CD-$k$ Gibbs chain \eqref{eq:en-gibbs-chain}458 \cite{hinton2002}. The chain starts at the data, alternates the459 factorized conditionals460 \eqref{eq:en-rbm-condh}--\eqref{eq:en-rbm-condv} for $k$ block461 steps, and the weight update contrasts correlations at the two ends462 of the chain.}463 \label{fig:en-cdk}464\end{figure}465466\section{Deep Belief Networks}467468A single RBM learns one layer of features. Deep belief networks469\cite{hinton2006} stack RBMs: train an RBM on the data, freeze it, feed470its hidden activations as ``data'' to a second RBM, and repeat. The471resulting generative model is a hybrid — the top two layers keep an472undirected RBM joint, while the lower layers become a directed belief473network:474\begin{equation}475 p\bigl(\vect{v}, \vect{h}^{(1)}, \dots, \vect{h}^{(L)}\bigr)476 \;=\;477 p\bigl(\vect{h}^{(L-1)}, \vect{h}^{(L)}\bigr)478 \prod_{\ell=L-2}^{0}479 p\bigl(\vect{h}^{(\ell)} \mid \vect{h}^{(\ell+1)}\bigr),480 \label{eq:en-dbn-joint}481\end{equation}482with $\vect{h}^{(0)} \equiv \vect{v}$. Greediness is not a mere483heuristic: each newly stacked RBM, initialized as the transpose of the484one below, starts from an equivalent model and can only improve a485variational lower bound on the data log-likelihood,486\begin{equation}487 \log p(\vect{v})488 \;\ge\;489 \E_{q(\vect{h}^{(1)}\mid\vect{v})}490 \Bigl[\log p\bigl(\vect{v}\mid\vect{h}^{(1)}\bigr)491 + \log p\bigl(\vect{h}^{(1)}\bigr)\Bigr]492 \;+\;493 \mathcal{H}\bigl[q(\vect{h}^{(1)}\mid\vect{v})\bigr],494 \label{eq:en-dbn-bound}495\end{equation}496where $q$ is the first RBM's posterior, $\mathcal{H}[\cdot]$ denotes the497entropy, and training the upper stack improves the prior term498$\log p(\vect{h}^{(1)})$. After pretraining, the stack is optionally499\emph{fine-tuned} — generatively by the wake--sleep procedure, or500discriminatively by appending a classifier head and running501backpropagation through the unrolled weights502(Algorithm~\ref{alg:en-dbn}, Figure~\ref{fig:en-dbn}).503504\begin{algorithm}[htbp]505\caption{Greedy layer-wise pretraining of a deep belief network}506\label{alg:en-dbn}507\begin{algorithmic}[1]508\Require dataset $\mathcal{D} = \{\vect{v}_n\}$, layer sizes509 $n_1,\dots,n_L$, Gibbs steps $k$510\State $\mathcal{D}^{(0)} \gets \mathcal{D}$511\For{$\ell = 1,\dots,L$}512 \State train RBM$_\ell$ with parameters $(\mat{W}^{(\ell)},513 \vect{a}^{(\ell)}, \vect{b}^{(\ell)})$ on514 $\mathcal{D}^{(\ell-1)}$ by CD-$k$515 (Algorithm~\ref{alg:en-cdk})516 \State $\mathcal{D}^{(\ell)} \gets517 \bigl\{\, \sigma\bigl(\vect{b}^{(\ell)} +518 \mat{W}^{(\ell)\top}\vect{x}\bigr)519 \;:\; \vect{x} \in \mathcal{D}^{(\ell-1)} \bigr\}$520 \Comment{propagate features upward}521\EndFor522\State \Return stack $\{\mat{W}^{(\ell)}\}_{\ell=1}^{L}$; optionally523 fine-tune (wake--sleep, or backpropagation with a supervised524 head)525\end{algorithmic}526\end{algorithm}527528\begin{figure}[htbp]529 \centering530 \begin{tikzpicture}[node distance=9mm]531 \node[blocinput, minimum width=3.4cm] (v) {$\vect{v}$};532 \node[blochidden, minimum width=3.4cm, above=of v] (h1)533 {$\vect{h}^{(1)}$};534 \node[blochidden, minimum width=3.4cm, above=of h1] (h2)535 {$\vect{h}^{(2)}$};536 \node[mem, minimum width=3.4cm, above=of h2] (h3)537 {$\vect{h}^{(3)}$};538 % upward recognition / pretraining arrows539 \draw[fleche, cgate!70!black]540 ([xshift=-9mm]v.north) -- ([xshift=-9mm]h1.south)541 node[etiquette, midway, left=1.5mm] {$\mat{W}^{(1)\top}$};542 \draw[fleche, cgate!70!black]543 ([xshift=-9mm]h1.north) -- ([xshift=-9mm]h2.south)544 node[etiquette, midway, left=1.5mm] {$\mat{W}^{(2)\top}$};545 \draw[fleche, cgate!70!black]546 ([xshift=-9mm]h2.north) -- ([xshift=-9mm]h3.south)547 node[etiquette, midway, left=1.5mm] {$\mat{W}^{(3)\top}$};548 % downward generative arrows549 \draw[flechep, cmem!80!black]550 ([xshift=9mm]h3.south) -- ([xshift=9mm]h2.north)551 node[etiquette, midway, right=1.5mm] {$\mat{W}^{(3)}$};552 \draw[flechep, cmem!80!black]553 ([xshift=9mm]h2.south) -- ([xshift=9mm]h1.north)554 node[etiquette, midway, right=1.5mm] {$\mat{W}^{(2)}$};555 \draw[flechep, cmem!80!black]556 ([xshift=9mm]h1.south) -- ([xshift=9mm]v.north)557 node[etiquette, midway, right=1.5mm] {$\mat{W}^{(1)}$};558 % RBM braces on the left559 \draw[decorate, decoration={brace, amplitude=5pt}, black!60]560 ($(h1.west)+(-1.65,0.12)$) -- ($(v.west)+(-1.65,-0.12)$)561 node[etiquette, midway, left=7pt] {RBM 1};562 \draw[decorate, decoration={brace, amplitude=5pt}, black!60]563 ($(h2.west)+(-2.5,0.12)$) -- ($(h1.west)+(-2.5,-0.12)$)564 node[etiquette, midway, left=7pt] {RBM 2};565 \draw[decorate, decoration={brace, amplitude=5pt}, black!60]566 ($(h3.west)+(-1.65,0.12)$) -- ($(h2.west)+(-1.65,-0.12)$)567 node[etiquette, midway, left=7pt] {RBM 3};568 % legend569 \node[etiquette, align=left, anchor=west]570 at ($(v.east)+(0.75,0.7)$)571 {\textcolor{cgate!70!black}{$\longrightarrow$}\ pretraining572 (recognition)\\[1pt]573 \textcolor{cmem!80!black}{$\dashrightarrow$}\ generation};574 % top RBM annotation575 \node[etiquette, above=2.5mm of h3]576 {top pair: undirected joint577 $p(\vect{h}^{(2)},\vect{h}^{(3)})$};578 \end{tikzpicture}579 \caption{A three-layer deep belief network \cite{hinton2006}. Each580 RBM (braces) is trained greedily on the features of the layer below581 (Algorithm~\ref{alg:en-dbn}); the trained stack generates by582 sampling the top RBM and propagating down the dashed directed583 connections, as in \eqref{eq:en-dbn-joint}.}584 \label{fig:en-dbn}585\end{figure}586587\begin{remark}[Historical impact]\label{rem:en-history}588Greedy DBN pretraining was the first broadly reliable method for589initializing deep networks, at a time when random initialization plus590backpropagation stalled. Modern practice — rectified activations,591normalization, residual connections and large datasets — later made592unsupervised pretraining unnecessary for most supervised tasks, but the593energy-based view survives: contrastive objectives, score matching, and594the attention--Hopfield correspondence of595Remark~\ref{rem:en-attention} are all its descendants.596\end{remark}597