spb/artificial-neural-networks-book Public
Artificial Neural Networks — Methods, Equations and Graphical Representations: a complete book, every method with rigorous equations, pseudocode and native TikZ figures.
TeX 100%
1% ============================================================================2% Artificial Neural Networks — Methods, Equations and Graphical3% Representations4% Author : Simon-Pierre Boucher — contact@spboucher.ai5% Chapter 7 : Generative Models (chapters/07-generative.tex)6% ============================================================================7\chapter{Generative Models}\label{chap:generative}89The architectures of the preceding chapters were, for the most part,10\emph{discriminative}: they learn a mapping from an input $\vect{x}$ to a11target $\vect{y}$. Generative models pursue a more ambitious goal — to learn12a representation of the data distribution $p_{\mathrm{data}}(\vect{x})$13itself, so that new samples can be drawn from it. The major families differ14in \emph{how} they represent this density: approximately, through a15variational lower bound (variational autoencoders); implicitly, through a16sampling procedure trained adversarially (generative adversarial networks);17or exactly, through the chain-rule factorization (autoregressive models).18This chapter presents each family with its objective function, its19gradients, and its computational structure, and closes with the iterative20denoising paradigm — diffusion models — that dominates image synthesis21today.2223\begin{definition}[Generative model]\label{def:gen-model}24A generative model is a parametric family $p_\theta(\vect{x})$ together25with a sampling procedure, trained so that $p_\theta \approx26p_{\mathrm{data}}$, where $p_{\mathrm{data}}$ is the (unknown) distribution27from which the training set $\{\vect{x}_1,\dots,\vect{x}_N\}$ was drawn.28\end{definition}2930% ============================================================================31\section{Autoencoders}\label{sec:gen-ae}3233The autoencoder is the conceptual ancestor of the modern deep generative34model. It consists of an \emph{encoder} $f_{\mathrm{enc}} \colon \R^{d}35\to \R^{m}$, which maps an input to a low-dimensional \emph{latent code},36and a \emph{decoder} $f_{\mathrm{dec}} \colon \R^{m} \to \R^{d}$, which37attempts to reconstruct the input from that code:38\begin{equation}39 \vect{z} = f_{\mathrm{enc}}(\vect{x}),40 \qquad41 \hat{\vect{x}} = f_{\mathrm{dec}}(\vect{z}),42 \qquad m \ll d .43 \label{eq:gen-ae}44\end{equation}45Both maps are neural networks — typically mirror-image MLPs or46convolutional networks — trained jointly to minimize the47\emph{reconstruction loss}48\begin{equation}49 \Loss_{\mathrm{AE}}50 = \frac{1}{N}\sum_{i=1}^{N}51 \bigl\lVert \vect{x}_i -52 f_{\mathrm{dec}}\bigl(f_{\mathrm{enc}}(\vect{x}_i)\bigr)53 \bigr\rVert_2^2 ,54 \label{eq:gen-ae-loss}55\end{equation}56or a cross-entropy loss when the inputs are modeled as Bernoulli variables57(e.g.\ binarized pixels). Because the bottleneck dimension $m$ in58\eqref{eq:gen-ae} is much smaller than the input dimension $d$, the network59cannot learn the identity map; it is forced to discover a compressed60representation that preserves exactly the information needed to reconstruct61the data. Figure~\ref{fig:gen-ae} shows the characteristic hourglass shape.6263\begin{figure}[ht]64 \centering65 \begin{tikzpicture}[scale=0.92, transform shape]66 % input layer: 6 neurons67 \foreach \i in {1,...,6}68 \node[ninput] (x\i) at (0, 3.0-\i) {};69 % encoder hidden: 4 neurons70 \foreach \i in {1,...,4}71 \node[nhidden] (e\i) at (2.0, 2.0-\i) {};72 % latent: 2 neurons (memory color)73 \foreach \i in {1,2}74 \node[neuron, fill=cmem!25, draw=cmem!70!black]75 (z\i) at (4.0, 1.0-\i) {};76 % decoder hidden: 4 neurons77 \foreach \i in {1,...,4}78 \node[nhidden] (d\i) at (6.0, 2.0-\i) {};79 % output layer: 6 neurons80 \foreach \i in {1,...,6}81 \node[noutput] (y\i) at (8.0, 3.0-\i) {};82 % connections83 \foreach \i in {1,...,6} \foreach \j in {1,...,4}84 \draw[black!30, semithick] (x\i) -- (e\j);85 \foreach \i in {1,...,4} \foreach \j in {1,2}86 \draw[black!30, semithick] (e\i) -- (z\j);87 \foreach \i in {1,2} \foreach \j in {1,...,4}88 \draw[black!30, semithick] (z\i) -- (d\j);89 \foreach \i in {1,...,4} \foreach \j in {1,...,6}90 \draw[black!30, semithick] (d\i) -- (y\j);91 % column labels92 \node[etiquette] at (0, 2.6) {$\vect{x}$};93 \node[etiquette] at (4.0, 0.6) {$\vect{z}$};94 \node[etiquette] at (8.0, 2.6) {$\hat{\vect{x}}$};95 % braces96 \draw[decorate, decoration={brace, mirror, amplitude=5pt}, thick]97 (-0.4,-3.6) -- (3.4,-3.6)98 node[midway, below=7pt, font=\small]99 {encoder $f_{\mathrm{enc}}$};100 \draw[decorate, decoration={brace, mirror, amplitude=5pt}, thick]101 (4.6,-3.6) -- (8.4,-3.6)102 node[midway, below=7pt, font=\small]103 {decoder $f_{\mathrm{dec}}$};104 \node[font=\small] at (4.0, -2.1) {latent code};105 \end{tikzpicture}106 \caption{The autoencoder: an hourglass architecture. The encoder107 compresses the input $\vect{x}$ into a latent code $\vect{z}$ (violet)108 of much smaller dimension; the decoder reconstructs109 $\hat{\vect{x}}$ from $\vect{z}$. Training minimizes the reconstruction110 loss~\eqref{eq:gen-ae-loss}.}111 \label{fig:gen-ae}112\end{figure}113114\begin{remark}\label{rem:gen-ae-not-generative}115A plain autoencoder is \emph{not} yet a generative model in the sense of116Definition~\ref{def:gen-model}: nothing constrains the geometry of the117latent space, so decoding an arbitrary $\vect{z}$ generally does not118produce a realistic sample. Imposing a probabilistic structure on the119latent space is precisely the contribution of the variational autoencoder.120\end{remark}121122% ============================================================================123\section{Variational Autoencoders}\label{sec:gen-vae}124125The variational autoencoder (VAE) of Kingma and126Welling~\cite{kingma2014vae} turns the autoencoder into a proper127latent-variable model. It posits a prior $p(\vect{z}) =128\mathcal{N}(\vect{0}, \mat{I})$ and a decoder that defines a conditional129likelihood $p_\theta(\vect{x} \mid \vect{z})$, so that the model density is130\begin{equation}131 p_\theta(\vect{x})132 = \int p_\theta(\vect{x} \mid \vect{z})\, p(\vect{z})\, d\vect{z} .133 \label{eq:gen-lvm}134\end{equation}135The integral in \eqref{eq:gen-lvm} is intractable, so the VAE introduces an136\emph{approximate posterior} $q_\phi(\vect{z} \mid \vect{x}) =137\mathcal{N}\bigl(\vect{\mu}_\phi(\vect{x}),138\operatorname{diag}(\vect{\sigma}_\phi^2(\vect{x}))\bigr)$ — the139probabilistic encoder — and maximizes a lower bound on the log-likelihood.140141\subsection{The evidence lower bound}142143For any $q_\phi$, the log-likelihood decomposes exactly as144\begin{equation}145 \log p_\theta(\vect{x})146 = \Loss_{\mathrm{ELBO}}(\theta, \phi; \vect{x})147 + \KL\bigl(q_\phi(\vect{z} \mid \vect{x}) \,\big\Vert\,148 p_\theta(\vect{z} \mid \vect{x})\bigr) ,149 \label{eq:gen-elbo-decomp}150\end{equation}151and since the Kullback--Leibler divergence is non-negative, the first term152— the \emph{evidence lower bound} (ELBO) — bounds $\log p_\theta(\vect{x})$153from below:154\begin{equation}155 \Loss_{\mathrm{ELBO}}(\theta, \phi; \vect{x})156 = \underbrace{\E_{q_\phi(\vect{z} \mid \vect{x})}157 \bigl[\log p_\theta(\vect{x} \mid \vect{z})\bigr]}_{\text{reconstruction}}158 \;-\;159 \underbrace{\KL\bigl(q_\phi(\vect{z} \mid \vect{x})160 \,\big\Vert\, p(\vect{z})\bigr)}_{\text{regularization}} .161 \label{eq:gen-elbo}162\end{equation}163The two terms of \eqref{eq:gen-elbo} recover, and generalize, the two164ingredients of the plain autoencoder: the first is a (probabilistic)165reconstruction objective, while the second pulls the encoder's output166distribution toward the prior, organizing the latent space so that samples167$\vect{z} \sim \mathcal{N}(\vect{0},\mat{I})$ decode to realistic data.168By \eqref{eq:gen-elbo-decomp}, maximizing the ELBO simultaneously raises169the likelihood and tightens the posterior approximation.170171For diagonal Gaussians the regularization term of \eqref{eq:gen-elbo} has172a closed form. With $q_\phi = \mathcal{N}(\vect{\mu},173\operatorname{diag}(\vect{\sigma}^2))$ and $p = \mathcal{N}(\vect{0},174\mat{I})$ in $m$ dimensions,175\begin{equation}176 \KL\bigl(q_\phi(\vect{z} \mid \vect{x}) \,\big\Vert\,177 \mathcal{N}(\vect{0},\mat{I})\bigr)178 = -\frac{1}{2}\sum_{j=1}^{m}179 \bigl(1 + \log \sigma_j^2 - \mu_j^2 - \sigma_j^2\bigr) ,180 \label{eq:gen-kl}181\end{equation}182so this term costs nothing to estimate: it is computed analytically from183the encoder outputs.184185\subsection{The reparameterization trick}186187One obstacle remains: the reconstruction term of \eqref{eq:gen-elbo} is an188expectation over $\vect{z} \sim q_\phi(\vect{z} \mid \vect{x})$, and the189sampling operation blocks the gradient with respect to $\phi$. The190\emph{reparameterization trick}~\cite{kingma2014vae} rewrites the sample as191a deterministic, differentiable function of the parameters plus exogenous192noise:193\begin{equation}194 \vect{z}195 = \vect{\mu}_\phi(\vect{x})196 + \vect{\sigma}_\phi(\vect{x}) \odot \vect{\varepsilon},197 \qquad198 \vect{\varepsilon} \sim \mathcal{N}(\vect{0}, \mat{I}) .199 \label{eq:gen-reparam}200\end{equation}201The randomness now enters only through $\vect{\varepsilon}$, which does not202depend on $\phi$; gradients flow through $\vect{\mu}_\phi$ and203$\vect{\sigma}_\phi$ by the ordinary chain rule, yielding an unbiased,204low-variance pathwise estimator of $\nabla_\phi \Loss_{\mathrm{ELBO}}$.205Figure~\ref{fig:gen-vae} traces the full computation.206207\begin{figure}[ht]208 \centering209 \begin{tikzpicture}[scale=0.88, transform shape]210 \node[blocinput, minimum width=1cm] (x) at (0,0) {$\vect{x}$};211 \node[blochidden, minimum width=1.9cm, minimum height=1.9cm]212 (enc) at (2.3,0)213 {Encoder\\ $q_\phi(\vect{z}\mid\vect{x})$};214 \node[bloc, minimum width=1.15cm] (mu) at (4.9, 1.0) {$\vect{\mu}$};215 \node[bloc, minimum width=1.15cm] (sg) at (4.9,-1.0) {$\log\vect{\sigma}^2$};216 \node[bloc, minimum width=1.6cm, densely dashed]217 (eps) at (4.9,-2.7)218 {$\vect{\varepsilon}\sim\mathcal{N}(\vect{0},\mat{I})$};219 \node[op] (odot) at (7.0,-1.0) {$\odot$};220 \node[op] (plus) at (8.1, 0) {$+$};221 \node[mem, minimum width=1cm] (z) at (9.4,0) {$\vect{z}$};222 \node[blochidden, minimum width=1.9cm, minimum height=1.9cm]223 (dec) at (11.7,0)224 {Decoder\\ $p_\theta(\vect{x}\mid\vect{z})$};225 \node[blocoutput, minimum width=1cm] (xh) at (14.0,0) {$\hat{\vect{x}}$};226 % arrows227 \draw[fleche] (x) -- (enc);228 \draw[fleche] ([yshift=5.5mm]enc.east) -- ++(0.35,0) |- (mu.west);229 \draw[fleche] ([yshift=-5.5mm]enc.east) -- ++(0.35,0) |- (sg.west);230 \draw[fleche] (sg) -- node[etiquette, above]{$\exp(\cdot/2)$} (odot);231 \draw[fleche] (eps.east) -| (odot);232 \draw[fleche] (odot) -- (plus);233 \draw[fleche] (mu.east) -| (plus);234 \draw[fleche] (plus) -- (z);235 \draw[fleche] (z) -- (dec);236 \draw[fleche] (dec) -- (xh);237 \end{tikzpicture}238 \caption{The variational autoencoder with the reparameterization239 trick~\eqref{eq:gen-reparam}. The encoder outputs the parameters240 $(\vect{\mu}, \log\vect{\sigma}^2)$ of the approximate posterior; the241 latent sample is assembled as $\vect{z} = \vect{\mu} + \vect{\sigma}242 \odot \vect{\varepsilon}$, so that gradients flow through243 $\vect{\mu}$ and $\vect{\sigma}$ while the stochasticity is confined244 to the exogenous noise $\vect{\varepsilon}$ (dashed).}245 \label{fig:gen-vae}246\end{figure}247248With the reparameterization in place, a VAE training step is ordinary249stochastic gradient ascent on the ELBO, summarized in250Algorithm~\ref{alg:gen-vae}: encode, sample through251\eqref{eq:gen-reparam}, decode, and differentiate the two terms of252\eqref{eq:gen-elbo} jointly with respect to both parameter sets.253254\begin{algorithm}[htbp]255 \caption{VAE training step (stochastic gradient ascent on the ELBO)}256 \label{alg:gen-vae}257 \begin{algorithmic}[1]258 \Require minibatch $\{\vect{x}^{(i)}\}_{i=1}^{m}$, encoder parameters259 $\phi$, decoder parameters $\theta$, learning rate $\eta$260 \For{$i = 1, \dots, m$}261 \State $\bigl(\vect{\mu}^{(i)}, \log\vect{\sigma}^{2(i)}\bigr)262 \gets \mathrm{Encoder}_\phi\!\left(\vect{x}^{(i)}\right)$263 \State sample $\vect{\varepsilon}^{(i)} \sim264 \mathcal{N}(\vect{0}, \mat{I})$;\quad265 $\vect{z}^{(i)} \gets \vect{\mu}^{(i)}266 + \vect{\sigma}^{(i)} \odot \vect{\varepsilon}^{(i)}$267 \Comment{reparameterization \eqref{eq:gen-reparam}}268 \State $\hat{\vect{x}}^{(i)} \gets269 \mathrm{Decoder}_\theta\!\left(\vect{z}^{(i)}\right)$270 \State $\Loss^{(i)} \gets271 \log p_\theta\!\left(\vect{x}^{(i)} \mid \vect{z}^{(i)}\right)272 - \KL\!\left( q_\phi(\vect{z} \mid \vect{x}^{(i)})273 \,\Vert\, p(\vect{z}) \right)$274 \Comment{ELBO \eqref{eq:gen-elbo}, KL \eqref{eq:gen-kl}}275 \EndFor276 \State $\Loss \gets \frac{1}{m}\sum_{i=1}^{m} \Loss^{(i)}$277 \State $\theta \gets \theta + \eta\, \nabla_\theta \Loss$;\quad278 $\phi \gets \phi + \eta\, \nabla_\phi \Loss$279 \Comment{ascent; gradients by backpropagation}280 \end{algorithmic}281\end{algorithm}282283\begin{remark}\label{rem:gen-vae-blur}284Because the reconstruction term of \eqref{eq:gen-elbo} is typically a285Gaussian (squared-error) likelihood averaged over the posterior, VAE286samples tend to be slightly blurry: the decoder learns to output the287conditional \emph{mean} of all plausible reconstructions. This is the288characteristic failure mode recorded in289Table~\ref{tab:gen-comparison}.290\end{remark}291292% ============================================================================293\section{Generative Adversarial Networks}\label{sec:gen-gan}294295Generative adversarial networks (GANs), introduced by Goodfellow et296al.~\cite{goodfellow2014gan}, abandon explicit densities altogether. A297\emph{generator} $G$ maps noise $\vect{z} \sim p_{\vect{z}} =298\mathcal{N}(\vect{0},\mat{I})$ to a sample $G(\vect{z})$, and a299\emph{discriminator} $D(\vect{x}) \in [0,1]$ estimates the probability300that its input came from the data rather than from $G$. The two networks301play a two-player \emph{minimax game}:302\begin{equation}303 \min_G \max_D \; V(D, G)304 = \E_{\vect{x} \sim p_{\mathrm{data}}}\bigl[\log D(\vect{x})\bigr]305 + \E_{\vect{z} \sim p_{\vect{z}}}306 \bigl[\log\bigl(1 - D(G(\vect{z}))\bigr)\bigr] .307 \label{eq:gen-minimax}308\end{equation}309The discriminator is trained to tell real from fake; the generator is310trained to fool it. Figure~\ref{fig:gen-gan} shows the adversarial311arrangement: crucially, the generator never sees the data directly — its312only training signal is the gradient that flows \emph{through} the313discriminator.314315\begin{figure}[ht]316 \centering317 \begin{tikzpicture}[scale=0.92, transform shape]318 \node[bloc, minimum width=1.9cm] (z) at (0,1.0)319 {$\vect{z}\sim\mathcal{N}(\vect{0},\mat{I})$};320 \node[blochidden, minimum width=2.2cm, minimum height=1.1cm]321 (G) at (3.2,1.0) {Generator\\ $G$};322 \node[bloc, minimum width=2.3cm] (xf) at (6.5,1.0)323 {fake $\tilde{\vect{x}} = G(\vect{z})$};324 \node[blocinput, minimum width=2.3cm] (xr) at (6.5,-1.0)325 {real $\vect{x}\sim p_{\mathrm{data}}$};326 \node[blochidden, minimum width=2.4cm, minimum height=1.1cm]327 (D) at (10.0,0) {Discriminator\\ $D$};328 \node[blocoutput, minimum width=1.7cm] (out) at (13.2,0)329 {real / fake};330 \draw[fleche] (z) -- (G);331 \draw[fleche] (G) -- (xf);332 \draw[fleche] (xf.east) -- ++(0.55,0) |- ([yshift=3mm]D.west);333 \draw[fleche] (xr.east) -- ++(0.55,0) |- ([yshift=-3mm]D.west);334 \draw[fleche] (D) -- (out);335 % adversarial gradient (dashed), routed above the fake-sample block336 \draw[flechep, draw=coutput!80!black]337 (D.north) -- ++(0,1.45) -| (G.north)338 node[pos=0.25, above, etiquette, text=coutput!80!black]339 {adversarial gradient $\nabla_{\theta_G} \Loss$};340 \end{tikzpicture}341 \caption{The generative adversarial network. The generator maps noise342 $\vect{z}$ to a fake sample $\tilde{\vect{x}}$; the discriminator343 receives both real and fake samples and outputs the probability that344 its input is real. The generator's only learning signal is the345 adversarial gradient (dashed, red) backpropagated through the346 discriminator, per the minimax objective~\eqref{eq:gen-minimax}.}347 \label{fig:gen-gan}348\end{figure}349350\subsection{The optimal discriminator and the Jensen--Shannon divergence}351352The minimax game \eqref{eq:gen-minimax} has a precise353distribution-matching interpretation.354355\begin{theorem}[Optimal discriminator]\label{thm:gen-dstar}356Let $p_g$ denote the distribution of $G(\vect{z})$ for fixed $G$. The357discriminator maximizing $V(D,G)$ in \eqref{eq:gen-minimax} is358\begin{equation}359 D^{*}(\vect{x})360 = \frac{p_{\mathrm{data}}(\vect{x})}361 {p_{\mathrm{data}}(\vect{x}) + p_g(\vect{x})} ,362 \label{eq:gen-dstar}363\end{equation}364and substituting $D^{*}$ into $V$ gives the generator's effective365objective366\begin{equation}367 C(G) = \max_D V(D,G)368 = -\log 4369 + 2\,\mathrm{JSD}\bigl(p_{\mathrm{data}} \,\big\Vert\, p_g\bigr) ,370 \label{eq:gen-jsd}371\end{equation}372where $\mathrm{JSD}$ is the Jensen--Shannon divergence. Hence $C(G)$ is373minimized if and only if $p_g = p_{\mathrm{data}}$, where $D^{*} \equiv374\tfrac{1}{2}$ and $C(G) = -\log 4$.375\end{theorem}376377\begin{proof}[Proof sketch]378For fixed $G$, $V(D,G) = \int \bigl[p_{\mathrm{data}}(\vect{x}) \log379D(\vect{x}) + p_g(\vect{x}) \log(1 - D(\vect{x}))\bigr] d\vect{x}$;380pointwise maximization of $a \log t + b \log(1-t)$ over $t \in (0,1)$381yields $t^{*} = a/(a+b)$, which is \eqref{eq:gen-dstar}. Substituting back382and completing each term to a KL divergence against the mixture383$(p_{\mathrm{data}} + p_g)/2$ gives \eqref{eq:gen-jsd};384see~\cite{goodfellow2014gan}.385\end{proof}386387\subsection{Training in practice: non-saturating and Wasserstein losses}388389Early in training, $D$ rejects fakes easily, $D(G(\vect{z})) \approx 0$,390and the generator's term $\log(1 - D(G(\vect{z})))$ in391\eqref{eq:gen-minimax} saturates — its gradient vanishes exactly when the392generator most needs guidance. The standard remedy, proposed already393in~\cite{goodfellow2014gan}, is the \emph{non-saturating} generator loss:394instead of minimizing $\E[\log(1 - D(G(\vect{z})))]$, the generator395maximizes396\begin{equation}397 \Loss_G^{\mathrm{NS}}398 = \E_{\vect{z} \sim p_{\vect{z}}}\bigl[\log D(G(\vect{z}))\bigr] ,399 \label{eq:gen-nonsat}400\end{equation}401which has the same fixed points but provides strong gradients precisely402when the discriminator is confident. The resulting estimation procedure,403Algorithm~\ref{alg:gen-gan}, alternates $k$ ascent steps on the404discriminator's objective with one non-saturating update of the generator.405406\begin{algorithm}[htbp]407 \caption{GAN alternating training with the non-saturating generator408 loss}409 \label{alg:gen-gan}410 \begin{algorithmic}[1]411 \Require generator $G$ (parameters $\theta_g$), discriminator $D$412 (parameters $\theta_d$), discriminator steps $k$, batch size $m$,413 learning rates $\eta_d, \eta_g$414 \While{not converged}415 \For{$j = 1, \dots, k$} \Comment{discriminator updates}416 \State sample $\{\vect{x}^{(1)}, \dots, \vect{x}^{(m)}\}$ from the417 data, $\{\vect{z}^{(1)}, \dots, \vect{z}^{(m)}\}$ from418 $p_{\vect{z}}$419 \State $\Loss_D \gets \frac{1}{m}\sum_{i=1}^{m}420 \Bigl[ \log D\!\left(\vect{x}^{(i)}\right)421 + \log\!\left(1 - D\!\left(G(\vect{z}^{(i)})\right)\right)422 \Bigr]$423 \Comment{value of \eqref{eq:gen-minimax}}424 \State $\theta_d \gets \theta_d + \eta_d\,425 \nabla_{\theta_d} \Loss_D$426 \Comment{gradient \emph{ascent}}427 \EndFor428 \State sample $\{\vect{z}^{(1)}, \dots, \vect{z}^{(m)}\}$ from429 $p_{\vect{z}}$430 \State $\Loss_G^{\mathrm{NS}} \gets \frac{1}{m}\sum_{i=1}^{m}431 \log D\!\left(G(\vect{z}^{(i)})\right)$432 \Comment{non-saturating loss \eqref{eq:gen-nonsat}}433 \State $\theta_g \gets \theta_g + \eta_g\,434 \nabla_{\theta_g} \Loss_G^{\mathrm{NS}}$435 \EndWhile436 \end{algorithmic}437\end{algorithm}438439A deeper pathology is that the Jensen--Shannon divergence in440\eqref{eq:gen-jsd} is poorly behaved when $p_{\mathrm{data}}$ and $p_g$441have (nearly) disjoint supports — as is typical for high-dimensional data442concentrated on low-dimensional manifolds — contributing to unstable443dynamics and \emph{mode collapse}, in which $G$ maps many latent vectors444onto a few high-scoring outputs. The Wasserstein GAN (WGAN) replaces the445JSD with the Wasserstein-1 distance, which by Kantorovich--Rubinstein446duality is447\begin{equation}448 W\bigl(p_{\mathrm{data}}, p_g\bigr)449 = \sup_{\lVert f \rVert_{L} \le 1}\;450 \E_{\vect{x} \sim p_{\mathrm{data}}}\bigl[f(\vect{x})\bigr]451 - \E_{\vect{x} \sim p_g}\bigl[f(\vect{x})\bigr] ,452 \label{eq:gen-wgan}453\end{equation}454where the supremum ranges over 1-Lipschitz functions. A \emph{critic}455network (a discriminator without the final sigmoid) approximates the456supremum in \eqref{eq:gen-wgan}, with the Lipschitz constraint enforced by457weight clipping or, in later refinements, a gradient penalty. Because $W$458remains finite and provides usable gradients even for disjoint supports,459WGAN training is markedly more stable and less prone to mode collapse.460461% ============================================================================462\section{Autoregressive Models and a Comparison of Families}463\label{sec:gen-ar}464465A third route to generation requires neither latent variables nor an466adversary: factor the joint density exactly by the chain rule,467\begin{equation}468 p_\theta(\vect{x})469 = \prod_{i=1}^{d} p_\theta\bigl(x_i \mid x_1, \dots, x_{i-1}\bigr) ,470 \label{eq:gen-ar}471\end{equation}472and train a network to model each conditional by maximum likelihood — the473factorization already encountered for sequence models, now applied to474arbitrary data (pixels in raster order, audio samples, discrete tokens).475Training is stable and parallelizable, and the likelihood476\eqref{eq:gen-ar} is exact; the price is sampling, which is inherently477sequential — one dimension at a time. Table~\ref{tab:gen-comparison}478summarizes the trade-offs among the three families479(see also~\cite{goodfellow2016book}).480481\begin{table}[ht]482 \centering483 \small484 \begin{tabular}{@{}l l l l l@{}}485 \toprule486 Family & Training objective & Sampling & Likelihood & Typical failure \\487 \midrule488 VAE & ELBO \eqref{eq:gen-elbo}, stable489 & one pass, fast & lower bound & blurry samples \\490 GAN & minimax \eqref{eq:gen-minimax}, unstable491 & one pass, fast & implicit (none) & mode collapse \\492 Autoregressive & exact MLE \eqref{eq:gen-ar}, stable493 & sequential, slow & exact & slow sampling \\494 \bottomrule495 \end{tabular}496 \caption{Comparison of the three classical generative families. Each497 optimizes a different surrogate of the same goal, $p_\theta \approx498 p_{\mathrm{data}}$, and each pays for its strengths with a499 characteristic weakness.}500 \label{tab:gen-comparison}501\end{table}502503% ============================================================================504\section{Toward Diffusion Models}\label{sec:gen-diffusion}505506The current state of the art in image, audio and video synthesis belongs507to a fourth family that turns generation into \emph{iterative denoising}.508A fixed forward process gradually destroys the data with Gaussian noise509over $T$ steps, according to a variance schedule $\beta_1, \dots,510\beta_T$:511\begin{equation}512 q(\vect{x}_t \mid \vect{x}_{t-1})513 = \mathcal{N}\bigl(\vect{x}_t;\;514 \sqrt{1 - \beta_t}\, \vect{x}_{t-1},\; \beta_t \mat{I}\bigr) .515 \label{eq:gen-diff-forward}516\end{equation}517Iterating \eqref{eq:gen-diff-forward} and writing $\alpha_t = 1 -518\beta_t$, $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$, the noisy state at519any timestep is available in closed form directly from the clean sample:520\begin{equation}521 q(\vect{x}_t \mid \vect{x}_0)522 = \mathcal{N}\bigl(\vect{x}_t;\;523 \sqrt{\bar{\alpha}_t}\, \vect{x}_0,\;524 (1 - \bar{\alpha}_t)\, \mat{I}\bigr)525 \;\Longleftrightarrow\;526 \vect{x}_t527 = \sqrt{\bar{\alpha}_t}\, \vect{x}_0528 + \sqrt{1 - \bar{\alpha}_t}\, \vect{\varepsilon},529 \quad \vect{\varepsilon} \sim \mathcal{N}(\vect{0}, \mat{I}) .530 \label{eq:gen-diff-closed}531\end{equation}532As $t \to T$, $\bar{\alpha}_t \to 0$ and the data dissolves into pure533noise. Generation runs the process in reverse: a learned Markov chain534$p_\theta(\vect{x}_{t-1} \mid \vect{x}_t)$ starts from535$\mathcal{N}(\vect{0},\mat{I})$ and removes the noise step by step, and a536variational argument in the spirit of \eqref{eq:gen-elbo} reduces its537training to a strikingly simple regression — predict the noise538$\vect{\varepsilon}$ injected in \eqref{eq:gen-diff-closed}:539\begin{equation}540 \Loss_{\mathrm{simple}}541 = \E_{t,\, \vect{x}_0,\, \vect{\varepsilon}}542 \Bigl[\bigl\lVert \vect{\varepsilon}543 - \vect{\varepsilon}_\theta(\vect{x}_t, t)544 \bigr\rVert_2^2\Bigr] .545 \label{eq:gen-diff-loss}546\end{equation}547Diffusion models thus combine the stable, likelihood-based training of the548VAE lineage with sample quality surpassing GANs, at the cost of an549iterative sampling procedure — a trade-off that a rapidly evolving550literature (accelerated samplers, latent-space diffusion, flow matching,551consistency models) continues to push toward few-step and even one-step552generation.553