# Generative Neural Networks: A Comprehensive Technical Overview

Generative models learn to represent a data distribution $p_{\text{data}}(x)$ so that new samples can be drawn from it. The major families differ in how they represent the density: explicitly (autoregressive models, normalizing flows), approximately via a variational bound (VAEs), implicitly via a sampling procedure (GANs), through an unnormalized energy (EBMs), or through an iterative denoising process (diffusion and score-based models). This section covers each family with its core architecture, training objective, and canonical equations.

# 1. Autoencoders (AE)

An autoencoder (Rumelhart, Hinton & Williams, 1986; popularized for deep learning by Hinton & Salakhutdinov, 2006, "Reducing the Dimensionality of Data with Neural Networks") consists of an encoder $f_\phi: \mathcal{X} \to \mathcal{Z}$ mapping an input $x$ to a low-dimensional latent code $z = f_\phi(x)$, and a decoder $g_\theta: \mathcal{Z} \to \mathcal{X}$ producing a reconstruction $\hat{x} = g_\theta(z)$. Training minimizes the reconstruction loss:

$$\mathcal{L}{\text{AE}}(\theta, \phi) = \frac{1}{N}\sum{i=1}^{N} | x_i - g_\theta(f_\phi(x_i)) |_2^2$$

(or binary cross-entropy for Bernoulli-modeled pixels). The bottleneck $\dim(z) \ll \dim(x)$ forces the network to learn a compressed representation. A plain AE is not a true generative model — its latent space has no imposed prior structure — but it is the conceptual ancestor of the VAE. Key regularized variants:

Denoising Autoencoder (DAE) (Vincent et al., 2008, "Extracting and Composing Robust Features with Denoising Autoencoders"): the input is corrupted, $\tilde{x} \sim C(\tilde{x}|x)$ (e.g., Gaussian noise or masking), and the network must recover the clean input: $$\mathcal{L}{\text{DAE}} = \mathbb{E}{x, \tilde{x}} \left[ | x - g_\theta(f_\phi(\tilde{x})) |_2^2 \right]$$ Vincent (2011) showed the DAE implicitly learns the score $\nabla_x \log p(x)$ — a direct precursor of score-based diffusion models.
Sparse Autoencoder: adds an L1 penalty on activations, $\mathcal{L} = \mathcal{L}_{\text{rec}} + \lambda |z|_1$, or a KL penalty $\sum_j \mathrm{KL}(\rho ,|, \hat{\rho}_j)$ forcing the average activation $\hat{\rho}_j$ of each latent unit toward a small target sparsity $\rho$.
Contractive Autoencoder (CAE) (Rifai et al., 2011): penalizes the Frobenius norm of the encoder's Jacobian to make the representation locally invariant to input perturbations: $$\mathcal{L}{\text{CAE}} = \mathcal{L}{\text{rec}} + \lambda \left| \frac{\partial f_\phi(x)}{\partial x} \right|_F^2$$

# 2. Variational Autoencoders (VAE)

Reference: Kingma & Welling, 2013/2014, "Auto-Encoding Variational Bayes" (ICLR 2014); also Rezende, Mohamed & Wierstra, 2014, "Stochastic Backpropagation and Approximate Inference in Deep Generative Models".

The VAE posits a latent-variable model $p_\theta(x) = \int p_\theta(x|z), p(z), dz$ with prior $p(z) = \mathcal{N}(0, I)$. The marginal likelihood is intractable, so we introduce an approximate posterior $q_\phi(z|x)$ (the probabilistic encoder).

ELBO derivation. Starting from the log-likelihood and inserting $q_\phi$:

$$\log p_\theta(x) = \mathbb{E}{q\phi(z|x)}!\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] + D_{\mathrm{KL}}!\big(q_\phi(z|x),|,p_\theta(z|x)\big)$$

Since $D_{\mathrm{KL}} \geq 0$, the first term is the Evidence Lower BOund (ELBO):

$$\log p_\theta(x) ;\geq; \mathcal{L}{\text{ELBO}}(\theta, \phi; x) = \underbrace{\mathbb{E}{q_\phi(z|x)}\left[\log p_\theta(x|z)\right]}{\text{reconstruction}} ;-; \underbrace{D{\mathrm{KL}}!\big(q_\phi(z|x),|,p(z)\big)}_{\text{regularization}}$$

The gap between $\log p_\theta(x)$ and the ELBO is exactly $D_{\mathrm{KL}}(q_\phi(z|x)|p_\theta(z|x))$: maximizing the ELBO simultaneously maximizes the likelihood and tightens the posterior approximation.

Reparameterization trick. To backpropagate through the sampling step $z \sim q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \mathrm{diag}(\sigma_\phi^2(x)))$, sampling is rewritten as a deterministic function of the parameters plus exogenous noise:

$$z = \mu_\phi(x) + \sigma_\phi(x) \odot \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, I)$$

This yields a low-variance, unbiased pathwise gradient estimator of the ELBO with respect to $\phi$.

Closed-form KL for Gaussians (Appendix B of Kingma & Welling). For $q = \mathcal{N}(\mu, \mathrm{diag}(\sigma^2))$ and $p = \mathcal{N}(0, I)$ in $J$ dimensions:

$$D_{\mathrm{KL}}\big(q_\phi(z|x),|,\mathcal{N}(0,I)\big) = -\frac{1}{2}\sum_{j=1}^{J}\left(1 + \log \sigma_j^2 - \mu_j^2 - \sigma_j^2\right)$$

β-VAE (Higgins et al., 2017, "β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework", ICLR): weights the KL term with $\beta > 1$,

$$\mathcal{L}{\beta\text{-VAE}} = \mathbb{E}{q_\phi}[\log p_\theta(x|z)] - \beta , D_{\mathrm{KL}}(q_\phi(z|x),|,p(z))$$

which encourages disentangled latent factors at the cost of reconstruction fidelity.

VQ-VAE (van den Oord, Vinyals & Kavukcuoglu, 2017, "Neural Discrete Representation Learning", NeurIPS): replaces the continuous latent with a discrete codebook ${e_k}_{k=1}^{K}$. The encoder output $z_e(x)$ is quantized to its nearest code: $z_q(x) = e_k$ with $k = \arg\min_j |z_e(x) - e_j|_2$. Since quantization is non-differentiable, gradients are passed to the encoder via the straight-through estimator (copying the decoder's gradient past the quantizer). The loss has three terms:

$$\mathcal{L}_{\text{VQ-VAE}} = \underbrace{|x - D(z_q(x))|2^2}{\text{reconstruction}} + \underbrace{|,\mathrm{sg}[z_e(x)] - e,|2^2}{\text{codebook loss}} + \beta \underbrace{|,z_e(x) - \mathrm{sg}[e],|2^2}{\text{commitment loss}}$$

where $\mathrm{sg}[\cdot]$ is the stop-gradient operator. A powerful autoregressive prior (PixelCNN, later Transformers) is then fit over the discrete codes — the blueprint for DALL·E 1 and modern latent tokenizers. VQ-VAE-2 (Razavi et al., 2019) added a hierarchical codebook.

# 3. Generative Adversarial Networks (GAN)

Reference: Goodfellow et al., 2014, "Generative Adversarial Nets" (NeurIPS).

A generator $G(z)$, $z \sim p_z$ (e.g., $\mathcal{N}(0,I)$), and a discriminator $D(x) \in [0,1]$ play a two-player minimax game:

$$\min_G \max_D V(D, G) = \mathbb{E}{x \sim p{\text{data}}}\big[\log D(x)\big] + \mathbb{E}_{z \sim p_z}\big[\log\big(1 - D(G(z))\big)\big]$$

For a fixed $G$, the optimal discriminator is $D^(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}$, and substituting back shows the generator minimizes $2,\mathrm{JSD}(p_{\text{data}} | p_g) - \log 4$. The unique Nash equilibrium is $p_g = p_{\text{data}}$, at which $D^ \equiv 1/2$ and $V = -\log 4$.

Non-saturating loss. Early in training $D$ easily rejects fakes and $\log(1 - D(G(z)))$ saturates (vanishing gradients). Goodfellow proposed instead maximizing $\log D(G(z))$, i.e., the generator minimizes:

$$\mathcal{L}G^{\text{NS}} = -\mathbb{E}{z}\big[\log D(G(z))\big], \qquad \mathcal{L}D = -\mathbb{E}{x}[\log D(x)] - \mathbb{E}_{z}[\log(1 - D(G(z)))]$$

Mode collapse is the classic failure mode: $G$ maps many $z$'s to a few high-scoring outputs, covering only part of $p_{\text{data}}$'s modes. Causes include the JSD's poor behavior on disjoint supports and the alternating-gradient dynamics not converging to the Nash equilibrium.

Key variants:

DCGAN (Radford, Metz & Chintala, 2015, "Unsupervised Representation Learning with Deep Convolutional GANs"): architectural recipe — strided/transposed convolutions instead of pooling, batch normalization, ReLU/LeakyReLU, no fully-connected hidden layers — that made GAN training stable on images.
cGAN (Mirza & Osindero, 2014, "Conditional Generative Adversarial Nets"): both networks receive a condition $y$: $\min_G \max_D ; \mathbb{E}{x,y}[\log D(x|y)] + \mathbb{E}{z,y}[\log(1 - D(G(z|y)|y))]$.
WGAN (Arjovsky, Chintala & Bottou, 2017, "Wasserstein GAN"): replaces JSD with the Wasserstein-1 (Earth Mover) distance, which by Kantorovich–Rubinstein duality is

$$W(p_{\text{data}}, p_g) = \sup_{|f|L \leq 1} ; \mathbb{E}{x \sim p_{\text{data}}}[f(x)] - \mathbb{E}_{x \sim p_g}[f(x)]$$

The "critic" $f_w$ (no sigmoid) approximates the supremum; the 1-Lipschitz constraint was originally enforced by crude weight clipping ($w \leftarrow \mathrm{clip}(w, -c, c)$, e.g., $c = 0.01$). $W$ provides meaningful gradients even for distributions with disjoint supports, greatly reducing mode collapse and correlating with sample quality.

WGAN-GP (Gulrajani et al., 2017, "Improved Training of Wasserstein GANs"): replaces clipping with a gradient penalty enforcing $|\nabla D| \approx 1$ on interpolates $\hat{x} = \epsilon x + (1-\epsilon)\tilde{x}$, $\epsilon \sim U[0,1]$, $x \sim p_{\text{data}}$, $\tilde{x} = G(z)$:

$$\mathcal{L}{\text{critic}} = \mathbb{E}{\tilde{x}}[D(\tilde{x})] - \mathbb{E}{x}[D(x)] + \lambda , \mathbb{E}{\hat{x}}\Big[\big(|\nabla_{\hat{x}} D(\hat{x})|_2 - 1\big)^2\Big], \quad \lambda = 10$$

StyleGAN 1/2/3 (Karras et al., 2019 "A Style-Based Generator Architecture for GANs"; 2020 "Analyzing and Improving the Image Quality of StyleGAN"; 2021 "Alias-Free GANs"). StyleGAN1: an 8-layer MLP mapping network transforms $z \in \mathcal{Z}$ into an intermediate, more disentangled latent $w \in \mathcal{W}$; $w$ modulates each resolution level of the synthesis network via Adaptive Instance Normalization:

$$\mathrm{AdaIN}(x_i, y) = y_{s,i} , \frac{x_i - \mu(x_i)}{\sigma(x_i)} + y_{b,i}$$

where $(y_s, y_b)$ are affine projections of $w$, plus per-pixel noise injection for stochastic detail. StyleGAN2 removed AdaIN's "droplet" artifacts by replacing it with weight modulation/demodulation ($w'{ijk} = s_i \cdot w{ijk}$, then $w''{ijk} = w'{ijk} / \sqrt{\sum_{i,k} {w'_{ijk}}^2 + \epsilon}$), and added path-length regularization. StyleGAN3 fixed aliasing ("texture sticking") by treating features as continuous signals with proper low-pass filtering, achieving translation/rotation equivariance.

Pix2Pix (Isola et al., 2017, "Image-to-Image Translation with Conditional Adversarial Networks"): paired image translation with a cGAN plus an L1 term, $\mathcal{L} = \mathcal{L}_{\text{cGAN}} + \lambda , \mathbb{E}|y - G(x)|_1$, using a U-Net generator and PatchGAN discriminator.
CycleGAN (Zhu et al., 2017, "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks"): unpaired translation with two generators $G: X \to Y$, $F: Y \to X$ and a cycle-consistency loss:

$$\mathcal{L}{\text{cyc}}(G, F) = \mathbb{E}{x}\big[|F(G(x)) - x|1\big] + \mathbb{E}{y}\big[|G(F(y)) - y|_1\big]$$

added to the two adversarial losses with weight $\lambda$ (typically 10).

# 4. Normalizing Flows

A normalizing flow (Rezende & Mohamed, 2015, "Variational Inference with Normalizing Flows"; earlier Tabak & Vanden-Eijnden, 2010) builds an invertible, differentiable map $f: \mathcal{X} \to \mathcal{Z}$ from data to a simple base density $p_Z$ (standard Gaussian). The exact likelihood follows from the change-of-variables formula:

$$\log p_X(x) = \log p_Z(f(x)) + \log \left| \det \frac{\partial f(x)}{\partial x} \right|$$

For a composition $f = f_K \circ \cdots \circ f_1$, the log-determinants add: $\log p_X(x) = \log p_Z(z_K) + \sum_{k=1}^{K} \log |\det J_{f_k}|$. Training maximizes exact log-likelihood; sampling inverts the flow: $x = f^{-1}(z)$, $z \sim p_Z$. The design challenge is making $\det J$ computable in $O(D)$ instead of $O(D^3)$.

RealNVP (Dinh, Sohl-Dickstein & Bengio, 2016, "Density Estimation Using Real NVP"): affine coupling layers. Split $x$ into $(x_{1:d}, x_{d+1:D})$:

$$y_{1:d} = x_{1:d}, \qquad y_{d+1:D} = x_{d+1:D} \odot \exp\big(s(x_{1:d})\big) + t(x_{1:d})$$

where $s, t$ are arbitrary neural networks (never inverted). The Jacobian is lower triangular, so $\log|\det J| = \sum_j s(x_{1:d})j$, and inversion is trivial: $x{d+1:D} = (y_{d+1:D} - t) \odot \exp(-s)$. Alternating masks and multi-scale squeezing give expressivity.

Glow (Kingma & Dhariwal, 2018, "Glow: Generative Flow with Invertible 1×1 Convolutions", NeurIPS): each step = actnorm (per-channel affine) → invertible 1×1 convolution (a learned, LU-decomposed permutation generalization, with $\log|\det| = H \cdot W \cdot \log|\det W_{1\times1}|$) → affine coupling. Produced the first high-quality flow-based face samples and smooth latent interpolations.
Autoregressive flows. MAF (Papamakarios, Pavlakou & Murray, 2017, "Masked Autoregressive Flow for Density Estimation") uses $x_i = z_i \sigma_i(x_{1:i-1}) + \mu_i(x_{1:i-1})$: density evaluation is one parallel pass (fast training), but sampling is sequential. IAF (Kingma et al., 2016, "Improved Variational Inference with Inverse Autoregressive Flow") inverts the conditioning — $x_i = z_i \sigma_i(z_{1:i-1}) + \mu_i(z_{1:i-1})$ — making sampling parallel and density evaluation sequential; ideal as a flexible VAE posterior. Both are triangular-Jacobian flows: $\log|\det J| = \sum_i \log \sigma_i$.

# 5. Diffusion Models

# 5.1 DDPM

Reference: Ho, Jain & Abbeel, 2020, "Denoising Diffusion Probabilistic Models" (NeurIPS); building on Sohl-Dickstein et al., 2015, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics".

Forward (diffusion) process — a fixed Markov chain gradually adding Gaussian noise over $T$ steps (typically $T = 1000$) with variance schedule $\beta_1, \dots, \beta_T$:

$$q(x_t \mid x_{t-1}) = \mathcal{N}\big(x_t;; \sqrt{1 - \beta_t}, x_{t-1},; \beta_t I\big), \qquad q(x_{1:T}|x_0) = \prod_{t=1}^{T} q(x_t|x_{t-1})$$

With $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}t = \prod{s=1}^{t} \alpha_s$, one can jump directly to any $t$ (the key computational trick):

$$q(x_t \mid x_0) = \mathcal{N}\big(x_t;; \sqrt{\bar{\alpha}_t}, x_0,; (1 - \bar{\alpha}_t) I\big) \quad\Longleftrightarrow\quad x_t = \sqrt{\bar{\alpha}_t}, x_0 + \sqrt{1 - \bar{\alpha}_t}, \varepsilon,;; \varepsilon \sim \mathcal{N}(0, I)$$

Reverse (generative) process — a learned Markov chain starting from $p(x_T) = \mathcal{N}(0, I)$:

$$p_\theta(x_{t-1} \mid x_t) = \mathcal{N}\big(x_{t-1};; \mu_\theta(x_t, t),; \sigma_t^2 I\big)$$

The true posterior $q(x_{t-1}|x_t, x_0)$ is a tractable Gaussian with mean $\tilde{\mu}t(x_t, x_0) = \frac{\sqrt{\bar\alpha{t-1}}\beta_t}{1-\bar\alpha_t}x_0 + \frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}x_t$ and variance $\tilde{\beta}t = \frac{1-\bar\alpha{t-1}}{1-\bar\alpha_t}\beta_t$. Parameterizing the model to predict the noise $\varepsilon$ instead of the mean,

$$\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}t}}, \varepsilon\theta(x_t, t)\right)$$

the variational bound reduces (dropping time-dependent weights) to the remarkably simple loss:

$$\mathcal{L}{\text{simple}} = \mathbb{E}{t \sim U[1,T],, x_0,, \varepsilon \sim \mathcal{N}(0,I)} \Big[ \big| \varepsilon - \varepsilon_\theta\big(\sqrt{\bar{\alpha}_t}, x_0 + \sqrt{1 - \bar{\alpha}_t}, \varepsilon,; t\big) \big|^2 \Big]$$

— i.e., train a U-Net to predict the added noise, at a random timestep, in one step. Ho et al. used a linear schedule ($\beta_1 = 10^{-4}$ to $\beta_T = 0.02$); Nichol & Dhariwal (2021, "Improved DDPM") proposed the cosine schedule $\bar\alpha_t = \cos^2!\big(\frac{t/T + s}{1+s}\cdot\frac{\pi}{2}\big)$ and learned variances.

# 5.2 DDIM

Song, Meng & Ermon, 2020, "Denoising Diffusion Implicit Models" (ICLR 2021): defines a family of non-Markovian processes sharing DDPM's marginals (so the same trained $\varepsilon_\theta$ works). The update

$$x_{t-1} = \sqrt{\bar{\alpha}{t-1}} \underbrace{\left(\frac{x_t - \sqrt{1 - \bar{\alpha}t}, \varepsilon\theta(x_t, t)}{\sqrt{\bar{\alpha}t}}\right)}{\text{predicted } x_0} + \sqrt{1 - \bar{\alpha}{t-1} - \sigma_t^2}; \varepsilon_\theta(x_t, t) + \sigma_t \varepsilon_t$$

with $\sigma_t = 0$ gives a deterministic sampler (a probability-flow ODE discretization), enabling 10–50 step sampling instead of 1000 and semantically meaningful latent interpolation/inversion.

# 5.3 Score-based models and the SDE formulation

Song & Ermon, 2019, "Generative Modeling by Estimating Gradients of the Data Distribution" (NeurIPS): learn the score function $s_\theta(x) \approx \nabla_x \log p(x)$ via denoising score matching at multiple noise levels ${\sigma_i}$:

$$\mathcal{L} = \frac{1}{L}\sum_{i=1}^{L} \lambda(\sigma_i), \mathbb{E}{x, \tilde{x} \sim \mathcal{N}(x, \sigma_i^2 I)} \left[ \left| s\theta(\tilde{x}, \sigma_i) + \frac{\tilde{x} - x}{\sigma_i^2} \right|^2 \right]$$

and sample with annealed Langevin dynamics: $x_{k+1} = x_k + \frac{\eta}{2} s_\theta(x_k, \sigma) + \sqrt{\eta}, \varepsilon_k$.

Song et al., 2021, "Score-Based Generative Modeling through Stochastic Differential Equations" (ICLR, oral) unified DDPM and score matching in continuous time. Forward SDE: $dx = f(x, t), dt + g(t), dw$. By Anderson (1982), the reverse-time SDE is

$$dx = \big[f(x, t) - g(t)^2, \nabla_x \log p_t(x)\big], dt + g(t), d\bar{w}$$

DDPM corresponds to a variance-preserving SDE; NCSN to variance-exploding. There is also a deterministic probability-flow ODE, $dx = [f(x,t) - \tfrac{1}{2}g(t)^2 \nabla_x \log p_t(x)],dt$, with the same marginals — the basis for DDIM-style samplers and exact likelihoods. Note the identity $\nabla_{x_t} \log p(x_t) = -\varepsilon_\theta(x_t, t)/\sqrt{1 - \bar\alpha_t}$: noise prediction is score estimation.

# 5.4 Guidance

Classifier guidance (Dhariwal & Nichol, 2021, "Diffusion Models Beat GANs on Image Synthesis") shifts the score by $\nabla_{x_t} \log p_\phi(y|x_t)$ from an external classifier. Classifier-free guidance (Ho & Salimans, 2021/2022, "Classifier-Free Diffusion Guidance", NeurIPS workshop) instead trains one network with the condition randomly dropped ($y \to \varnothing$ with ~10% probability), then extrapolates at sampling time:

$$\tilde{\varepsilon}\theta(x_t, y) = (1 + w), \varepsilon\theta(x_t, y) - w, \varepsilon_\theta(x_t, \varnothing)$$

(equivalently $\varepsilon_\theta(x_t,\varnothing) + s,[\varepsilon_\theta(x_t,y) - \varepsilon_\theta(x_t,\varnothing)]$ with $s = 1 + w$). Larger $w$ trades diversity for fidelity/prompt-adherence; CFG is the workhorse of all modern text-to-image systems.

# 5.5 Latent diffusion / Stable Diffusion

Rombach et al., 2022, "High-Resolution Image Synthesis with Latent Diffusion Models" (CVPR): run diffusion not in pixel space but in the latent space of a pretrained perceptual autoencoder (KL- or VQ-regularized, ~8× spatial downsampling), slashing compute. Conditioning (text via a frozen CLIP encoder in Stable Diffusion) enters the denoising U-Net through cross-attention: $\mathrm{Attention}(Q, K, V) = \mathrm{softmax}(QK^\top/\sqrt{d})V$ with $Q$ from image features and $K, V$ from text embeddings. Loss: $\mathcal{L}{\text{LDM}} = \mathbb{E}{\mathcal{E}(x), \varepsilon, t}\big[|\varepsilon - \varepsilon_\theta(z_t, t, \tau_\theta(y))|^2\big]$.

# 5.6 Flow Matching

Lipman et al., 2023, "Flow Matching for Generative Modeling" (ICLR); concurrently Liu et al., 2022 ("Rectified Flow") and Albergo & Vanden-Eijnden, 2022. Instead of learning a score for an SDE, directly regress a velocity field $v_\theta(x, t)$ of an ODE $\frac{dx}{dt} = v_\theta(x, t)$ transporting noise $p_0 = \mathcal{N}(0,I)$ to data $p_1$. The marginal objective is intractable, but the Conditional Flow Matching loss has identical gradients ($\nabla_\theta \mathcal{L}{\text{CFM}} = \nabla\theta \mathcal{L}_{\text{FM}}$). With the simplest (optimal-transport / linear) path $x_t = (1-t)x_0 + t,x_1$:

$$\mathcal{L}{\text{CFM}}(\theta) = \mathbb{E}{t \sim U[0,1],, x_0 \sim p_0,, x_1 \sim p_{\text{data}}} \Big[ \big| v_\theta\big(x_t, t\big) - (x_1 - x_0) \big|^2 \Big]$$

Straight probability paths yield fewer ODE steps and simpler training; flow matching underlies Stable Diffusion 3, Flux, and Meta's Movie Gen.

# 6. Autoregressive Models

These models use the chain-rule factorization of the joint density — exact likelihood, sequential sampling:

$$p(x) = \prod_{i=1}^{n} p\big(x_i \mid x_1, \dots, x_{i-1}\big)$$

PixelRNN / PixelCNN (van den Oord, Kalchbrenner & Kavukcuoglu, 2016, "Pixel Recurrent Neural Networks", ICML best paper; and "Conditional Image Generation with PixelCNN Decoders", NeurIPS 2016): model images pixel by pixel in raster order, each pixel's 256-way (or mixture-of-logistics, PixelCNN++, Salimans et al. 2017) distribution conditioned on all previous pixels. PixelRNN uses row/diagonal LSTMs; PixelCNN uses masked convolutions (type A/B masks) for parallel training; gated PixelCNN fixes the blind spot with vertical + horizontal stacks.
WaveNet (van den Oord et al., 2016, "WaveNet: A Generative Model for Raw Audio"): the same factorization on raw audio samples, using stacks of dilated causal convolutions (dilations 1, 2, 4, …, 512, repeated) so the receptive field grows exponentially with depth, gated activations $z = \tanh(W_f * x) \odot \sigma(W_g * x)$, residual/skip connections, and 8-bit μ-law quantized outputs via softmax.

This paradigm — next-token prediction — scaled up over discrete tokens is precisely what powers GPT-style LLMs, and, combined with VQ tokenizers, image generators like DALL·E 1 and Parti (Yu et al., 2022).

# 7. Energy-Based Models (EBM)

References: LeCun et al., 2006, "A Tutorial on Energy-Based Learning"; Du & Mordatch, 2019, "Implicit Generation and Modeling with Energy-Based Models". An EBM defines a Boltzmann/Gibbs density via a scalar energy network $E_\theta$:

$$p_\theta(x) = \frac{e^{-E_\theta(x)}}{Z(\theta)}, \qquad Z(\theta) = \int e^{-E_\theta(x)}, dx$$

The partition function $Z$ is intractable, so maximum-likelihood gradients use the contrastive identity

$$\nabla_\theta \log p_\theta(x) = -\nabla_\theta E_\theta(x) + \mathbb{E}{x' \sim p\theta}\big[\nabla_\theta E_\theta(x')\big]$$

with negative samples $x'$ drawn by MCMC — typically Langevin dynamics, $x_{k+1} = x_k - \frac{\eta}{2}\nabla_x E_\theta(x_k) + \sqrt{\eta},\varepsilon_k$ (note $\nabla_x \log p_\theta(x) = -\nabla_x E_\theta(x)$: EBMs and score-based models are two views of the same object). Historical instances: Boltzmann machines and RBMs (Hinton) trained with contrastive divergence. EBMs are flexible (any architecture, easy composition of constraints) but slow to sample.

# 8. Recent Landmark Systems

DALL·E (Ramesh et al., 2021, "Zero-Shot Text-to-Image Generation"): a 12B discrete-VAE + autoregressive Transformer over text-and-image tokens. DALL·E 2 / unCLIP (Ramesh et al., 2022, "Hierarchical Text-Conditional Image Generation with CLIP Latents"): a diffusion prior maps text to CLIP image embeddings, then a diffusion decoder generates the image. DALL·E 3 (2023) emphasized recaptioned training data for prompt fidelity.
Imagen (Saharia et al., 2022, "Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding", NeurIPS): a frozen large language model (T5-XXL) as text encoder + a cascade of pixel-space diffusion models (64² → 256² → 1024²), with dynamic thresholding to allow high guidance weights. Key finding: scaling the text encoder matters more than scaling the image U-Net.
Diffusion Transformer (DiT) (Peebles & Xie, 2022/2023, "Scalable Diffusion Models with Transformers", ICCV): replaces the U-Net with a ViT over latent patches, conditioned via adaLN-Zero (adaptive layer norm whose scale/shift/gating come from timestep + class embeddings). FID scales smoothly with compute (Gflops), establishing transformers as the diffusion backbone.
Sora (OpenAI, 2024, technical report "Video Generation Models as World Simulators"; Sora 2 in 2025): a latent diffusion transformer over spacetime patches — video is compressed by a video autoencoder into a spatiotemporal latent, cut into patch tokens, and denoised by a scaled DiT — enabling variable durations, resolutions, and aspect ratios, with emergent 3D consistency and object permanence. The same DiT recipe underlies Stable Diffusion 3 (Esser et al., 2024, MM-DiT + rectified flow) and most 2024–2026 video models.
Consistency Models (Song, Dhariwal, Chen & Sutskever, 2023, "Consistency Models", ICML): learn a function $f_\theta(x_t, t)$ that maps any point on a probability-flow ODE trajectory directly to its origin, enforcing self-consistency $f_\theta(x_t, t) = f_\theta(x_{t'}, t')$ for all $t, t'$ on the same trajectory, with boundary condition $f_\theta(x_\epsilon, \epsilon) = x_\epsilon$. Trained by distilling a diffusion teacher (consistency distillation: $\mathcal{L} = \mathbb{E}[,d(f_\theta(x_{t_{n+1}}, t_{n+1}), f_{\theta^-}(\hat{x}_{t_n}, t_n)),]$ with an EMA target $\theta^-$) or standalone (consistency training). Result: one-step generation with quality approaching multi-step diffusion; successors include Latent Consistency Models (Luo et al., 2023) and sCM (Lu & Song, 2024).

# Summary Comparison

Family	Density	Sampling	Training	Weakness
VAE	Lower bound (ELBO)	1 pass, fast	Stable	Blurry samples
GAN	Implicit	1 pass, fast	Unstable (Nash)	Mode collapse
Flows	Exact	Fast	Stable (MLE)	Invertibility constrains architecture
Autoregressive	Exact	Slow (sequential)	Stable (MLE)	No latent; slow generation
EBM	Unnormalized	Slow (MCMC)	Contrastive, tricky	Intractable $Z$
Diffusion / FM	Bound / exact (ODE)	Many steps (→1 with consistency)	Very stable	Sampling cost

Diffusion (increasingly in its flow-matching formulation on transformer backbones) dominates image/video/audio generation as of 2026, while autoregressive transformers dominate text — and hybrids of the two (e.g., diffusion heads on AR backbones, AR-over-VQ-latents) are an active frontier.

Sources: