spb/neural-networks Public

1# Generative Neural Networks: A Comprehensive Technical Overview23Generative models learn to represent a data distribution $p_{\text{data}}(x)$ so that new samples can be drawn from it. The major families differ in how they represent the density: explicitly (autoregressive models, normalizing flows), approximately via a variational bound (VAEs), implicitly via a sampling procedure (GANs), through an unnormalized energy (EBMs), or through an iterative denoising process (diffusion and score-based models). This section covers each family with its core architecture, training objective, and canonical equations.45---67## 1. Autoencoders (AE)89An autoencoder (Rumelhart, Hinton & Williams, 1986; popularized for deep learning by Hinton & Salakhutdinov, 2006, "Reducing the Dimensionality of Data with Neural Networks") consists of an **encoder** $f_\phi: \mathcal{X} \to \mathcal{Z}$ mapping an input $x$ to a low-dimensional latent code $z = f_\phi(x)$, and a **decoder** $g_\theta: \mathcal{Z} \to \mathcal{X}$ producing a reconstruction $\hat{x} = g_\theta(z)$. Training minimizes the **reconstruction loss**:1011$$\mathcal{L}_{\text{AE}}(\theta, \phi) = \frac{1}{N}\sum_{i=1}^{N} \| x_i - g_\theta(f_\phi(x_i)) \|_2^2$$1213(or binary cross-entropy for Bernoulli-modeled pixels). The bottleneck $\dim(z) \ll \dim(x)$ forces the network to learn a compressed representation. A plain AE is *not* a true generative model — its latent space has no imposed prior structure — but it is the conceptual ancestor of the VAE. Key regularized variants:1415- **Denoising Autoencoder (DAE)** (Vincent et al., 2008, "Extracting and Composing Robust Features with Denoising Autoencoders"): the input is corrupted, $\tilde{x} \sim C(\tilde{x}|x)$ (e.g., Gaussian noise or masking), and the network must recover the clean input:16$$\mathcal{L}_{\text{DAE}} = \mathbb{E}_{x, \tilde{x}} \left[ \| x - g_\theta(f_\phi(\tilde{x})) \|_2^2 \right]$$17Vincent (2011) showed the DAE implicitly learns the score $\nabla_x \log p(x)$ — a direct precursor of score-based diffusion models.1819- **Sparse Autoencoder**: adds an L1 penalty on activations, $\mathcal{L} = \mathcal{L}_{\text{rec}} + \lambda \|z\|_1$, or a KL penalty $\sum_j \mathrm{KL}(\rho \,\|\, \hat{\rho}_j)$ forcing the average activation $\hat{\rho}_j$ of each latent unit toward a small target sparsity $\rho$.2021- **Contractive Autoencoder (CAE)** (Rifai et al., 2011): penalizes the Frobenius norm of the encoder's Jacobian to make the representation locally invariant to input perturbations:22$$\mathcal{L}_{\text{CAE}} = \mathcal{L}_{\text{rec}} + \lambda \left\| \frac{\partial f_\phi(x)}{\partial x} \right\|_F^2$$2324---2526## 2. Variational Autoencoders (VAE)2728**Reference:** Kingma & Welling, 2013/2014, "Auto-Encoding Variational Bayes" (ICLR 2014); also Rezende, Mohamed & Wierstra, 2014, "Stochastic Backpropagation and Approximate Inference in Deep Generative Models".2930The VAE posits a latent-variable model $p_\theta(x) = \int p_\theta(x|z)\, p(z)\, dz$ with prior $p(z) = \mathcal{N}(0, I)$. The marginal likelihood is intractable, so we introduce an approximate posterior $q_\phi(z|x)$ (the probabilistic encoder).3132**ELBO derivation.** Starting from the log-likelihood and inserting $q_\phi$:3334$$\log p_\theta(x) = \mathbb{E}_{q_\phi(z|x)}\!\left[\log \frac{p_\theta(x, z)}{q_\phi(z|x)}\right] + D_{\mathrm{KL}}\!\big(q_\phi(z|x)\,\|\,p_\theta(z|x)\big)$$3536Since $D_{\mathrm{KL}} \geq 0$, the first term is the **Evidence Lower BOund (ELBO)**:3738$$\log p_\theta(x) \;\geq\; \mathcal{L}_{\text{ELBO}}(\theta, \phi; x) = \underbrace{\mathbb{E}_{q_\phi(z|x)}\left[\log p_\theta(x|z)\right]}_{\text{reconstruction}} \;-\; \underbrace{D_{\mathrm{KL}}\!\big(q_\phi(z|x)\,\|\,p(z)\big)}_{\text{regularization}}$$3940The gap between $\log p_\theta(x)$ and the ELBO is exactly $D_{\mathrm{KL}}(q_\phi(z|x)\|p_\theta(z|x))$: maximizing the ELBO simultaneously maximizes the likelihood and tightens the posterior approximation.4142**Reparameterization trick.** To backpropagate through the sampling step $z \sim q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \mathrm{diag}(\sigma_\phi^2(x)))$, sampling is rewritten as a deterministic function of the parameters plus exogenous noise:4344$$z = \mu_\phi(x) + \sigma_\phi(x) \odot \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, I)$$4546This yields a low-variance, unbiased pathwise gradient estimator of the ELBO with respect to $\phi$.4748**Closed-form KL for Gaussians** (Appendix B of Kingma & Welling). For $q = \mathcal{N}(\mu, \mathrm{diag}(\sigma^2))$ and $p = \mathcal{N}(0, I)$ in $J$ dimensions:4950$$D_{\mathrm{KL}}\big(q_\phi(z|x)\,\|\,\mathcal{N}(0,I)\big) = -\frac{1}{2}\sum_{j=1}^{J}\left(1 + \log \sigma_j^2 - \mu_j^2 - \sigma_j^2\right)$$5152**β-VAE** (Higgins et al., 2017, "β-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework", ICLR): weights the KL term with $\beta > 1$,5354$$\mathcal{L}_{\beta\text{-VAE}} = \mathbb{E}_{q_\phi}[\log p_\theta(x|z)] - \beta \, D_{\mathrm{KL}}(q_\phi(z|x)\,\|\,p(z))$$5556which encourages **disentangled** latent factors at the cost of reconstruction fidelity.5758**VQ-VAE** (van den Oord, Vinyals & Kavukcuoglu, 2017, "Neural Discrete Representation Learning", NeurIPS): replaces the continuous latent with a **discrete codebook** $\{e_k\}_{k=1}^{K}$. The encoder output $z_e(x)$ is quantized to its nearest code: $z_q(x) = e_k$ with $k = \arg\min_j \|z_e(x) - e_j\|_2$. Since quantization is non-differentiable, gradients are passed to the encoder via the **straight-through estimator** (copying the decoder's gradient past the quantizer). The loss has three terms:5960$$\mathcal{L}_{\text{VQ-VAE}} = \underbrace{\|x - D(z_q(x))\|_2^2}_{\text{reconstruction}} + \underbrace{\|\,\mathrm{sg}[z_e(x)] - e\,\|_2^2}_{\text{codebook loss}} + \beta \underbrace{\|\,z_e(x) - \mathrm{sg}[e]\,\|_2^2}_{\text{commitment loss}}$$6162where $\mathrm{sg}[\cdot]$ is the stop-gradient operator. A powerful autoregressive prior (PixelCNN, later Transformers) is then fit over the discrete codes — the blueprint for DALL·E 1 and modern latent tokenizers. VQ-VAE-2 (Razavi et al., 2019) added a hierarchical codebook.6364---6566## 3. Generative Adversarial Networks (GAN)6768**Reference:** Goodfellow et al., 2014, "Generative Adversarial Nets" (NeurIPS).6970A generator $G(z)$, $z \sim p_z$ (e.g., $\mathcal{N}(0,I)$), and a discriminator $D(x) \in [0,1]$ play a two-player **minimax game**:7172$$\min_G \max_D V(D, G) = \mathbb{E}_{x \sim p_{\text{data}}}\big[\log D(x)\big] + \mathbb{E}_{z \sim p_z}\big[\log\big(1 - D(G(z))\big)\big]$$7374For a fixed $G$, the optimal discriminator is $D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)}$, and substituting back shows the generator minimizes $2\,\mathrm{JSD}(p_{\text{data}} \| p_g) - \log 4$. The unique **Nash equilibrium** is $p_g = p_{\text{data}}$, at which $D^* \equiv 1/2$ and $V = -\log 4$.7576**Non-saturating loss.** Early in training $D$ easily rejects fakes and $\log(1 - D(G(z)))$ saturates (vanishing gradients). Goodfellow proposed instead maximizing $\log D(G(z))$, i.e., the generator minimizes:7778$$\mathcal{L}_G^{\text{NS}} = -\mathbb{E}_{z}\big[\log D(G(z))\big], \qquad \mathcal{L}_D = -\mathbb{E}_{x}[\log D(x)] - \mathbb{E}_{z}[\log(1 - D(G(z)))]$$7980**Mode collapse** is the classic failure mode: $G$ maps many $z$'s to a few high-scoring outputs, covering only part of $p_{\text{data}}$'s modes. Causes include the JSD's poor behavior on disjoint supports and the alternating-gradient dynamics not converging to the Nash equilibrium.8182**Key variants:**8384- **DCGAN** (Radford, Metz & Chintala, 2015, "Unsupervised Representation Learning with Deep Convolutional GANs"): architectural recipe — strided/transposed convolutions instead of pooling, batch normalization, ReLU/LeakyReLU, no fully-connected hidden layers — that made GAN training stable on images.8586- **cGAN** (Mirza & Osindero, 2014, "Conditional Generative Adversarial Nets"): both networks receive a condition $y$: $\min_G \max_D \; \mathbb{E}_{x,y}[\log D(x|y)] + \mathbb{E}_{z,y}[\log(1 - D(G(z|y)|y))]$.8788- **WGAN** (Arjovsky, Chintala & Bottou, 2017, "Wasserstein GAN"): replaces JSD with the **Wasserstein-1 (Earth Mover) distance**, which by Kantorovich–Rubinstein duality is8990$$W(p_{\text{data}}, p_g) = \sup_{\|f\|_L \leq 1} \; \mathbb{E}_{x \sim p_{\text{data}}}[f(x)] - \mathbb{E}_{x \sim p_g}[f(x)]$$9192The "critic" $f_w$ (no sigmoid) approximates the supremum; the 1-Lipschitz constraint was originally enforced by crude **weight clipping** ($w \leftarrow \mathrm{clip}(w, -c, c)$, e.g., $c = 0.01$). $W$ provides meaningful gradients even for distributions with disjoint supports, greatly reducing mode collapse and correlating with sample quality.9394- **WGAN-GP** (Gulrajani et al., 2017, "Improved Training of Wasserstein GANs"): replaces clipping with a **gradient penalty** enforcing $\|\nabla D\| \approx 1$ on interpolates $\hat{x} = \epsilon x + (1-\epsilon)\tilde{x}$, $\epsilon \sim U[0,1]$, $x \sim p_{\text{data}}$, $\tilde{x} = G(z)$:9596$$\mathcal{L}_{\text{critic}} = \mathbb{E}_{\tilde{x}}[D(\tilde{x})] - \mathbb{E}_{x}[D(x)] + \lambda \, \mathbb{E}_{\hat{x}}\Big[\big(\|\nabla_{\hat{x}} D(\hat{x})\|_2 - 1\big)^2\Big], \quad \lambda = 10$$9798- **StyleGAN 1/2/3** (Karras et al., 2019 "A Style-Based Generator Architecture for GANs"; 2020 "Analyzing and Improving the Image Quality of StyleGAN"; 2021 "Alias-Free GANs"). StyleGAN1: an 8-layer MLP **mapping network** transforms $z \in \mathcal{Z}$ into an intermediate, more disentangled latent $w \in \mathcal{W}$; $w$ modulates each resolution level of the synthesis network via **Adaptive Instance Normalization**:99100$$\mathrm{AdaIN}(x_i, y) = y_{s,i} \, \frac{x_i - \mu(x_i)}{\sigma(x_i)} + y_{b,i}$$101102where $(y_s, y_b)$ are affine projections of $w$, plus per-pixel noise injection for stochastic detail. StyleGAN2 removed AdaIN's "droplet" artifacts by replacing it with **weight modulation/demodulation** ($w'_{ijk} = s_i \cdot w_{ijk}$, then $w''_{ijk} = w'_{ijk} / \sqrt{\sum_{i,k} {w'_{ijk}}^2 + \epsilon}$), and added path-length regularization. StyleGAN3 fixed **aliasing** ("texture sticking") by treating features as continuous signals with proper low-pass filtering, achieving translation/rotation equivariance.103104- **Pix2Pix** (Isola et al., 2017, "Image-to-Image Translation with Conditional Adversarial Networks"): paired image translation with a cGAN plus an L1 term, $\mathcal{L} = \mathcal{L}_{\text{cGAN}} + \lambda \, \mathbb{E}\|y - G(x)\|_1$, using a U-Net generator and PatchGAN discriminator.105106- **CycleGAN** (Zhu et al., 2017, "Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks"): *unpaired* translation with two generators $G: X \to Y$, $F: Y \to X$ and a **cycle-consistency loss**:107108$$\mathcal{L}_{\text{cyc}}(G, F) = \mathbb{E}_{x}\big[\|F(G(x)) - x\|_1\big] + \mathbb{E}_{y}\big[\|G(F(y)) - y\|_1\big]$$109110added to the two adversarial losses with weight $\lambda$ (typically 10).111112---113114## 4. Normalizing Flows115116A normalizing flow (Rezende & Mohamed, 2015, "Variational Inference with Normalizing Flows"; earlier Tabak & Vanden-Eijnden, 2010) builds an **invertible**, differentiable map $f: \mathcal{X} \to \mathcal{Z}$ from data to a simple base density $p_Z$ (standard Gaussian). The exact likelihood follows from the **change-of-variables formula**:117118$$\log p_X(x) = \log p_Z(f(x)) + \log \left| \det \frac{\partial f(x)}{\partial x} \right|$$119120For a composition $f = f_K \circ \cdots \circ f_1$, the log-determinants add: $\log p_X(x) = \log p_Z(z_K) + \sum_{k=1}^{K} \log |\det J_{f_k}|$. Training maximizes exact log-likelihood; sampling inverts the flow: $x = f^{-1}(z)$, $z \sim p_Z$. The design challenge is making $\det J$ computable in $O(D)$ instead of $O(D^3)$.121122- **RealNVP** (Dinh, Sohl-Dickstein & Bengio, 2016, "Density Estimation Using Real NVP"): **affine coupling layers**. Split $x$ into $(x_{1:d}, x_{d+1:D})$:123124$$y_{1:d} = x_{1:d}, \qquad y_{d+1:D} = x_{d+1:D} \odot \exp\big(s(x_{1:d})\big) + t(x_{1:d})$$125126where $s, t$ are arbitrary neural networks (never inverted). The Jacobian is lower triangular, so $\log|\det J| = \sum_j s(x_{1:d})_j$, and inversion is trivial: $x_{d+1:D} = (y_{d+1:D} - t) \odot \exp(-s)$. Alternating masks and multi-scale squeezing give expressivity.127128- **Glow** (Kingma & Dhariwal, 2018, "Glow: Generative Flow with Invertible 1×1 Convolutions", NeurIPS): each step = **actnorm** (per-channel affine) → **invertible 1×1 convolution** (a learned, LU-decomposed permutation generalization, with $\log|\det| = H \cdot W \cdot \log|\det W_{1\times1}|$) → affine coupling. Produced the first high-quality flow-based face samples and smooth latent interpolations.129130- **Autoregressive flows.** **MAF** (Papamakarios, Pavlakou & Murray, 2017, "Masked Autoregressive Flow for Density Estimation") uses $x_i = z_i \sigma_i(x_{1:i-1}) + \mu_i(x_{1:i-1})$: density evaluation is one parallel pass (fast training), but sampling is sequential. **IAF** (Kingma et al., 2016, "Improved Variational Inference with Inverse Autoregressive Flow") inverts the conditioning — $x_i = z_i \sigma_i(z_{1:i-1}) + \mu_i(z_{1:i-1})$ — making *sampling* parallel and density evaluation sequential; ideal as a flexible VAE posterior. Both are triangular-Jacobian flows: $\log|\det J| = \sum_i \log \sigma_i$.131132---133134## 5. Diffusion Models135136### 5.1 DDPM137138**Reference:** Ho, Jain & Abbeel, 2020, "Denoising Diffusion Probabilistic Models" (NeurIPS); building on Sohl-Dickstein et al., 2015, "Deep Unsupervised Learning using Nonequilibrium Thermodynamics".139140**Forward (diffusion) process** — a fixed Markov chain gradually adding Gaussian noise over $T$ steps (typically $T = 1000$) with variance schedule $\beta_1, \dots, \beta_T$:141142$$q(x_t \mid x_{t-1}) = \mathcal{N}\big(x_t;\; \sqrt{1 - \beta_t}\, x_{t-1},\; \beta_t I\big), \qquad q(x_{1:T}|x_0) = \prod_{t=1}^{T} q(x_t|x_{t-1})$$143144With $\alpha_t = 1 - \beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^{t} \alpha_s$, one can jump directly to any $t$ (the key computational trick):145146$$q(x_t \mid x_0) = \mathcal{N}\big(x_t;\; \sqrt{\bar{\alpha}_t}\, x_0,\; (1 - \bar{\alpha}_t) I\big) \quad\Longleftrightarrow\quad x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \varepsilon,\;\; \varepsilon \sim \mathcal{N}(0, I)$$147148**Reverse (generative) process** — a learned Markov chain starting from $p(x_T) = \mathcal{N}(0, I)$:149150$$p_\theta(x_{t-1} \mid x_t) = \mathcal{N}\big(x_{t-1};\; \mu_\theta(x_t, t),\; \sigma_t^2 I\big)$$151152The true posterior $q(x_{t-1}|x_t, x_0)$ is a tractable Gaussian with mean $\tilde{\mu}_t(x_t, x_0) = \frac{\sqrt{\bar\alpha_{t-1}}\beta_t}{1-\bar\alpha_t}x_0 + \frac{\sqrt{\alpha_t}(1-\bar\alpha_{t-1})}{1-\bar\alpha_t}x_t$ and variance $\tilde{\beta}_t = \frac{1-\bar\alpha_{t-1}}{1-\bar\alpha_t}\beta_t$. Parameterizing the model to predict the noise $\varepsilon$ instead of the mean,153154$$\mu_\theta(x_t, t) = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}}\, \varepsilon_\theta(x_t, t)\right)$$155156the variational bound reduces (dropping time-dependent weights) to the remarkably **simple loss**:157158$$\mathcal{L}_{\text{simple}} = \mathbb{E}_{t \sim U[1,T],\, x_0,\, \varepsilon \sim \mathcal{N}(0,I)} \Big[ \big\| \varepsilon - \varepsilon_\theta\big(\sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1 - \bar{\alpha}_t}\, \varepsilon,\; t\big) \big\|^2 \Big]$$159160— i.e., train a U-Net to predict the added noise, at a random timestep, in one step. Ho et al. used a **linear schedule** ($\beta_1 = 10^{-4}$ to $\beta_T = 0.02$); Nichol & Dhariwal (2021, "Improved DDPM") proposed the **cosine schedule** $\bar\alpha_t = \cos^2\!\big(\frac{t/T + s}{1+s}\cdot\frac{\pi}{2}\big)$ and learned variances.161162### 5.2 DDIM163164Song, Meng & Ermon, 2020, "Denoising Diffusion Implicit Models" (ICLR 2021): defines a family of **non-Markovian** processes sharing DDPM's marginals (so the same trained $\varepsilon_\theta$ works). The update165166$$x_{t-1} = \sqrt{\bar{\alpha}_{t-1}} \underbrace{\left(\frac{x_t - \sqrt{1 - \bar{\alpha}_t}\, \varepsilon_\theta(x_t, t)}{\sqrt{\bar{\alpha}_t}}\right)}_{\text{predicted } x_0} + \sqrt{1 - \bar{\alpha}_{t-1} - \sigma_t^2}\; \varepsilon_\theta(x_t, t) + \sigma_t \varepsilon_t$$167168with $\sigma_t = 0$ gives a **deterministic** sampler (a probability-flow ODE discretization), enabling 10–50 step sampling instead of 1000 and semantically meaningful latent interpolation/inversion.169170### 5.3 Score-based models and the SDE formulation171172Song & Ermon, 2019, "Generative Modeling by Estimating Gradients of the Data Distribution" (NeurIPS): learn the **score function** $s_\theta(x) \approx \nabla_x \log p(x)$ via denoising score matching at multiple noise levels $\{\sigma_i\}$:173174$$\mathcal{L} = \frac{1}{L}\sum_{i=1}^{L} \lambda(\sigma_i)\, \mathbb{E}_{x, \tilde{x} \sim \mathcal{N}(x, \sigma_i^2 I)} \left[ \left\| s_\theta(\tilde{x}, \sigma_i) + \frac{\tilde{x} - x}{\sigma_i^2} \right\|^2 \right]$$175176and sample with **annealed Langevin dynamics**: $x_{k+1} = x_k + \frac{\eta}{2} s_\theta(x_k, \sigma) + \sqrt{\eta}\, \varepsilon_k$.177178Song et al., 2021, "Score-Based Generative Modeling through Stochastic Differential Equations" (ICLR, oral) unified DDPM and score matching in continuous time. Forward SDE: $dx = f(x, t)\, dt + g(t)\, dw$. By Anderson (1982), the **reverse-time SDE** is179180$$dx = \big[f(x, t) - g(t)^2\, \nabla_x \log p_t(x)\big]\, dt + g(t)\, d\bar{w}$$181182DDPM corresponds to a variance-preserving SDE; NCSN to variance-exploding. There is also a deterministic **probability-flow ODE**, $dx = [f(x,t) - \tfrac{1}{2}g(t)^2 \nabla_x \log p_t(x)]\,dt$, with the same marginals — the basis for DDIM-style samplers and exact likelihoods. Note the identity $\nabla_{x_t} \log p(x_t) = -\varepsilon_\theta(x_t, t)/\sqrt{1 - \bar\alpha_t}$: noise prediction *is* score estimation.183184### 5.4 Guidance185186**Classifier guidance** (Dhariwal & Nichol, 2021, "Diffusion Models Beat GANs on Image Synthesis") shifts the score by $\nabla_{x_t} \log p_\phi(y|x_t)$ from an external classifier. **Classifier-free guidance** (Ho & Salimans, 2021/2022, "Classifier-Free Diffusion Guidance", NeurIPS workshop) instead trains one network with the condition randomly dropped ($y \to \varnothing$ with ~10% probability), then extrapolates at sampling time:187188$$\tilde{\varepsilon}_\theta(x_t, y) = (1 + w)\, \varepsilon_\theta(x_t, y) - w\, \varepsilon_\theta(x_t, \varnothing)$$189190(equivalently $\varepsilon_\theta(x_t,\varnothing) + s\,[\varepsilon_\theta(x_t,y) - \varepsilon_\theta(x_t,\varnothing)]$ with $s = 1 + w$). Larger $w$ trades diversity for fidelity/prompt-adherence; CFG is the workhorse of all modern text-to-image systems.191192### 5.5 Latent diffusion / Stable Diffusion193194Rombach et al., 2022, "High-Resolution Image Synthesis with Latent Diffusion Models" (CVPR): run diffusion not in pixel space but in the **latent space of a pretrained perceptual autoencoder** (KL- or VQ-regularized, ~8× spatial downsampling), slashing compute. Conditioning (text via a frozen CLIP encoder in Stable Diffusion) enters the denoising U-Net through **cross-attention**: $\mathrm{Attention}(Q, K, V) = \mathrm{softmax}(QK^\top/\sqrt{d})V$ with $Q$ from image features and $K, V$ from text embeddings. Loss: $\mathcal{L}_{\text{LDM}} = \mathbb{E}_{\mathcal{E}(x), \varepsilon, t}\big[\|\varepsilon - \varepsilon_\theta(z_t, t, \tau_\theta(y))\|^2\big]$.195196### 5.6 Flow Matching197198Lipman et al., 2023, "Flow Matching for Generative Modeling" (ICLR); concurrently Liu et al., 2022 ("Rectified Flow") and Albergo & Vanden-Eijnden, 2022. Instead of learning a score for an SDE, directly regress a **velocity field** $v_\theta(x, t)$ of an ODE $\frac{dx}{dt} = v_\theta(x, t)$ transporting noise $p_0 = \mathcal{N}(0,I)$ to data $p_1$. The marginal objective is intractable, but the **Conditional Flow Matching** loss has identical gradients ($\nabla_\theta \mathcal{L}_{\text{CFM}} = \nabla_\theta \mathcal{L}_{\text{FM}}$). With the simplest (optimal-transport / linear) path $x_t = (1-t)x_0 + t\,x_1$:199200$$\mathcal{L}_{\text{CFM}}(\theta) = \mathbb{E}_{t \sim U[0,1],\, x_0 \sim p_0,\, x_1 \sim p_{\text{data}}} \Big[ \big\| v_\theta\big(x_t, t\big) - (x_1 - x_0) \big\|^2 \Big]$$201202Straight probability paths yield fewer ODE steps and simpler training; flow matching underlies Stable Diffusion 3, Flux, and Meta's Movie Gen.203204---205206## 6. Autoregressive Models207208These models use the **chain-rule factorization** of the joint density — exact likelihood, sequential sampling:209210$$p(x) = \prod_{i=1}^{n} p\big(x_i \mid x_1, \dots, x_{i-1}\big)$$211212- **PixelRNN / PixelCNN** (van den Oord, Kalchbrenner & Kavukcuoglu, 2016, "Pixel Recurrent Neural Networks", ICML best paper; and "Conditional Image Generation with PixelCNN Decoders", NeurIPS 2016): model images pixel by pixel in raster order, each pixel's 256-way (or mixture-of-logistics, PixelCNN++, Salimans et al. 2017) distribution conditioned on all previous pixels. PixelRNN uses row/diagonal LSTMs; PixelCNN uses **masked convolutions** (type A/B masks) for parallel training; gated PixelCNN fixes the blind spot with vertical + horizontal stacks.213214- **WaveNet** (van den Oord et al., 2016, "WaveNet: A Generative Model for Raw Audio"): the same factorization on raw audio samples, using stacks of **dilated causal convolutions** (dilations 1, 2, 4, …, 512, repeated) so the receptive field grows exponentially with depth, gated activations $z = \tanh(W_f * x) \odot \sigma(W_g * x)$, residual/skip connections, and 8-bit μ-law quantized outputs via softmax.215216This paradigm — next-token prediction — scaled up over discrete tokens is precisely what powers GPT-style LLMs, and, combined with VQ tokenizers, image generators like DALL·E 1 and Parti (Yu et al., 2022).217218---219220## 7. Energy-Based Models (EBM)221222**References:** LeCun et al., 2006, "A Tutorial on Energy-Based Learning"; Du & Mordatch, 2019, "Implicit Generation and Modeling with Energy-Based Models". An EBM defines a Boltzmann/Gibbs density via a scalar energy network $E_\theta$:223224$$p_\theta(x) = \frac{e^{-E_\theta(x)}}{Z(\theta)}, \qquad Z(\theta) = \int e^{-E_\theta(x)}\, dx$$225226The partition function $Z$ is intractable, so maximum-likelihood gradients use the contrastive identity227228$$\nabla_\theta \log p_\theta(x) = -\nabla_\theta E_\theta(x) + \mathbb{E}_{x' \sim p_\theta}\big[\nabla_\theta E_\theta(x')\big]$$229230with negative samples $x'$ drawn by MCMC — typically **Langevin dynamics**, $x_{k+1} = x_k - \frac{\eta}{2}\nabla_x E_\theta(x_k) + \sqrt{\eta}\,\varepsilon_k$ (note $\nabla_x \log p_\theta(x) = -\nabla_x E_\theta(x)$: EBMs and score-based models are two views of the same object). Historical instances: Boltzmann machines and RBMs (Hinton) trained with contrastive divergence. EBMs are flexible (any architecture, easy composition of constraints) but slow to sample.231232---233234## 8. Recent Landmark Systems235236- **DALL·E** (Ramesh et al., 2021, "Zero-Shot Text-to-Image Generation"): a 12B discrete-VAE + autoregressive Transformer over text-and-image tokens. **DALL·E 2 / unCLIP** (Ramesh et al., 2022, "Hierarchical Text-Conditional Image Generation with CLIP Latents"): a diffusion *prior* maps text to CLIP image embeddings, then a diffusion decoder generates the image. DALL·E 3 (2023) emphasized recaptioned training data for prompt fidelity.237238- **Imagen** (Saharia et al., 2022, "Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding", NeurIPS): a frozen large language model (T5-XXL) as text encoder + a cascade of pixel-space diffusion models (64² → 256² → 1024²), with **dynamic thresholding** to allow high guidance weights. Key finding: scaling the *text encoder* matters more than scaling the image U-Net.239240- **Diffusion Transformer (DiT)** (Peebles & Xie, 2022/2023, "Scalable Diffusion Models with Transformers", ICCV): replaces the U-Net with a ViT over latent patches, conditioned via **adaLN-Zero** (adaptive layer norm whose scale/shift/gating come from timestep + class embeddings). FID scales smoothly with compute (Gflops), establishing transformers as the diffusion backbone.241242- **Sora** (OpenAI, 2024, technical report "Video Generation Models as World Simulators"; Sora 2 in 2025): a latent **diffusion transformer** over **spacetime patches** — video is compressed by a video autoencoder into a spatiotemporal latent, cut into patch tokens, and denoised by a scaled DiT — enabling variable durations, resolutions, and aspect ratios, with emergent 3D consistency and object permanence. The same DiT recipe underlies Stable Diffusion 3 (Esser et al., 2024, MM-DiT + rectified flow) and most 2024–2026 video models.243244- **Consistency Models** (Song, Dhariwal, Chen & Sutskever, 2023, "Consistency Models", ICML): learn a function $f_\theta(x_t, t)$ that maps *any* point on a probability-flow ODE trajectory directly to its origin, enforcing **self-consistency** $f_\theta(x_t, t) = f_\theta(x_{t'}, t')$ for all $t, t'$ on the same trajectory, with boundary condition $f_\theta(x_\epsilon, \epsilon) = x_\epsilon$. Trained by distilling a diffusion teacher (consistency distillation: $\mathcal{L} = \mathbb{E}[\,d(f_\theta(x_{t_{n+1}}, t_{n+1}), f_{\theta^-}(\hat{x}_{t_n}, t_n))\,]$ with an EMA target $\theta^-$) or standalone (consistency training). Result: **one-step generation** with quality approaching multi-step diffusion; successors include Latent Consistency Models (Luo et al., 2023) and sCM (Lu & Song, 2024).245246---247248## Summary Comparison249250| Family | Density | Sampling | Training | Weakness |251|---|---|---|---|---|252| VAE | Lower bound (ELBO) | 1 pass, fast | Stable | Blurry samples |253| GAN | Implicit | 1 pass, fast | Unstable (Nash) | Mode collapse |254| Flows | Exact | Fast | Stable (MLE) | Invertibility constrains architecture |255| Autoregressive | Exact | Slow (sequential) | Stable (MLE) | No latent; slow generation |256| EBM | Unnormalized | Slow (MCMC) | Contrastive, tricky | Intractable $Z$ |257| Diffusion / FM | Bound / exact (ODE) | Many steps (→1 with consistency) | Very stable | Sampling cost |258259Diffusion (increasingly in its flow-matching formulation on transformer backbones) dominates image/video/audio generation as of 2026, while autoregressive transformers dominate text — and hybrids of the two (e.g., diffusion heads on AR backbones, AR-over-VQ-latents) are an active frontier.260261**Sources:**262- [DDPM theory (LearnOpenCV)](https://learnopencv.com/denoising-diffusion-probabilistic-models/), [Improved DDPM (Nichol & Dhariwal)](https://arxiv.org/pdf/2102.09672), [Lecture Notes in Probabilistic Diffusion Models](https://arxiv.org/pdf/2312.10393)263- [Kingma & Welling VAE slides](https://berkeley-deep-learning.github.io/cs294-131-s17/slides/VAE%20talk.compressed.pdf), [ELBO derivation tutorial](https://arxiv.org/pdf/1907.08956), [Reparameterization trick (Gundersen)](https://gregorygundersen.com/blog/2018/04/29/reparameterization/)264- [WGAN-GP overview (EmergentMind)](https://www.emergentmind.com/topics/wasserstein-gan-loss-with-gradient-penalty-wgan-gp), [Gradient penalty analysis](https://arxiv.org/pdf/1910.06922)265- [VQ-VAE explained](https://leeyngdo.github.io/blog/generative-model/2023-09-02-VQ-VAE/), [VQ-VAE-2 paper](http://papers.neurips.cc/paper/9625-generating-diverse-high-fidelity-images-with-vq-vae-2.pdf), [HuggingFace VQ post](https://huggingface.co/blog/ariG23498/understand-vq)266- [Classifier-Free Diffusion Guidance (arXiv 2207.12598)](https://arxiv.org/abs/2207.12598), [CFG notes (Khungurn)](https://pkhungurn.github.io/notes/notes/ml/ddpm-classifier-free-guidance/ddpm-classifier-free-guidance.pdf)267- [StyleGAN2 paper (Karras et al.)](https://users.aalto.fi/~laines9/publications/karras2020cvpr_paper.pdf), [StyleGAN3 project page](https://nvlabs.github.io/stylegan3/), [StyleGAN3 explained](https://medium.com/@steinsfu/stylegan3-clearly-explained-793edbcc8048)268- [Score SDE (arXiv 2011.13456)](https://arxiv.org/abs/2011.13456), [score_sde repo](https://github.com/yang-song/score_sde)269- [Consistency Models (arXiv 2303.01469)](https://arxiv.org/abs/2303.01469), [ICML PMLR version](https://proceedings.mlr.press/v202/song23a.html), [openai/consistency_models](https://github.com/openai/consistency_models)270- Flow matching: [closed-form analysis (OpenReview)](https://openreview.net/pdf?id=kVz9uvqUna), [CFM for Bayesian inference](https://arxiv.org/pdf/2510.09534)271