# Recurrent Networks and Sequence Models

# 1. Vanilla Recurrent Neural Networks (Elman, Jordan)

Recurrent neural networks (RNNs) process sequences $x_1, x_2, \dots, x_T$ by maintaining a hidden state $h_t$ that acts as a compressed summary of everything seen so far. Two foundational architectures established this idea:

Jordan networks (Jordan, 1986, "Serial Order: A Parallel Distributed Processing Approach") feed the network's output back into a set of "state units" that serve as extra inputs at the next step.
Elman networks (Elman, 1990, "Finding Structure in Time", Cognitive Science 14:179–211) instead copy the hidden layer into "context units" and feed those back into the hidden layer — this is the modern "vanilla RNN."

Elman RNN equations. At each time step $t$:

$$h_t = \tanh\left(W_{hh}, h_{t-1} + W_{xh}, x_t + b_h\right)$$

$$y_t = W_{hy}, h_t + b_y \qquad \text{(often followed by a softmax: } \hat{y}t = \mathrm{softmax}(W{hy} h_t + b_y)\text{)}$$

where $x_t \in \mathbb{R}^d$ is the input, $h_t \in \mathbb{R}^n$ the hidden state, $W_{xh} \in \mathbb{R}^{n \times d}$, $W_{hh} \in \mathbb{R}^{n \times n}$, $W_{hy} \in \mathbb{R}^{m \times n}$. The crucial property is weight sharing across time: the same $(W_{hh}, W_{xh})$ are applied at every step, making the RNN a dynamical system $h_t = f(h_{t-1}, x_t; \theta)$ and, in principle, Turing-complete (Siegelmann & Sontag, 1995).

Jordan RNN differs only in the recurrence source:

$$h_t = \tanh\left(W_{hh}, y_{t-1} + W_{xh}, x_t + b_h\right), \qquad y_t = \sigma_y(W_{hy} h_t + b_y)$$

# 2. Backpropagation Through Time (BPTT) and the Vanishing/Exploding Gradient Problem

BPTT (Werbos, 1990, "Backpropagation through time: what it does and how to do it", Proc. IEEE) trains an RNN by unrolling it into a deep feedforward network with $T$ layers sharing the same weights, then applying standard backpropagation. For a loss $L = \sum_t L_t$, the gradient with respect to the recurrent matrix sums contributions over all time-step pairs:

$$\frac{\partial L}{\partial W_{hh}} = \sum_{t=1}^{T} \sum_{k=1}^{t} \frac{\partial L_t}{\partial h_t} \left( \prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}} \right) \frac{\partial h_k}{\partial W_{hh}}$$

The critical term is the product of Jacobians:

$$\frac{\partial h_t}{\partial h_k} = \prod_{i=k+1}^{t} \frac{\partial h_i}{\partial h_{i-1}} = \prod_{i=k+1}^{t} W_{hh}^\top , \mathrm{diag}!\left(\tanh'(a_i)\right)$$

Eigenvalue analysis (Bengio, Simard & Frasconi, 1994, "Learning long-term dependencies with gradient descent is difficult", IEEE Trans. Neural Networks; Pascanu, Mikolov & Bengio, 2013, "On the difficulty of training recurrent neural networks", ICML): let $\rho(W_{hh})$ be the spectral radius (largest absolute eigenvalue). Since $|\partial h_t / \partial h_k| \le (\sigma_{\max}(W_{hh}) \cdot \gamma)^{t-k}$ where $\gamma$ bounds $|\tanh'| \le 1$:

If the largest singular value satisfies $\sigma_{\max} < 1/\gamma$, gradients vanish exponentially: $|\partial h_t/\partial h_k| \sim \lambda^{t-k} \to 0$. It is sufficient for the largest eigenvalue to be $< 1$ for long-term components to vanish. The network then cannot learn dependencies longer than a few dozen steps.
If $\rho(W_{hh}) > 1$ (a necessary condition), gradients can explode exponentially, causing loss spikes and NaNs.

Gradient clipping (Pascanu et al., 2013) is the standard remedy for explosion — rescale the gradient when its norm exceeds a threshold $\tau$:

$$g \leftarrow \begin{cases} \dfrac{\tau}{|g|}, g & \text{if } |g| > \tau \ g & \text{otherwise} \end{cases}$$

Vanishing gradients have no such simple fix; they motivated gated architectures (LSTM/GRU), careful initialization (orthogonal/identity recurrent matrices), and truncated BPTT (backpropagating only $k$ steps, trading bias for tractability).

# 3. Long Short-Term Memory (LSTM)

Introduced by Hochreiter & Schmidhuber (1997, "Long Short-Term Memory", Neural Computation 9(8):1735–1780), the LSTM solves vanishing gradients with a cell state $c_t$ traversed by an additive (rather than multiplicative) recurrence — the "constant error carousel." The original 1997 paper had input and output gates only; the forget gate was added by Gers, Schmidhuber & Cummins (2000, "Learning to Forget: Continual Prediction with LSTM").

Complete equations of the standard (modern) LSTM, with $\sigma$ the logistic sigmoid and $\odot$ elementwise product:

$$f_t = \sigma\left(W_f x_t + U_f h_{t-1} + b_f\right) \qquad \text{(forget gate: how much of } c_{t-1} \text{ to keep)}$$

$$i_t = \sigma\left(W_i x_t + U_i h_{t-1} + b_i\right) \qquad \text{(input gate: how much new content to write)}$$

$$\tilde{c}t = \tanh\left(W_c x_t + U_c h{t-1} + b_c\right) \qquad \text{(candidate cell content)}$$

$$c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \qquad \text{(cell state update — additive path)}$$

$$o_t = \sigma\left(W_o x_t + U_o h_{t-1} + b_o\right) \qquad \text{(output gate: how much cell to expose)}$$

$$h_t = o_t \odot \tanh(c_t) \qquad \text{(hidden state)}$$

Why it works: the gradient through the cell path is $\partial c_t / \partial c_{t-1} = \mathrm{diag}(f_t)$ (plus gate-dependent terms). When $f_t \approx 1$, error flows back essentially unattenuated over hundreds of steps — no repeated multiplication by $W_{hh}$. A practical trick is initializing $b_f$ to a positive value (e.g., 1 or 2) so the network starts by remembering (Jozefowicz, Zaremba & Sutskever, 2015).

Peephole variant (Gers & Schmidhuber, 2000, "Recurrent Nets that Time and Count"): the gates also see the cell state directly, enabling precise timing behavior:

$$f_t = \sigma(W_f x_t + U_f h_{t-1} + V_f \odot c_{t-1} + b_f), \quad i_t = \sigma(W_i x_t + U_i h_{t-1} + V_i \odot c_{t-1} + b_i)$$

$$o_t = \sigma(W_o x_t + U_o h_{t-1} + V_o \odot c_{t} + b_o)$$

The large ablation study of Greff et al. (2017, "LSTM: A Search Space Odyssey", IEEE TNNLS) found the forget gate and output activation to be the most critical components, with most variants (including peepholes) not significantly beating the vanilla formulation.

# 4. Gated Recurrent Unit (GRU)

Proposed by Cho et al. (2014, "Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation", EMNLP; arXiv:1406.1078), the GRU merges the cell and hidden state and uses only two gates:

$$z_t = \sigma\left(W_z x_t + U_z h_{t-1} + b_z\right) \qquad \text{(update gate)}$$

$$r_t = \sigma\left(W_r x_t + U_r h_{t-1} + b_r\right) \qquad \text{(reset gate)}$$

$$\tilde{h}t = \tanh\left(W_h x_t + U_h (r_t \odot h{t-1}) + b_h\right) \qquad \text{(candidate state)}$$

$$h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde{h}_t$$

(The convex-combination convention $h_t = z_t \odot h_{t-1} + (1-z_t)\odot \tilde h_t$ also appears in the literature; the two are equivalent up to relabeling $z \leftrightarrow 1-z$.) The reset gate $r_t$ controls how much past state contributes to the candidate (allowing the unit to "forget" and act like a fresh network), while the update gate $z_t$ interpolates between copying $h_{t-1}$ and writing $\tilde{h}_t$ — the same leaky-integration principle as the LSTM's forget/input pair, with ~25% fewer parameters ($3$ weight blocks vs $4$). Empirically, GRU and LSTM perform comparably (Chung et al., 2014, "Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling"), with LSTM slightly more robust on tasks needing precise counting.

# 5. Bidirectional and Deep (Stacked) RNNs

Bidirectional RNNs (Schuster & Paliwal, 1997, "Bidirectional Recurrent Neural Networks", IEEE Trans. Signal Processing 45(11):2673–2681) run two independent RNNs over the sequence — one forward, one backward — and combine their states, so each output sees both past and future context:

$$\overrightarrow{h}t = f\left(\overrightarrow{W} x_t + \overrightarrow{U}, \overrightarrow{h}{t-1}\right), \qquad \overleftarrow{h}t = f\left(\overleftarrow{W} x_t + \overleftarrow{U}, \overleftarrow{h}{t+1}\right)$$

$$y_t = g\left(V, [\overrightarrow{h}_t ; \overleftarrow{h}_t] + b\right)$$

BiLSTMs (Graves & Schmidhuber, 2005) became the workhorse of speech recognition, tagging, and pre-Transformer contextual encoders (e.g., ELMo, 2018). They require the full sequence in advance, so they suit offline labeling, not streaming generation.

Deep (stacked) RNNs (Graves, Mohamed & Hinton, 2013, "Speech Recognition with Deep Recurrent Neural Networks") stack $L$ recurrent layers, layer $\ell$ taking layer $\ell-1$'s states as input:

$$h_t^{(\ell)} = f\left(W^{(\ell)} h_t^{(\ell-1)} + U^{(\ell)} h_{t-1}^{(\ell)} + b^{(\ell)}\right), \qquad h_t^{(0)} = x_t$$

This adds depth "vertically" (representation hierarchy) on top of depth "in time." Typical setups use 2–8 layers with dropout applied only to non-recurrent connections (Zaremba et al., 2014) or variational dropout with masks shared across time (Gal & Ghahramani, 2016).

# 6. Seq2Seq / Encoder–Decoder

The encoder–decoder paradigm was introduced concurrently by Cho et al. (2014) and Sutskever, Vinyals & Le (2014, "Sequence to Sequence Learning with Neural Networks", NeurIPS; arXiv:1409.3215). An encoder RNN consumes the source $x_1,\dots,x_{T_x}$ into a fixed vector $v = h_{T_x}$ (Sutskever used a 4-layer LSTM); a decoder RNN then models the target autoregressively:

$$p(y_1, \dots, y_{T'} \mid x_1, \dots, x_T) = \prod_{t=1}^{T'} p\left(y_t \mid v, y_1, \dots, y_{t-1}\right)$$

with $s_t = \mathrm{LSTM}(s_{t-1}, y_{t-1})$, $s_0$ initialized from $v$, and $p(y_t \mid \cdot) = \mathrm{softmax}(W_o s_t)$. Training maximizes log-likelihood with teacher forcing; inference uses beam search. Two findings from Sutskever et al. proved influential: (i) reversing the source sentence markedly improved BLEU (34.8 on WMT'14 En→Fr) by creating short-range dependencies that ease optimization; (ii) the fixed-size vector $v$ is an information bottleneck — performance degrades on long sentences — which directly motivated attention.

# 7. Attention: Bahdanau (2014) and Luong (2015)

Bahdanau, Cho & Bengio (2014/2015, "Neural Machine Translation by Jointly Learning to Align and Translate", ICLR 2015; arXiv:1409.0473) removed the bottleneck by letting the decoder attend to all encoder states $h_1, \dots, h_{T_x}$ (from a bidirectional GRU encoder). At decoder step $t$, with previous decoder state $s_{t-1}$:

Alignment scores (additive/MLP attention):

$$e_{tj} = a(s_{t-1}, h_j) = v_a^\top \tanh\left(W_a s_{t-1} + U_a h_j\right)$$

Softmax normalization into attention weights:

$$\alpha_{tj} = \frac{\exp(e_{tj})}{\sum_{k=1}^{T_x} \exp(e_{tk})}$$

Context vector (expected annotation):

$$c_t = \sum_{j=1}^{T_x} \alpha_{tj}, h_j$$

Decoder update and prediction:

$$s_t = f(s_{t-1}, y_{t-1}, c_t), \qquad p(y_t \mid \cdot) = \mathrm{softmax}\left(g(s_t, y_{t-1}, c_t)\right)$$

Luong, Pham & Manning (2015, "Effective Approaches to Attention-based Neural Machine Translation", EMNLP; arXiv:1508.04025) simplified and systematized this. Differences: attention uses the current decoder state $s_t$ (not $s_{t-1}$); the context is combined after the RNN step via $\tilde{h}_t = \tanh(W_c [c_t; s_t])$, then $p(y_t) = \mathrm{softmax}(W_s \tilde{h}_t)$. Luong proposed three score functions:

$$\mathrm{score}(s_t, h_j) = \begin{cases} s_t^\top h_j & \text{(dot)} \ s_t^\top W_a h_j & \text{(general)} \ v_a^\top \tanh\left(W_a [s_t; h_j]\right) & \text{(concat)} \end{cases}$$

plus global attention (over all source positions) versus local attention (a Gaussian-weighted window around a predicted position $p_t$). The dot-product form is the direct ancestor of Transformer attention $\mathrm{softmax}(QK^\top/\sqrt{d_k})V$ (Vaswani et al., 2017, "Attention Is All You Need"), which discarded recurrence entirely.

# 8. Echo State Networks, Reservoir Computing, Liquid State Machines

Reservoir computing sidesteps BPTT entirely: keep a large, random, fixed recurrent network (the reservoir) and train only a linear readout.

Echo State Networks (Jaeger, 2001, "The 'Echo State' Approach to Analysing and Training Recurrent Neural Networks", GMD Report 148):

$$h_t = (1-\alpha), h_{t-1} + \alpha \tanh\left(W_{\text{in}} x_t + W, h_{t-1}\right), \qquad y_t = W_{\text{out}} [x_t; h_t]$$

with leak rate $\alpha$, sparse random $W$ (~1% connectivity) rescaled so its spectral radius $\rho(W)$ is typically just below 1. This (heuristically) ensures the echo state property: the reservoir asymptotically washes out initial conditions and becomes a fading-memory function of the input history; larger $\rho$ gives longer memory, smaller $\rho$ shorter. Only $W_{\text{out}}$ is learned, in closed form by ridge regression:

$$W_{\text{out}} = Y H^\top \left(H H^\top + \lambda I\right)^{-1}$$

Training is thus convex, fast, and immune to vanishing gradients. ESNs excel at chaotic time-series prediction (e.g., Mackey–Glass; Jaeger & Haas, 2004, Science).

Liquid State Machines (Maass, Natschläger & Markram, 2002, "Real-Time Computing Without Stable States", Neural Computation 14:2531–2560) are the spiking-neuron, biologically motivated counterpart: a recurrent "liquid" of leaky integrate-and-fire neurons provides a high-dimensional temporal expansion of input spike trains; a memoryless readout is trained on the liquid state. Maass et al. proved universal real-time computing power given the separation property (liquid) and approximation property (readout). The field survives today in physical reservoir computing (photonic, memristive, mechanical reservoirs).

# 9. Hopfield Networks (1982) and Modern Hopfield Networks (2020)

Classical Hopfield network (Hopfield, 1982, "Neural networks and physical systems with emergent collective computational abilities", PNAS 79:2554–2558): a fully connected network of $N$ binary units $s_i \in {-1, +1}$ with symmetric weights ($w_{ij} = w_{ji}$, $w_{ii} = 0$) acting as content-addressable associative memory.

Energy function:

$$E = -\frac{1}{2} \sum_{i,j} w_{ij}, s_i s_j + \sum_i \theta_i s_i$$

Asynchronous update rule (pick a unit, update):

$$s_i \leftarrow \mathrm{sign}\left(\sum_j w_{ij} s_j - \theta_i\right)$$

Each update never increases $E$, so the dynamics converge to a local minimum — an attractor. Patterns ${\xi^\mu}{\mu=1}^{P}$ are stored via the Hebbian rule $w{ij} = \frac{1}{N} \sum_\mu \xi_i^\mu \xi_j^\mu$.

Capacity: reliable retrieval holds only up to $P_{\max} \approx 0.138, N$ patterns (Amit, Gutfreund & Sompolinsky, 1985, via spin-glass statistical mechanics); beyond this, spurious states and catastrophic interference dominate. (For essentially error-free storage the bound tightens to $N / (2 \ln N)$.)

Modern Hopfield networks (Ramsauer et al., 2020, "Hopfield Networks is All You Need", ICLR 2021; building on Krotov & Hopfield, 2016, dense associative memories with polynomial energies): continuous states $q \in \mathbb{R}^d$, stored patterns as columns of $X = [x_1, \dots, x_P]$, and the log-sum-exp energy

$$E(q) = -\frac{1}{\beta} \log \sum_{i=1}^{P} \exp\left(\beta, x_i^\top q\right) + \frac{1}{2} |q|^2 + \text{const}$$

whose update rule (a concave–convex procedure step) is

$$q^{\text{new}} = X, \mathrm{softmax}\left(\beta X^\top q\right)$$

This yields exponential storage capacity (in $d$), retrieval in typically one step, and — the celebrated result — is exactly the Transformer attention update with $q$ as query and $X$ providing keys/values, unifying associative memory and attention. Hopfield received the 2024 Nobel Prize in Physics (shared with Hinton) for this line of work.

# 10. Boltzmann Machines, RBMs, Contrastive Divergence, Deep Belief Networks

Boltzmann machines (Ackley, Hinton & Sejnowski, 1985, "A Learning Algorithm for Boltzmann Machines", Cognitive Science) are stochastic Hopfield networks with hidden units: binary units sampled from a Boltzmann distribution over the energy

$$p(s) = \frac{e^{-E(s)/T}}{Z}, \qquad Z = \sum_{s'} e^{-E(s')/T}$$

Exact learning requires intractable expectations over $Z$, so general Boltzmann machines were impractical.

Restricted Boltzmann Machines (Smolensky, 1986, as "Harmonium"; popularized by Hinton) impose a bipartite structure — visible units $v$, hidden units $h$, no intra-layer connections — with energy

$$E(v, h) = -\sum_i b_i v_i - \sum_j c_j h_j - \sum_{i,j} v_i, w_{ij}, h_j = -b^\top v - c^\top h - v^\top W h$$

and joint distribution $p(v,h) = e^{-E(v,h)}/Z$. Bipartiteness makes the conditionals factorize:

$$p(h_j = 1 \mid v) = \sigma\left(c_j + \sum_i w_{ij} v_i\right), \qquad p(v_i = 1 \mid h) = \sigma\left(b_i + \sum_j w_{ij} h_j\right)$$

enabling efficient block Gibbs sampling. The exact log-likelihood gradient is

$$\frac{\partial \log p(v)}{\partial w_{ij}} = \langle v_i h_j \rangle_{\text{data}} - \langle v_i h_j \rangle_{\text{model}}$$

The model term requires equilibrium sampling. Contrastive Divergence (Hinton, 2002, "Training Products of Experts by Minimizing Contrastive Divergence", Neural Computation 14:1771–1800) approximates it with just $k$ Gibbs steps (usually $k=1$) started from the data:

$$\Delta w_{ij} \propto \langle v_i h_j \rangle_{0} - \langle v_i h_j \rangle_{k} \qquad \text{(CD-}k\text{)}$$

Biased but effective; Persistent CD (Tieleman, 2008) improves the negative-phase samples.

Deep Belief Networks (Hinton, Osindero & Teh, 2006, "A Fast Learning Algorithm for Deep Belief Nets", Neural Computation 18:1527–1554): stack RBMs, training each layer greedily on the hidden activations of the layer below, then optionally fine-tune with backprop or wake–sleep. This greedy layer-wise unsupervised pre-training was the spark that launched the deep learning renaissance — it was the first practical recipe for training deep networks (pre-ReLU, pre-good-init), even though modern practice (ReLU, batch norm, residuals, large data) later made pre-training unnecessary for supervised tasks. Hinton's 2024 Nobel Prize citation prominently features Boltzmann machines.

# 11. Temporal Convolutional Networks (TCN)

Bai, Kolter & Koltun (2018, "An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling", arXiv:1803.01271) distilled convolutional sequence modeling (lineage: WaveNet, van den Oord et al., 2016) into a generic architecture and showed it outperforms LSTMs/GRUs on a broad benchmark suite while exhibiting longer effective memory. Ingredients:

Causal convolutions: output at time $t$ depends only on inputs $\le t$ (achieved by left padding).
Dilated convolutions: with dilation $d$ and kernel size $k$,

$$F(t) = \sum_{i=0}^{k-1} f(i) \cdot x_{t - d \cdot i}$$

Doubling $d$ per layer ($d = 1, 2, 4, 8, \dots$) makes the receptive field grow exponentially with depth: $R = 1 + (k-1)\sum_{\ell} d_\ell \approx 1 + (k-1)(2^L - 1)$.

Residual blocks (two dilated conv layers + weight norm + ReLU + dropout, with a $1{\times}1$ skip projection) stabilize deep stacks.

Trade-offs vs RNNs: TCNs train in parallel across time (no sequential state dependency), have stable gradients (backprop path length is depth, not sequence length — no temporal vanishing gradient), and offer flexible receptive-field control; but they need the whole input window at inference (larger memory for streaming), and history beyond the receptive field is truly gone, whereas an RNN's state can in principle carry unbounded history in $O(1)$ memory. The TCN paper, together with Transformers, drove the field's conclusion that recurrence is not necessary for most sequence tasks.

# 12. Neural Turing Machines and Differentiable Neural Computers

Neural Turing Machines (Graves, Wayne & Danihelka, 2014, "Neural Turing Machines", arXiv:1410.5401, DeepMind) couple a controller network (LSTM or feedforward) to an external memory matrix $M_t \in \mathbb{R}^{N \times W}$ through fully differentiable read/write heads, so the whole system trains end-to-end by gradient descent.

Reading is attention-weighted: with weighting $w_t$ over $N$ locations ($\sum_i w_t(i) = 1$),

$$r_t = \sum_i w_t(i), M_t(i)$$

Writing decomposes into erase ($e_t \in [0,1]^W$) and add ($a_t$) vectors:

$$\tilde{M}t(i) = M{t-1}(i)\left[\mathbf{1} - w_t(i), e_t\right], \qquad M_t(i) = \tilde{M}_t(i) + w_t(i), a_t$$

Addressing combines: (i) content-based — cosine similarity to an emitted key $k_t$, sharpened by $\beta_t$:

$$w_t^c(i) = \frac{\exp\left(\beta_t, K[k_t, M_t(i)]\right)}{\sum_j \exp\left(\beta_t, K[k_t, M_t(j)]\right)}, \qquad K[u,v] = \frac{u \cdot v}{|u|,|v|}$$

with (ii) location-based addressing: interpolation with the previous weighting ($g_t$), convolutional rotational shift ($s_t$), and sharpening ($\gamma_t$). NTMs learn algorithmic tasks — copy, repeat-copy, associative recall, sorting — and generalize to longer sequences than seen in training.

Differentiable Neural Computers (Graves et al., 2016, "Hybrid computing using a neural network with dynamic external memory", Nature 538:471–476) refine the NTM: they drop the location-shift mechanism and add

dynamic memory allocation via per-slot usage vectors $u_t$ (a differentiable "free list" enabling allocation and de-allocation),
a temporal link matrix $L_t \in [0,1]^{N \times N}$ recording write order, letting read heads step forward/backward through the sequence in which data was written,
multiple read heads combining content, forward, and backward modes.

DNCs solved bAbI question answering, graph traversal (e.g., London Underground shortest paths), and blocks-puzzle planning. Though hard to train and now superseded by Transformers in practice, NTM/DNC established the memory-augmented neural network paradigm and prefigured today's retrieval-augmented and tool-using architectures.

# Key references

Jordan, M. I. (1986). Serial Order: A Parallel Distributed Processing Approach. ICS Report 8604, UCSD.
Elman, J. L. (1990). Finding Structure in Time. Cognitive Science, 14(2), 179–211.
Werbos, P. (1990). Backpropagation Through Time: What It Does and How to Do It. Proc. IEEE, 78(10).
Bengio, Y., Simard, P., Frasconi, P. (1994). Learning Long-Term Dependencies with Gradient Descent is Difficult. IEEE Trans. Neural Networks, 5(2).
Hochreiter, S., Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780.
Schuster, M., Paliwal, K. K. (1997). Bidirectional Recurrent Neural Networks. IEEE Trans. Signal Processing, 45(11), 2673–2681.
Gers, F., Schmidhuber, J., Cummins, F. (2000). Learning to Forget: Continual Prediction with LSTM. Neural Computation, 12(10).
Jaeger, H. (2001). The "Echo State" Approach to Analysing and Training Recurrent Neural Networks. GMD Report 148.
Maass, W., Natschläger, T., Markram, H. (2002). Real-Time Computing Without Stable States. Neural Computation, 14(11), 2531–2560.
Hinton, G. E. (2002). Training Products of Experts by Minimizing Contrastive Divergence. Neural Computation, 14(8), 1771–1800.
Hinton, G. E., Osindero, S., Teh, Y. W. (2006). A Fast Learning Algorithm for Deep Belief Nets. Neural Computation, 18(7), 1527–1554.
Pascanu, R., Mikolov, T., Bengio, Y. (2013). On the Difficulty of Training Recurrent Neural Networks. ICML.
Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y. (2014). Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. EMNLP.
Sutskever, I., Vinyals, O., Le, Q. V. (2014). Sequence to Sequence Learning with Neural Networks. NeurIPS.
Bahdanau, D., Cho, K., Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. ICLR (arXiv:1409.0473, 2014).
Luong, M.-T., Pham, H., Manning, C. D. (2015). Effective Approaches to Attention-based Neural Machine Translation. EMNLP.
Graves, A., Wayne, G., Danihelka, I. (2014). Neural Turing Machines. arXiv:1410.5401.
Graves, A., et al. (2016). Hybrid Computing Using a Neural Network with Dynamic External Memory. Nature, 538, 471–476.
Greff, K., et al. (2017). LSTM: A Search Space Odyssey. IEEE TNNLS, 28(10).
Bai, S., Kolter, J. Z., Koltun, V. (2018). An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv:1803.01271.
Hopfield, J. J. (1982). Neural Networks and Physical Systems with Emergent Collective Computational Abilities. PNAS, 79(8), 2554–2558.
Amit, D. J., Gutfreund, H., Sompolinsky, H. (1985). Storing Infinite Numbers of Patterns in a Spin-Glass Model of Neural Networks. Phys. Rev. Lett., 55(14).
Ackley, D. H., Hinton, G. E., Sejnowski, T. J. (1985). A Learning Algorithm for Boltzmann Machines. Cognitive Science, 9(1).
Ramsauer, H., et al. (2021). Hopfield Networks is All You Need. ICLR (arXiv:2008.02217, 2020).

Sources consulted for validation: Pascanu et al. 2013 (arXiv), Sutskever et al. 2014 (arXiv), Bai et al. 2018 overview, Baeldung — Luong vs Bahdanau attention, LMU seminar — Attention for NLP, Scholarpedia — Echo State Network, Elman 1990 (PDF), Schuster & Paliwal 1997 (PDF), Modern Hopfield Networks (arXiv 2502.10122), GeeksforGeeks — Contrastive Divergence in RBMs, Tieleman — PCD (PDF), GM-RKB — GRU, Brain-inspired DNC (arXiv 2301.02809).