RNNs, LSTMs and Sequence Models

“The dog bit the man” and “the man bit the dog” contain exactly the same words. A model that counts words — a bag of words, or the dense networks on the previous pages fed a list of word counts — cannot tell them apart, yet one is an everyday event and the other is news. Sentences, speech, stock prices, heartbeats, rainfall readings and DNA all share this property: the order is part of the meaning. This page is about the family of networks built to read things in order, and about the flaw that eventually made the field look for something else.

Sequences, and a network with memory

A sequence is a list where position matters: x₁, x₂, …, x_T. Each x_t might be a word’s embedding, an audio frame or one day’s temperature. Two naive approaches fail. Feeding a dense network the whole sequence at once fixes the length and gives every position its own weights, so a pattern learned at position 3 is unknown at position 30. Averaging or counting throws away the order entirely.

A recurrent neural network takes a third way. It reads one element at a time and keeps a hidden state h, a vector that summarises everything seen so far. At each step it combines the new input with the old state:

The crucial detail is that the same weights are used at every step. That is weight sharing again — the idea behind convolutional networks, applied across time instead of space. A pattern learned anywhere in a sequence can be recognised anywhere else, and the network handles sequences of any length with a fixed number of parameters. To see it work, here is a tiny RNN with a three-number memory, run on two sequences containing the same values in a different order:

import numpy as np

np.random.seed(0)
W_xh = np.random.randn(3, 1) * 0.8            # input (1 number) -> hidden (3 numbers)
W_hh = np.random.randn(3, 3) * 0.8            # hidden -> hidden: the memory loop
b = np.zeros(3)

def run_rnn(sequence):
    h = np.zeros(3)                           # empty memory before the first step
    for t, x in enumerate(sequence):
        h = np.tanh(W_xh @ [x] + W_hh @ h + b)   # same weights at every step
        print(f"  t={t} x={x}  h={np.round(h, 3)}")
    return h

print("sequence 1, 0, 0:")
h1 = run_rnn([1, 0, 0])
print("sequence 0, 0, 1:")
h2 = run_rnn([0, 0, 1])

sequence 1, 0, 0:
  t=0 x=1  h=[0.888 0.31  0.654]
  t=1 x=0  h=[0.913 0.525 0.796]
  t=2 x=0  h=[0.947 0.511 0.858]
sequence 0, 0, 1:
  t=0 x=0  h=[0. 0. 0.]
  t=1 x=0  h=[0. 0. 0.]
  t=2 x=1  h=[0.888 0.31  0.654]

Read it step by step. In the first sequence, the 1 arrives first and writes a pattern into the memory; the following zeros contribute no new input, but W_hh keeps mixing the memory with itself, so the state evolves and ends at [0.947, 0.511, 0.858]. In the second sequence, nothing happens until the last step — zero input and a zero state give tanh(0) = 0 — and then the 1 arrives and produces [0.888, 0.31, 0.654], the same numbers the first sequence had at its first step. Same contents, different order, different final memory. A classifier reading that final state can now tell the dog from the man.

Drawing the loop as a chain — one copy of the network per time step, each passing its state to the next — is called unrolling. An unrolled RNN over 50 steps looks like a 50-layer network in which every layer shares the same weights. Hold on to that picture: it explains both how RNNs are trained and why they struggle.

Backpropagation through time and vanishing gradients

To train an RNN, unroll it over the sequence, compute the loss, and run ordinary backpropagation (see Backpropagation and Gradient Descent) backwards through the unrolled chain. That is all backpropagation through time (BPTT) is. Because the weights are shared, each weight’s gradient is the sum of its contributions at every time step. For long sequences, practitioners often cut the chain into windows of, say, 100 steps (truncated BPTT) to limit memory use.

The trouble lies in the chain rule. The gradient that tells step 1 how it affected the loss at step 50 is a product of 49 factors, one per step, each involving W_hh and the slope of tanh. Multiply many numbers together and the result is ruled by whether they are typically below or above 1:

for factor in (0.5, 0.9, 1.0, 1.1, 1.5):
    row = "  ".join(f"{factor ** t:10.3g}" for t in (1, 10, 50, 100))
    print(f"factor {factor}:  {row}")

factor 0.5:         0.5    0.000977    8.88e-16    7.89e-31
factor 0.9:         0.9       0.349     0.00515    2.66e-05
factor 1.0:           1           1           1           1
factor 1.1:         1.1        2.59         117    1.38e+04
factor 1.5:         1.5        57.7    6.38e+08    4.07e+17

A factor of 0.9 — barely below 1 — leaves 0.005 after 50 steps and 0.00003 after 100. A factor of 1.1 grows to 117 and then 13,800. Only exactly 1 is stable, and nothing in training keeps it there. With matrices instead of single numbers the same thing happens, governed by how much W_hh stretches or shrinks vectors. Here is the real gradient from an RNN, computed by PyTorch’s autograd: how much does the state after t steps depend on the very first input?

import torch

torch.manual_seed(0)
for scale in (0.5, 1.0, 3.0):
    W_hh = torch.randn(16, 16) * scale / 16 ** 0.5     # scale sets the size of W_hh
    x0 = torch.randn(16, requires_grad=True)           # the first input in the sequence
    h = torch.tanh(x0)
    grads = []
    for t in range(1, 101):
        h = torch.tanh(W_hh @ h)                       # 100 steps, no new input
        if t in (1, 10, 50, 100):
            g, = torch.autograd.grad(h.sum(), x0, retain_graph=True)
            grads.append(f"{g.norm().item():9.2e}")
    print(f"scale {scale}: gradient size after 1/10/50/100 steps:", " ".join(grads))

scale 0.5: gradient size after 1/10/50/100 steps:  9.85e-01  1.74e-03  1.00e-15  0.00e+00
scale 1.0: gradient size after 1/10/50/100 steps:  2.98e+00  2.08e+00  1.41e-01  1.98e+00
scale 3.0: gradient size after 1/10/50/100 steps:  2.98e+00  4.48e+01  5.08e+06  7.92e+09

With small recurrent weights (scale 0.5), the first input’s influence falls a thousandfold within ten steps and is a round zero by step 100 — the floating-point number underflowed. With large weights (scale 3.0) it explodes to almost 10¹⁰. Only around scale 1.0 does it survive, wobbling. This is the vanishing-gradient problem (and its twin, exploding gradients), identified by Sepp Hochreiter in 1991 and analysed by Bengio, Simard and Frasconi in 1994. Its practical meaning is harsh: a plain RNN cannot learn that a word at the start of a paragraph matters for a word at its end, because the training signal connecting them has vanished before it arrives.

Exploding gradients have a crude but effective fix: gradient clipping. If the gradient’s overall length exceeds a threshold, scale it down before the update — in PyTorch, one line, torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0), between backward() and step(). Clipping is still standard when training large models of every kind. Vanishing gradients need a change of architecture.

LSTM gates and the GRU

Sepp Hochreiter and Jürgen Schmidhuber’s answer, published in 1997, was the long short-term memory unit (the history is on Backprop and the Connectionist Revival). Its key move is a second state, the cell state c, which is updated mostly by addition rather than by repeated multiplication through a weight matrix. Small learned “gates” — sigmoid layers outputting numbers between 0 (closed) and 1 (open) — decide what to erase, what to write and what to reveal. The forget gate, now part of every LSTM, was added by Felix Gers, Schmidhuber and Fred Cummins in 2000. Step round one time step of a modern LSTM:

The heart of it is step 4, written out:

Compare that with the plain RNN, where the old state is pushed through W_hh and tanh at every step whether or not anything new has happened. The LSTM’s default, once it learns to use it, is to leave memory alone. It is the same insight as ResNet’s skip connection, discovered eighteen years earlier for depth in time: make “pass it through unchanged” easy, and gradients survive.

In 2014 Kyunghyun Cho and colleagues proposed a slimmer alternative, the gated recurrent unit (GRU). It merges the cell and hidden state into one, and uses two gates: an update gate that blends old state and new candidate (doing the forget and input gates’ jobs at once), and a reset gate that decides how much of the old state to use when forming the candidate.

An LSTM in PyTorch

PyTorch’s nn.LSTM runs the whole unrolled loop for you. The toy task: given the last 30 values of a sine wave, predict the next one. Each training example has its own random frequency and starting point, so the network cannot memorise one curve; it must work out from the 30 values how fast this wave is moving and where it is heading.

import torch
from torch import nn

torch.manual_seed(0)

def make_batch(n, steps=30):
    freq = torch.rand(n, 1) * 0.4 + 0.1        # every wave has its own speed...
    phase = torch.rand(n, 1) * 6.28            # ...and starting point
    t = torch.arange(steps + 1).float()
    wave = torch.sin(freq * t + phase)         # (n, steps + 1)
    return wave[:, :-1].unsqueeze(-1), wave[:, -1:]   # inputs (n, 30, 1), next value (n, 1)

class Forecaster(nn.Module):
    def __init__(self, hidden=32):
        super().__init__()
        self.rnn = nn.LSTM(input_size=1, hidden_size=hidden, batch_first=True)
        self.head = nn.Linear(hidden, 1)

    def forward(self, x):
        out, _ = self.rnn(x)                   # out: the hidden state at every step
        return self.head(out[:, -1])           # predict from the last step's state

model = Forecaster()
opt, loss_fn = torch.optim.Adam(model.parameters(), lr=0.01), nn.MSELoss()
X_test, y_test = make_batch(500)
print(f"baseline 'repeat the last value': test MSE {loss_fn(X_test[:, -1], y_test).item():.4f}")
for step in range(1, 401):
    X, y = make_batch(64)                      # fresh random waves every step
    opt.zero_grad()
    loss = loss_fn(model(X), y)
    loss.backward()
    opt.step()
    if step % 100 == 0:
        with torch.no_grad():
            print(f"step {step}: train MSE {loss.item():.4f}  test MSE {loss_fn(model(X_test), y_test).item():.4f}")

baseline 'repeat the last value': test MSE 0.0486
step 100: train MSE 0.0012  test MSE 0.0014
step 200: train MSE 0.0003  test MSE 0.0002
step 300: train MSE 0.0002  test MSE 0.0003
step 400: train MSE 0.0001  test MSE 0.0001

Notes on the code:

• Shapes. With batch_first=True, input is (batch, steps, features) — here 64 waves × 30 steps × 1 number. The LSTM returns out, the hidden state at every step (64, 30, 32), plus the final (h, c), which we ignore.
• Many-to-one. We use only the last step’s hidden state, out[:, -1], and a linear layer turns its 32 numbers into one prediction. For tagging every word in a sentence you would apply the head at every step instead.
• Fresh data every step. Because make_batch generates new random waves each time, the training loss is already an honest estimate of performance on unseen waves; the separate test set confirms it.
• A baseline first. “Tomorrow will be like today” — predicting the last value again — scores a mean squared error of 0.0486. After 400 steps the LSTM is at 0.0001, hundreds of times lower. Without the baseline, 0.0001 would be just a small number.

Swap nn.LSTM for nn.GRU and nothing else needs to change, because the forward method ignores the second return value; in our run the GRU also reached a test error of 0.0001 by step 400. The whole script runs in under ten seconds on a laptop CPU with PyTorch 2.14.1 As of October 2026; on other machines or versions the last digits may differ slightly.

An honest caution about toys. This task needs only a short memory. When we tried a harder one while writing this page — remembering a single number seen 50 steps earlier, through 49 steps of noise — the LSTM and GRU stayed at chance in every short run we tried (a few hundred training steps, default settings), while a plain RNN reached 96% in one configuration at 50 steps and then fell to chance at 100. Long-range memory tasks are notoriously sensitive to initialisation, learning rate and training length; LSTMs do learn them, but the textbook advantage is not something to expect from five minutes of untuned code.

From seq2seq to attention

Many problems map one sequence to another of a different length: translating a Malay sentence into English, turning speech into text, summarising. In 2014 two groups — Ilya Sutskever, Oriol Vinyals and Quoc Le at Google, and Kyunghyun Cho and colleagues in Montreal — showed how to do it with two RNNs. The design is called sequence-to-sequence (seq2seq), or encoder–decoder. Tap through how it works and where it led:

The attention idea is simple to state. At each decoding step, score every encoder state for relevance to what the decoder is about to write, turn the scores into weights that add to 1 with a softmax, and use the weighted average of encoder states as extra input. Bahdanau and colleagues called it “alignment”; the name that stuck was attention. Google’s neural translation system, launched in 2016, combined deep LSTMs with attention.

Attention fixed the bottleneck, but the RNN underneath still had a deeper limit: it processes step t only after step t − 1, so a 1,000-word document needs 1,000 sequential steps that no amount of hardware can run in parallel. The Transformer removed the recurrence and kept the attention. How that works is the subject of Transformers in Depth; the story of the 2017 paper is on Attention Is All You Need.

Where RNNs live on, and state-space models

Transformers replaced RNNs at the frontier of language, but recurrent models never disappeared. Their strengths are exactly the Transformer’s weaknesses: they need a fixed amount of memory and computation per step no matter how long the sequence has run, and they can process a stream one element at a time as it arrives. That makes them natural for on-device speech and wake-word detection, for sensor and time-series forecasting — predicting tomorrow’s electricity demand or a river’s level from recent readings — and for any setting where latency and memory are tight. In many forecasting problems, though, a well-tuned gradient-boosted tree on hand-built features remains a strong rival, so test against one.

Meanwhile, research revived the recurrent idea in a new form. State-space models such as S4 (Gu, Goel and Ré, 2022) and Mamba (Gu and Dao, 2023) keep a recurrent state like an RNN, but are structured so that training can be computed in parallel across the sequence like a Transformer, while generation stays a cheap step-by-step recurrence. Mamba makes its state updates depend on the input — a selective, gate-like mechanism that echoes the LSTM. Related designs include RWKV and xLSTM, the latter from Hochreiter’s own group. Several released models interleave state-space or linear-recurrent layers with attention layers, aiming to keep attention’s precise recall while cutting the cost of long contexts As of 2026. Speculative Whether such hybrids will displace pure Transformers in frontier models, or remain a niche for long-context and efficient deployment, is an open question; attention-based Transformers remain the dominant design for the largest language models at the time of writing.

Key terms on this page

Sources & further reading

• Sepp Hochreiter & Jürgen Schmidhuber, “Long Short-Term Memory”, Neural Computation 9(8) (1997); Felix A. Gers, Jürgen Schmidhuber & Fred Cummins, “Learning to Forget: Continual Prediction with LSTM”, Neural Computation 12(10) (2000).
• Jeffrey L. Elman, “Finding Structure in Time”, Cognitive Science 14 (1990); Paul J. Werbos, “Backpropagation Through Time: What It Does and How to Do It”, Proceedings of the IEEE 78(10) (1990).
• Yoshua Bengio, Patrice Simard & Paolo Frasconi, “Learning Long-Term Dependencies with Gradient Descent is Difficult”, IEEE Transactions on Neural Networks 5(2) (1994); Razvan Pascanu, Tomas Mikolov & Yoshua Bengio, “On the difficulty of training recurrent neural networks”, ICML (2013).
• Kyunghyun Cho et al., “Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation”, EMNLP (2014); Ilya Sutskever, Oriol Vinyals & Quoc V. Le, “Sequence to Sequence Learning with Neural Networks”, NeurIPS (2014).
• Dzmitry Bahdanau, Kyunghyun Cho & Yoshua Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate”, ICLR (2015).
• Junyoung Chung, Caglar Gulcehre, KyungHyun Cho & Yoshua Bengio, “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling” (2014); Klaus Greff et al., “LSTM: A Search Space Odyssey”, IEEE Transactions on Neural Networks and Learning Systems (2017).
• Albert Gu, Karan Goel & Christopher Ré, “Efficiently Modeling Long Sequences with Structured State Spaces”, ICLR (2022); Albert Gu & Tri Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces” (2023).
• PyTorch documentation: torch.nn.LSTM and torch.nn.GRU As of 2026; Christopher Olah, “Understanding LSTM Networks” (blog post, 2015).