Recurrent Neural Networks: Modelling Sequences

Text, speech, sensor streams, stock prices, DNA — much of the world's data comes as sequences where order matters and inputs have variable length. A feedforward network takes a fixed-size input and has no memory. Recurrent neural networks (RNNs) process one element at a time while maintaining a hidden state that summarises everything seen so far. Although transformers now dominate many sequence tasks, RNN concepts — hidden state, unrolling, BPTT, gating — are foundational and are experiencing a revival in efficient modern architectures.

The vanilla RNN#

At each time step $t$, with input $\mathbf{x}_t$ and previous hidden state $\mathbf{h}_{t-1}$:

The same weights are used at every time step — weight sharing across time, analogous to convolution's sharing across space. This lets the RNN handle sequences of any length with a fixed number of parameters.

Unrolling#

Conceptually, we "unroll" the loop into a deep feedforward network with one layer per time step, all layers sharing weights:

x1 → [RNN] → h1 → [RNN] → h2 → [RNN] → h3 → ...
             ↓             ↓             ↓
             y1            y2            y3

An RNN processing a sequence of length 100 is a 100-layer-deep network — which is why the vanishing-gradient problem hits RNNs especially hard.

Sequence task patterns#

Backpropagation through time (BPTT)#

Training applies backpropagation to the unrolled graph. The total loss is the sum over time steps, and because weights are shared, the gradient for $\mathbf{W}_{hh}$ sums contributions from every step:

where $\partial^+$ denotes the immediate (direct) partial derivative. The crucial term is the product of Jacobians

If the largest singular value of these Jacobians is below 1, contributions from distant steps vanish exponentially; if above 1, they explode. Consequently vanilla RNNs struggle to learn dependencies more than roughly 10–20 steps apart — for instance, agreeing a verb with a subject far back in a long sentence.

Truncated BPTT limits backpropagation to a window of (say) 100 steps for efficiency, carrying the hidden state forward but not gradients beyond the window.

A character-level language model#

import torch
import torch.nn as nn

text = "machine learning is the study of algorithms that improve through experience. " * 50
chars = sorted(set(text)); stoi = {c: i for i, c in enumerate(chars)}
data = torch.tensor([stoi[c] for c in text])
V, H, T = len(chars), 128, 64

class CharRNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.emb = nn.Embedding(V, 32)
        self.rnn = nn.RNN(32, H, batch_first=True)          # vanilla tanh RNN
        self.out = nn.Linear(H, V)
    def forward(self, x, h=None):
        o, h = self.rnn(self.emb(x), h)
        return self.out(o), h

model = CharRNN(); opt = torch.optim.Adam(model.parameters(), lr=3e-3)
for step in range(600):
    i = torch.randint(0, len(data) - T - 1, (32,))
    x = torch.stack([data[j:j + T] for j in i]); y = torch.stack([data[j + 1:j + T + 1] for j in i])
    logits, _ = model(x)
    loss = nn.functional.cross_entropy(logits.reshape(-1, V), y.reshape(-1))
    opt.zero_grad(); loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 1.0)       # essential for RNNs
    opt.step()
print("loss:", round(loss.item(), 3))

# Sample text
x, h, out = torch.tensor([[stoi["m"]]]), None, "m"
for _ in range(80):
    logits, h = model(x, h)
    x = torch.multinomial(logits[0, -1].softmax(-1), 1)[None]
    out += chars[x.item()]
print(out)

This model predicts the next character — the same objective, in miniature, as modern large language models.

Bidirectional and deep RNNs#

• Bidirectional RNNs run one RNN forwards and another backwards and concatenate their states, so each position sees both past and future context. Useful for tagging and classification (not for generation, where the future is unknown).
• Stacked (deep) RNNs feed the hidden-state sequence of one RNN layer into another.

Strengths and limitations#

Strengths: constant memory per step at inference; natural streaming; handles arbitrary lengths.

Limitations: sequential computation cannot be parallelised across time during training (slow on GPUs); vanishing gradients limit memory; a fixed-size hidden state is an information bottleneck.

The fixes came in stages: gating (LSTM, GRU) for memory, attention for the bottleneck, and transformers for parallelism. Recently, state-space models and linear recurrent architectures (e.g. Mamba, RWKV) revisit recurrence with parallelisable training, aiming to combine transformer-like quality with RNN-like inference efficiency.