Recurrent neural networks: how networks process sequences step by step
Recurrent neural networks (RNNs) process sequential data by maintaining a hidden state that carries information across time steps. They update the hidden state at each step using the current input and the previous hidden state, and they are trained with backpropagation through time. Variants such as LSTM and GRU address training challenges like vanishing gradients.
What is a recurrent neural network
A recurrent neural network (RNN) is a deep learning model trained on sequential or time series data to make predictions based on prior elements in the sequence, as explained in IBM's article. It is used for tasks such as language translation, speech recognition, sentiment analysis, and time series forecasting, which the same source lists.
RNNs differ from feedforward networks because they retain information from earlier inputs, allowing the current output to depend on the entire sequence seen so far. This makes them effective for problems where order and context matter, such as understanding an idiom or predicting the next word in a sentence.
How recurrent networks work
An RNN processes a sequence one element at a time. At each step, it computes a hidden state that combines the current input with the hidden state from the previous step, as described in the Dive into Deep Learning RNN chapter. This hidden state acts as a memory that stores information about the sequence up to that point.
The recurrence can be written as h_t = φ(x_t W_xh + h_{t-1} W_hh + b_h), where φ is an activation function such as tanh or ReLU. In modern frameworks, this computation is implemented efficiently; for example, the PyTorch RNN module allows specifying the number of layers, the activation function, and whether the network is bidirectional, among other options.
Training with backpropagation through time
Training an RNN uses backpropagation through time (BPTT), which unrolls the network across the sequence and applies the chain rule to compute gradients, as explained in the Dive into Deep Learning BPTT chapter. The recursive multiplication of gradients during this process can cause them to shrink or grow exponentially, leading to vanishing or exploding gradients that make training unstable for long sequences. Gradient clipping is a common countermeasure to keep gradient magnitudes under control during optimization.
BPTT also requires storing intermediate states for each time step, which increases memory consumption and can be computationally expensive for very long sequences.
Variants and applications
RNN architectures can be arranged in different input-output configurations. For example, a one-to-many model generates multiple outputs from a single input, a many-to-many model maps sequences to sequences, and a many-to-one model aggregates a sequence into a single output, as described in the AWS RNN overview. These configurations adapt RNNs to tasks like image captioning, machine translation, and sentiment analysis respectively.
Several variants of the basic RNN have been developed to improve training and performance. Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) introduce gating mechanisms that control the flow of information, helping to mitigate the vanishing gradient problem, as noted in the NVIDIA RNN guide. Bidirectional RNNs process the sequence in both directions to capture context from past and future elements, which is also mentioned in the same source.
| Configuration | Description | Example use |
|---|---|---|
| One-to-many | One input produces multiple outputs. | Image captioning |
| Many-to-many | Multiple inputs produce multiple outputs. | Machine translation |
| Many-to-one | Multiple inputs produce a single output. | Sentiment analysis |
Frequently asked questions
What is the difference between a hidden layer and a hidden state?
A hidden layer is a set of neurons that transforms inputs within the network, while a hidden state is a vector that stores information from previous time steps and is passed to the next step in the sequence.
Why do RNNs have vanishing and exploding gradients?
During backpropagation through time, gradients are multiplied repeatedly across many time steps. If the multiplication factors are smaller than one, gradients shrink exponentially; if larger, they grow exponentially, making training unstable.
What are LSTM and GRU?
LSTM and GRU are RNN variants that use gating mechanisms to control the flow of information. They help mitigate the vanishing gradient problem by allowing the network to remember information over longer periods.
What input-output architectures can an RNN have?
An RNN can be configured as one-to-many, many-to-many, or many-to-one, depending on the task. For example, one-to-many is used for image captioning, many-to-many for translation, and many-to-one for sentiment analysis.