Long short-term memory (LSTM): gates that let networks remember longer
LSTM networks address the vanishing gradient problem by using gated memory cells. Input, forget, and output gates control how information is written, kept, or read. LSTM is used for sequence tasks such as translation and time series forecasting.
The vanishing gradient problem in recurrent networks
Recurrent neural networks (RNNs) process sequences by carrying a hidden state from one step to the next. IBM's LSTM overview defines LSTM as a special type of recurrent neural network designed to learn and remember information over long sequences of data.
Training plain RNNs on long sequences is difficult because the gradients used for learning can become too small or too large. The paper On the difficulty of training recurrent neural networks by Pascanu, Mikolov, and Bengio analyzes these vanishing and exploding gradient problems and proposes gradient norm clipping as a remedy.
LSTM was introduced by Hochreiter and Schmidhuber to address these issues. Instead of an ordinary recurrent node, LSTM uses a memory cell with gates that decide what to store and what to forget.
Gated memory cells in LSTM
The Dive into Deep Learning chapter explains that each LSTM memory cell has an internal state with a self-connected edge of fixed weight 1, ensuring that gradients can pass across many time steps without vanishing. The cell also has three gates: an input gate, a forget gate, and an output gate, computed with sigmoid activations, while the candidate value uses tanh.
The input gate controls how much of the candidate value is added to the cell state, the forget gate determines whether to keep the current memory or flush it, and the output gate decides whether the cell state should influence the output at that time step.
The PyTorch documentation provides the exact formulas for the LSTM layer: the cell state is updated as c_t = f_t * c_{t-1} + i_t * g_t, and the hidden state is h_t = o_t * tanh(c_t). The parameters include hidden_size, num_layers, and a bidirectional flag.
Applications and observed performance
The NVIDIA LSTM page lists applications such as language modeling, machine translation, acoustic modeling, speech synthesis, and handwriting recognition. The same page notes that GPUs can accelerate LSTM training by 6x and inference by 140x compared to CPU implementations.
A sequence-to-sequence learning paper from Google used a multilayered LSTM for English to French translation. On the WMT'14 test set, the LSTM model achieved a BLEU score of 34.8, outperforming a phrase-based system that scored 33.3.
LSTM networks also appear in time series forecasting and speech recognition, as described in general overviews of the architecture.
| Gate | Role | Activation |
|---|---|---|
| Input gate | Controls how much of the candidate value is added to the cell state | Sigmoid |
| Forget gate | Determines whether to keep or flush the current memory | Sigmoid |
| Output gate | Decides whether the cell state influences the hidden output | Sigmoid |
| Cell candidate (g_t) | Provides the new value to be potentially written to the cell state | Tanh |
Frequently asked questions
What is an LSTM?
An LSTM is a type of recurrent neural network that uses gated memory cells to learn long-term dependencies in sequential data.
What are the three main gates?
The input gate, forget gate, and output gate control how information flows into and out of the memory cell.
Why do LSTM networks avoid vanishing gradients?
The memory cell has a self-connected edge with a constant weight of 1, which allows gradients to flow across many time steps without decaying.
What are typical uses of LSTM?
LSTM is used for language modeling, machine translation, speech recognition, and time series forecasting.