The attention mechanism: queries, keys and values in neural networks
Attention mechanisms allow neural networks to weigh the importance of different input parts. Queries, keys, and values are used to compute attention weights and produce outputs. This technique underpins modern transformer models.
What is an attention mechanism?
An attention mechanism is a machine learning technique that directs deep learning models to focus on the most relevant parts of input data, as described in IBM's article on attention mechanisms. This selective focus helps models process information efficiently and accurately.
The technique was introduced by Bahdanau and colleagues to improve machine translation with recurrent neural networks. Their neural machine translation paper proposed a soft-search mechanism that allowed the model to align relevant parts of a source sentence when predicting each target word.
An attention model computes attention weights that reflect the relative importance of each input part and applies those weights to increase or decrease the influence of each part accordingly.
Queries, keys, and values
The attention mechanism uses concepts from database retrieval. The Dive into Deep Learning's chapter explains that a database consists of keys and values, and a query is used to match keys. In deep learning, a query is compared against each key to compute a relevance score.
The scores are used as weights to compute a weighted sum of the values, which is called attention pooling. This operation generates a linear combination of values, where the weights determine how much each value contributes.
Special cases include uniform averaging over all values and the traditional database lookup where only one weight is nonzero. These illustrate the flexibility of the attention mechanism.
Computing attention weights
To ensure that attention weights are nonnegative and sum to one, a softmax operation is applied to the scoring function. This is a common strategy in deep learning frameworks and keeps the attention mechanism differentiable.
Attention can also be used to focus on external memory locations, as shown in Distill's article on attention and augmented RNNs. In neural Turing machines, an attention distribution describes how much the model reads from or writes to each memory position.
This distribution is a weighted sum, making the read and write operations differentiable and trainable.
Multi-head attention and the Transformer
Multi-head attention extends the mechanism by allowing the model to jointly attend to information from different representation subspaces. The PyTorch MultiheadAttention documentation describes how the module splits the embedding dimension across multiple heads.
The Transformer architecture, introduced in the Attention Is All You Need paper, uses only attention layers and feedforward layers, without recurrence or convolutions. This design achieved BLEU scores of 28.4 for English-to-German and 41.8 for English-to-French translation.
The parallel nature of attention allows efficient use of GPUs and better capture of long-range dependencies compared to sequential models.
Applications and impact
Attention mechanisms have become essential in natural language processing tasks such as summarization, question answering, and text generation. They are also applied in computer vision for image captioning and visual question answering.
The ability to focus on relevant parts of input without fixed input size makes attention versatile for many domains. Its success in transformers has driven much of the recent progress in generative AI.
| Term | Definition | Role in attention |
|---|---|---|
| Query | The vector that seeks information from the set | Compared against keys to compute attention weights |
| Key | An identifier for each value | Matched with query to determine relevance |
| Value | The actual content associated with a key | Weighted by attention and summed to produce output |
| Attention weight | A scalar measuring the relevance of a key to a query | Determines how much influence each value has |
| Attention pooling | The weighted sum of values using attention weights | Produces the attention output |
Frequently asked questions
How does attention differ from recurrent networks?
Attention can process all input parts simultaneously and does not rely on sequential hidden states, which helps handle long-range dependencies.
What are queries, keys, and values in attention?
They are vectors where the query matches against keys to produce attention weights, and the values are weighted and summed to give the output.
Why is softmax used in attention weights?
Softmax normalizes the scores to be nonnegative and sum to one, which is a common practice for a differentiable attention mechanism.
What is multi-head attention?
It runs multiple attention computations in parallel, each on a different representation subspace, and concatenates the results.