The Transformer Architecture: Self-Attention, Encoders, and Decoders
The transformer is a neural network architecture that relies solely on attention mechanisms, introduced in a research paper. It uses self-attention to weigh the relevance of every token in a sequence, and it is built from stacks of encoder and decoder layers. The architecture enables parallel processing and has become the foundation for large language models.
Introduction to the Transformer
The transformer architecture was proposed in the paper 'Attention Is All You Need' as a new sequence transduction model that relies solely on attention mechanisms, dispensing with recurrence and convolutions entirely. According to the arXiv abstract of Attention Is All You Need, the model achieved superior quality on machine translation tasks while being more parallelizable and requiring significantly less time to train.
The transformer consists of an encoder and a decoder. The encoder reads the input and generates a representation, while the decoder generates the output word by word while consulting the encoder representation. This general structure is typical of sequence-to-sequence models, and the transformer implements it without recurrence or convolution.
Self-Attention Mechanism
Self-attention is the central mechanism of the transformer. It determines, for each token, how much every other token should affect its interpretation. The Google Machine Learning Crash Course asks, 'How much does each other token of input affect the interpretation of this token?' This allows the model to capture dependencies between far apart words in a single step.
The transformer performs a small, constant number of steps, as explained in the Google Research blog post on the Transformer, and it directly models relationships between all words in a sentence regardless of their positions. This contrasts with RNNs, which must read each word sequentially.
Encoder and Decoder Structure
The encoder and decoder are each stacks of identical layers. As described in the Dive into Deep Learning chapter on the transformer, each encoder layer has a multi-head self-attention sublayer and a positionwise feed-forward network, with residual connections and layer normalization around each sublayer.
The decoder has an additional encoder-decoder attention sublayer. In the decoder self-attention, queries, keys, and values come from the previous decoder layer, but each position can only attend to positions up to itself, preserving the autoregressive property. The encoder-decoder attention uses queries from the decoder and keys and values from the encoder output.
Positional Encoding and Other Components
Because the transformer has no recurrence, it must encode token order explicitly. Positional encoding is added to the input embeddings before they enter the encoder and decoder. Multi-head attention, as noted in the IBM article on transformer models, allows the model to focus on different subspaces simultaneously.
Residual connections and layer normalization are used throughout the architecture to make training stable and to enable deep networks. These components were part of the original design and remain standard in modern transformer implementations.
Impact and Applications
The transformer's ability to process sequences in parallel has made it possible to train very large models on massive datasets. According to the AWS page on transformers in AI, transformers are used for machine translation, DNA sequence analysis, and multimodal tasks, and they underpin large language models with billions of parameters.
The architecture has also spurred further research and new variants, such as encoder-only models like BERT and decoder-only models like GPT. The self-attention mechanism is now a cornerstone of modern deep learning across many domains.
| Sublayer | Encoder | Decoder |
|---|---|---|
| Self-attention | Queries, keys, and values come from the previous layer | Queries, keys, and values come from the previous decoder layer, but only positions up to the current one are allowed |
| Multi-head attention | Used to compute self-attention | Used for both self-attention and encoder-decoder attention |
| Positionwise feed-forward | Applied after self-attention | Applied after self-attention and encoder-decoder attention |
| Residual connection | Around each of the two sublayers | Around each of the three sublayers |
| Layer normalization | After each residual connection | After each residual connection |
| Encoder-decoder attention | Not present | Queries from decoder self-attention, keys and values from encoder output |
Frequently asked questions
What is self-attention?
Self-attention is a mechanism that computes a relevance score between every pair of tokens in a sequence. It allows the model to weigh the importance of each token when updating the representation of another token, capturing context across the entire sequence.
How does the transformer handle word order without recurrence?
The transformer adds positional encoding to the input embeddings to encode the position of each token. This allows the self-attention mechanism to incorporate order information even though it processes all tokens in parallel.
What is the difference between encoder-only and decoder-only transformers?
Encoder-only transformers map input text to an intermediate representation, often used for classification or embedding tasks. Decoder-only transformers generate new tokens from previously generated text, typically used for language generation.
Why are transformers faster to train than RNNs?
Transformers process the entire sequence in parallel because self-attention computes interactions between all tokens simultaneously, whereas RNNs must process tokens sequentially. This parallelization allows transformers to take advantage of GPUs more effectively.