Positional encoding: how transformers keep track of word order

Updated October 21, 2026

Transformers process tokens in parallel and have no built-in sense of order, so they need positional encoding to inject word position information. The original Transformer used fixed sinusoidal functions, while later methods like relative position representations, rotary position embedding, and ALiBi offer different trade-offs. This article explains the core idea and compares these approaches.

Why transformers need positional encoding

Recurrent neural networks and long short-term memory models process tokens one at a time, so they naturally learn word order. Transformers, in contrast, process all tokens in parallel using self-attention, which means they have no inherent representation of position. Without extra information, a transformer would treat the words in "Allen walks dog" and "dog walks Allen" identically, even though the meanings differ, as explained in IBM's positional encoding overview.

Positional encoding solves this by adding a vector that encodes the position index to the word embedding. The resulting vector represents both what the word is and where it appears in the sequence. This allows the transformer to retain word order while still benefiting from parallel processing and efficient training, as described in the same IBM article.

The original Transformer paper introduced a fixed, deterministic encoding based on sine and cosine functions. The authors of Attention Is All You Need proposed using sinusoidal functions with different frequencies across dimensions, which produces a smooth, periodic pattern that can represent positions in a differentiable space.

Sinusoidal positional encoding in the original Transformer

The sinusoidal approach assigns each position a vector where each dimension follows a sine or cosine wave with a different frequency. Because the functions are periodic and continuous, small changes in position produce small changes in the encoding, which is useful for learning. The IBM article notes that varying the frequency across dimensions creates a rich, multiscale representation of position.

This fixed encoding does not require learned parameters, which simplifies training. The original paper reported that the Transformer achieved strong results on machine translation tasks, but the paper does not provide the exact formulas for the sinusoidal encoding in its abstract; the details are in the full text.

Sinusoidal encodings have a limitation: they are designed for sequences of a fixed maximum length. When a model trained on short sequences is asked to handle longer ones, the encoding may not generalize well. This motivated research into alternative position methods.

Relative position representations

Instead of adding absolute position information to the input, the paper Self-Attention with Relative Position Representations extends the self-attention mechanism to consider the distances between sequence elements. This approach models relative positions, which can be more natural for capturing relationships like adjacency or long-range dependencies.

The authors reported improvements over absolute position representations on the WMT English-to-German and English-to-French translation tasks, with gains of 1.3 BLEU and 0.3 BLEU respectively. They also observed that combining relative and absolute representations did not yield further improvement.

This method can be seen as a form of relation-aware self-attention that generalizes to graph-labeled inputs, making it more flexible than fixed absolute encodings.

Rotary position embedding and ALiBi

Rotary Position Embedding (RoPE), introduced in RoFormer: Enhanced Transformer with Rotary Position Embedding, encodes absolute position using a rotation matrix while also incorporating relative position dependency in the self-attention formulation. RoPE offers properties such as flexibility in sequence length and decaying inter-token dependency with increasing relative distance.

Another approach, Attention with Linear Biases (ALiBi), does not add positional embeddings to word embeddings at all. Instead, it biases the query-key attention scores with a penalty proportional to the distance between tokens, as described in Train Short, Test Long: Attention with Linear Biases. This method allows a model trained on shorter sequences to extrapolate to longer ones, achieving the same perplexity as a sinusoidal model trained on the longer length, while training faster and using less memory.

ALiBi's inductive bias toward recency also helps it outperform several strong position methods on the WikiText-103 benchmark, according to the same paper.

How to choose a positional encoding method

The choice of positional encoding depends on the application. For standard transformer models with fixed input lengths, sinusoidal or learned absolute encodings are simple and effective. For tasks that require extrapolation to longer sequences, relative or bias-based methods like RoPE and ALiBi may be better suited.

When evaluating a model, one can check the documentation or source code to see which positional encoding is used. Many implementations, such as those in Hugging Face, include the method in the model configuration. The RoFormer paper notes that RoPE is integrated into Hugging Face, making it easy to experiment with.

The Dive into Deep Learning textbook provides a practical comparison of CNNs, RNNs, and self-attention, noting that self-attention has constant sequential operations and a maximum path length of O(1), which makes it easier to learn long-range dependencies. This comparison is useful for understanding the architectural context in which positional encoding operates, as explained in Dive into Deep Learning.

Frequently asked questions

Why can't transformers just use the order of tokens in the input?

Transformers process all tokens in parallel, so they have no built-in sense of order. Without positional encoding, the model would treat tokens as if they were in a bag, ignoring their positions.

What is the difference between absolute and relative positional encoding?

Absolute positional encoding adds a vector that represents the token's index, while relative positional encoding models the distance between tokens, often by modifying attention scores. Relative methods can generalize better to longer sequences.

Does positional encoding require training?

Some methods, like sinusoidal encoding, are fixed and do not require training. Others, like learned absolute embeddings, are trainable. Methods like RoPE and ALiBi are also fixed but use different mechanisms.

Can a transformer work without positional encoding?

Technically yes, but it would lose all information about word order, which is essential for understanding language. The model would not know the difference between "dog bites man" and "man bites dog".

How does one know which positional encoding a model uses?

Check the model's documentation or source code. Many libraries, such as Hugging Face, expose the positional encoding type in the model configuration. Research papers also describe the method they use.