AI glossary · Advanced · Language models
RoPE
RoPE (Rotary Position Embedding) is a method that rotates vectors by a set angle according to position, so that a transformer can tell where each token sits in a sentence. Many LLMs today use it as their default way of handling position information.
About this term
- Level
- Advanced · Specialist terms that need the inner workings
- Category
- Infrastructure & using models
- Fields
- Language models
- Added
- 2026-10-12
How it works
On its own, a transformer's attention does not know the order of words. RoPE adds order information by rotating the query and key vectors used in attention by an angle proportional to their position. When two tokens are compared, the difference in their rotation angles remains, so the model naturally reads 'how far apart they are' rather than their absolute positions. Different parts of the vector rotate at different speeds, so both short and long distances can be represented.
Where you'll see it
You will find it in the config.json of a Hugging Face model repository, in fields such as rope_theta (the base value for rotation speed), rope_scaling (the context extension method) and max_position_embeddings (the maximum position seen in training). When increasing context length in an inference engine's settings, you may also set the rope_scaling option. If a model card says something like 'context extended with YaRN,' it means the RoPE settings were adjusted so the model can handle longer inputs. In other words, RoPE settings are directly tied to the context window a model can handle.
Things to know
If you feed in an input much longer than the lengths seen during training, the model encounters rotation angles it has never seen, and quality can drop sharply. So extending the context requires following the model's recommended rope_scaling settings or doing additional training. If you change values in the config file arbitrarily, the model may still run but produce garbage answers. In research on generating long text or long video, how to handle positions outside the training range remains an important problem.
Example
When an engineer wants to feed a long contract into a model in one go for internal document summarization, they check max_position_embeddings and rope_scaling in the model's config.json and put the extension settings recommended by the model card into the inference engine as is. In long-video generation research too, such as the 'In-Distribution Forcing' paper covered on this site, how to handle positions beyond the training range in the KV cache is treated as a core problem.