Long context: Positional encoding extension and 'Lost in the Middle'

Why long context became necessary

The maximum length of input an LLM (large language model) can read at once is called the context window. At first, a few thousand tokens were enough. But as it became more common to feed in entire long contracts or sets of papers, or to run agents that call tools over many hours, windows started to fall short. A study on agents performing long-horizon tasks paper points out that reasoning traces, actions and tool observations accumulate until they eventually exceed a fixed window.

There are also more explanations than one might expect for why seeing further back helps. A time-series forecasting study paper questions the common explanation that long observation windows help only because they 'capture long-range dependencies.' According to this study, a long window reduces uncertainty about which generating process produced the incoming data. It also proves that even for a process with memory length P, the window size must be strictly larger than P to reach the minimum error. This means long context is used not only to reference the distant past directly but also to recognize the situation itself.

The problem is that the window cannot simply be extended. Positions not seen in training make the model unstable, memory grows in proportion to length, and important information in longer inputs is missed more often. This note covers these three issues in turn.

Key idea: Desk size and reading ability are different

Long context can be compared to a desk. The context window is the size of the desk, and the model is the person looking at the materials spread out on it. To widen the desk, the method of marking 'which position each item was placed in' must first work on a wider desk as well. This is the problem of positional encoding extension. Next, there must be enough space and budget, that is, memory, to handle the wider desk.

A wider desk does not mean the person carefully reads every sheet on it. Materials at the very front and back get noticed, but those buried in the middle are easily overlooked. The industry commonly calls this tendency 'Lost in the Middle.' The source materials for this note do not include the study that first measured this phenomenon. So rather than citing specific figures, this note looks at recent studies that measure the same problem from different angles.

In short, long-context research has three branches: methods for handling position, methods for handling memory and speed, and methods for getting the model to actually use the information provided and for measuring that ability properly. Solving only one of the three does not make a 'model that uses long context well.'

How it works: Positional encoding and length extension

Attention in a transformer compares all tokens at once. So unless it is told otherwise, it does not know the order of tokens. RoPE (rotary position embedding), widely used today, conveys order by rotating each token's query and key vectors by an angle proportional to its position. When two tokens are compared, the difference in rotation angles, that is, their relative distance, is reflected in the result. As a general principle, when large position indices not seen in training come in, the model encounters unfamiliar rotation angles and its performance can become unstable.

So one major direction for extending length is to remap position indices back into the range seen in training. A study on long-context speculative decoding paper is a good example. Speculative decoding is an acceleration technique in which a cheap draft model (drafter) writes a few tokens ahead and the large model verifies them all at once. According to this study, when the context becomes very long, the draft model must also attend to the entire preceding context, so its cost grows and the speedup shrinks and eventually reverses. The common solution is to train a new draft model for long context. Instead, this study keeps a short-context draft model frozen and adds only two inference-time mechanisms. One limits the range of the attention cache the draft model sees, keeping its cost fixed; the other remaps (re-bases) rotary position indices into the training range.

This works because the draft model reads the large model's hidden representation, that is, the internal state the large model computed after seeing the entire preceding context. The distant context is already contained in that state, and whatever the draft model misses is corrected in the verification step. The researchers proved the conditions under which the draft model's computation stays exactly the same when position indices are shifted uniformly. With a single vLLM setting and no changes to the weights, it could be used from 4k up to 2 million tokens, and they report that the speedup lost at short lengths was at most 0.16x.

As length grows, memory also becomes a problem. The KV cache, which stores the keys and values of previous tokens during generation, grows in proportion to input length. TideKV paper focuses not on generation-stage memory, which has already been studied extensively, but on peak memory during the prefill stage, when the prompt is first read.

'Lost in the Middle': Long does not mean fully read

Extending the window and handling the memory does not end the problem. Whether the model actually finds and uses information located somewhere in a long input remains. 'Lost in the Middle' is one visible form of that failure. Recent studies are broadening the problem from 'where the information is' to 'how scattered it is.'

The LURCH benchmark paper breaks the difficulty of long-context tasks into three properties: the amount of evidence to find (evidence scope), how widely the evidence is spread across the input (evidence dispersion), and how many reasoning steps must be chained to reach the answer (reasoning depth). In an evaluation on 6,550 questions, all three properties showed a strong negative correlation with performance, even at the same length. As length grew, performance dropped sharply on questions with scattered evidence but stayed relatively stable on questions with concentrated evidence. It is also notable that questions where all three properties were moderately high were harder than those where only one was extremely high.

These results place 'Lost in the Middle' in a broader context. The core of the problem is not just the middle position. It is that models are weak at gathering and connecting pieces scattered throughout a long input. Passing a test that finds a single needle in one place (Needle-in-a-Haystack) does not guarantee the ability to connect scattered evidence.

How it developed: From evaluation to memory mechanisms

The foundational studies fall into two broad branches. The first is evaluation research that asks, 'Are we really measuring long-context ability?' 100-LongBench paper pointed out two weaknesses in existing real-task benchmarks such as LongBench: they lack a metric that separates long-context performance from a model's baseline ability, and their fixed input lengths cannot show at what length a model breaks down. The study proposed a benchmark with adjustable length and a metric that separates a model's prior knowledge from its true long-context ability. Ref-Long paper created a 'referencing' task that, instead of simple retrieval, asks for the indices of documents that reference a specific key. Testing 13 models, it found that even models such as GPT-4o showed clear weaknesses on this task.

Some studies have also examined the effect of extending the window in real work. In a literature review generation study paper, two researchers evaluated 20 AI-written reviews against 15 criteria. Longer windows captured broader information and maintained coherence over long inputs, but problems such as repeated content, missing key studies and listing rather than synthesizing actually got worse. The study concluded that expert review is still needed.

The second branch is methods that let models remember without being bound by window size. ARC paper stores an agent's tool observations in an append-only, ID-addressed record. When compression is needed, older observations are replaced with short citations (IDs), and the agent can retrieve the original text by that ID when needed. There is no need to rerun tools or rely solely on similarity search. In experiments with Qwen3-8B at a 16k window and Qwen3-32B at a 32k window, average Needle-in-a-Haystack accuracy was 99.40%, higher than the 88.12% of the best comparison method. On LongBench-v2 Hard, it was 29.97% versus 28.25%. A residual vector reconstruction study paper proposed a method that, without additional training, stores facts from the source text as residual vectors in feed-forward layer activations and then reconstructs the facts relevant to a question. It reports that GPU memory stayed nearly constant as context grew, and that it could answer single-fact questions over a 2-million-token story.

Where things are heading now

Three trends stand out in recent research. First, difficulty is no longer measured by length alone. As with LURCH paper discussed earlier, evaluation is shifting toward measuring the amount of evidence, how scattered it is and the depth of reasoning separately, because only then does it become clear why a model gets things wrong.

Second, the question of 'RAG or long context' is examined at the level of evidence. The grounded-generation diagnostic study paper had models produce line-level citations along with their answers and scored both answer correctness and evidence together. After testing 10 frontier commercial models and 3 open-weight models, it found that RAG (retrieval-augmented generation) has two limitations that long-context models avoid. Similarity search was good at finding 'one' relevant chunk, but much weaker at finding 'all the chunks needed' for questions that require multiple facts. Chunking (splitting documents) also erased document metadata needed to disambiguate identities. Long context was not perfect either, however. In the two-needle setting, citations were sometimes missing, and when the context conflicted with the model's existing knowledge, the gap between 'only the answer is correct' and 'both the answer and the evidence are correct' widened.

Third, the memory bottleneck is being addressed through system design. TideKV paper uses 'deferred eviction': at each layer, it processes all prompt chunks and then evicts KV entries when that layer finishes. This lets it choose which KV entries to keep after seeing the full prompt while still reducing peak memory. On an A100 at 64K length, it cut peak memory during cache construction by 83.89%. On an RTX 4090, it enabled 128K prefill while increasing TTFT (time to first token) by only 1.06%.

Limitations and open issues

The biggest issue is that the 'window size' number can easily overstate actual capability. As 100-LongBench paper pointed out, even with high scores it is hard to tell whether they reflect long-context ability or the model's existing knowledge. Ref-Long paper and LURCH paper show that models are still weak on tasks that go beyond simple retrieval. Even if a model advertises a long window, you should check it yourself if your work involves 'connecting scattered evidence.'

Increasing length can also worsen other aspects of quality. In the literature review study paper, longer windows increased repetition and omission of key points. Techniques like position remapping also have a limited scope. The long-context draft model study paper worked well because of its structure: the draft model reads the large model's internal states, and the verification step corrects errors. This result should not be stretched to mean that any model can extend its length simply through position remapping.

Memory-based approaches also come with conditions. ARC paper itself states that it outperforms the baselines it evaluated 'within the settings it tested.' The residual-vector method paper also reported results on single-fact questions. How it would perform on questions that require connecting multiple facts cannot be determined from these sources alone.

Further reading

To establish an evaluation perspective, start with 100-LongBench paper and understand how it isolates and measures 'long-context ability.' Reading LURCH paper next lets you revisit 'Lost in the Middle'-type failures through the framework of evidence dispersion and reasoning depth. Ref-Long paper is supplementary material that shows the difference between retrieval and referencing.

If you want to see how positional encoding extension is used in practice, the long-context draft model study paper is a good choice. It covers in concrete terms the idea of mapping RoPE position indices back into the training range and the conditions under which this does not change the computation. For serving costs, read TideKV paper next.

If you are designing agents or RAG, we recommend reading ARC paper and the grounded-generation diagnostic study paper together. They give different answers to the context engineering question of what to keep inside the window and what to keep outside and retrieve when needed. If you want a theoretical view of why longer windows help, the time series study paper is short and clear.