Paper review · October 21, 2026
MemT-1.0: A memory-native evolution that builds memory into the model
MemT-1.0: Towards Memory-Native Evolution
The authors trained a foundation model with token-level memory updates based on Gated DeltaNet-2 and Mixture-of-Storage, achieving 1.86x faster inference and 66x less storage than a full-context KV cache on a 128K history.
Problem
Today's LLMs must either re-insert the full context or attach an external retrieval system to remember past conversations or experiences. As context grows longer, the KV cache grows and inference slows down. The authors argue that, as happened with reasoning and multimodality, memory should be internalized as part of the model's computation rather than handled by an external system. Earlier work, Metis, was a first attempt, but token-selection-based memory updates and limited memory capacity remained open problems.
Method
A memory attention block that runs in parallel with the original attention is attached to each selected Transformer block. Memory is split into dynamic local memory, updated every turn, and hyper memory, obtained through training. Metis's token selection is replaced with token-level updates based on GDN-2 (Gated DeltaNet-2), and a stabilization term that applies a square root to the erase gate is added to keep the memory state from diverging. To increase capacity, the authors introduce Mixture-of-Storage (MoS); unlike ordinary sparse MoE, it uses dense routing in which all memory experts participate, and the storage and retrieval routers share the same address matrix while learning different projections. After a backbone preparation stage, training proceeds in two stages: continued pretraining (CPT) on 34B tokens and post-training. Two sizes were released: Nano at 36B (3B active, 10M dynamic memory) and Flash at 288B (17B active, 336M dynamic memory).
Results
- The authors reported that it outperformed parametric memory baselines on average across 11 memory benchmarks and was competitive with retrieval-augmented approaches. It used past information without accessing the original context at inference time.
- At a history length of 128K tokens, MemT-1.0-Nano ran end-to-end inference about 1.86x faster than a full-context KV cache and reduced persistent storage by 66x.
- The authors reported that general capabilities were largely preserved in the noise setting.
- By internalizing retrieved procedural experience into native memory, it improved average success rate and consistency across repeated runs on SWE-bench Verified and SWE-bench Multilingual compared with an agent without experience, reaching a level similar to explicit text-based experience.
- In the backbone preparation experiments, tasks never solved in 8 attempts reached only 1.6–4.0% on 4 benchmarks even after post-training, whereas on a separate set of hard knowledge questions, providing the relevant facts as context raised accuracy from 27% to 91%.
Limitations
As the authors noted, memory capacity is itself a new scaling axis and is still at an early stage of exploration. The results are 'competitive' with the retrieval-augmented approaches compared, not better than them, and general capabilities are described as 'largely' preserved, implying some loss. The 34B tokens of training data are small relative to typical foundation model training, and because inference is stateful, dedicated infrastructure is needed to isolate memory per trajectory. The provided text is cut off partway through the backbone preparation section, so detailed per-benchmark figures could not be checked.
Why it matters
Keeping a fixed-size memory state instead of re-inserting long context every time means storage does not grow as interactions accumulate. This approach could change the cost structure of long-running agents. The result of raising success rates for coding agents by putting past procedural experience into memory shows that memory can go beyond simple recall and act as multi-step execution guidance. The code and models are released, enabling verification and follow-up research.
Key terms
- Memory-Native
- A design direction that builds memory into the model's own computation rather than relying on an external database or retrieval system.
- Gated DeltaNet-2(GDN-2)
- A linear-attention-family architecture that updates its state by writing and erasing memories through gates. It maintains a fixed-size state instead of a KV cache that grows in proportion to context length.
- Mixture-of-Storage(MoS)
- An approach that expands the memory block into multiple experts but uses dense routing in which all experts participate. The storage and retrieval routers share the same addresses.
- KV cache
- A buffer that stores the keys and values of tokens a Transformer has already processed to avoid recomputation. As context grows, it takes up a lot of memory and becomes a bottleneck in inference cost.
Original
Authors: Zhiyu Li, Zeyu Zhang, Linyun Liu, Yiwei Bu, Chenyang Xi, Xichong Zhang, Yi Yu, Yu Yu, et al. · Posted 2026-10-19 · License CC BY-NC-SA 4.0 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.