Paper review · November 19, 2026
Weight-Space Programming does not need Transformers
Weight-Space Programming Doesn't Need Transformers
When LLMs were asked to write weights directly without gradient descent, they chose a sparse fact-table structure on their own. Building on this, the authors proposed an associative memory model much smaller and faster than a Transformer.
Problem
Transformers became the standard because they train well with gradient descent and run fast on GPUs. But in weight-space programming (WSP), where an LLM writes the target model's weights directly, those reasons do not apply. The paper asks whether the Transformer architecture should still be kept for WSP.
Method
Three frontier LLMs (Claude Opus 5.5, GPT-6.1 Sol, GPT-6 Astra) were given only 10 ARC training examples and a sandbox, and asked to do two things: write weights for existing architectures (Transformer, SSM, RNN), and design an architecture from scratch. The LLMs converged on models that compile plain-text fact sentences into sparse tables and look them up with a fixed procedure. The authors interpreted this as associative memory. They then proposed a model that uses a concept table A and a concept-pair table addressed directly by hashing, reading only a fixed number of rows (120 pair rows) per question.
Results
- The model designed by Claude Opus 5.5 reached 50.8% accuracy on the full ARC test set (0.94M parameters, generation 1) and 54.4% (generation 5). A 2-layer Transformer written by the same LLM reached 36.1%
- The LLM-designed model was more accurate than the most accurate existing-architecture Transformer written by the same LLM, with about 1,800x fewer parameters
- Deleting a single row changed 0.5–1.8 answers on average in the table model. In the LLM-written Transformer it changed 4.8 answers (38% of answers in another run), showing that the table structure supports local edits well
- The proposed associative memory model stayed within about 3 percentage points of the LLM-designed model's accuracy while answering up to 17x faster on GPU than a Transformer of the same size
- A fine-tuned pretrained model (Qwen3.5-0.8B, 82.1%) is still far more accurate. WSP scored higher than a 2-layer Transformer trained with gradient descent without pretraining (29.8%)
Limitations
The main evaluation is ARC four-choice science questions, and the gap with the pretrained model is large (54.4% vs. 82.1%). The authors also limit the scope of their claim to "what architecture to build when writing weights directly." Results for generative tasks, math and vision extensions are not examined in detail within the provided text. LLM API costs grow with more generations (e.g., $490 for 8 generations of GPT-6 Astra).
Why it matters
The paper suggests that in an era when LLMs "program" other models, architectures suited to gradient descent may not be the best choice. Because the meaning of each weight can be traced and edited, it also has implications for interpretability and knowledge editing.
Key terms
- Weight-Space Programming(WSP)
- An approach in which an LLM writes the target model's weights directly, without gradient descent or training on data.
- Associative memory
- A structure that finds keys similar to a query and retrieves the values linked to those keys. This paper compares both Transformers and sparse tables within this framework.
- Local edit
- The property that changing one row of the weights changes only the predictions that depend on that row. It is needed for debugging a model without training.
Original
Authors: Hyunjong Ok, Jaeho Lee · Posted 2026-11-17 · License arXiv 비독점 배포 라이선스 · Based on the full text
This review is an AI-written summary of the paper. Please treat the original paper as the reference.