← Key papers

Paper review · November 19, 2026

Weight-Space Programming does not need Transformers

Weight-Space Programming Doesn't Need Transformers

Language modelsClassic ML, recsys & search Intermediate Upvotes 47 cs.LG Pohang University of Science and Technology

When LLMs were asked to write weights directly without gradient descent, they chose a sparse fact-table structure on their own. Building on this, the authors proposed an associative memory model much smaller and faster than a Transformer.

Problem

Transformers became the standard because they train well with gradient descent and run fast on GPUs. But in weight-space programming (WSP), where an LLM writes the target model's weights directly, those reasons do not apply. The paper asks whether the Transformer architecture should still be kept for WSP.

Method

Three frontier LLMs (Claude Opus 5.5, GPT-6.1 Sol, GPT-6 Astra) were given only 10 ARC training examples and a sandbox, and asked to do two things: write weights for existing architectures (Transformer, SSM, RNN), and design an architecture from scratch. The LLMs converged on models that compile plain-text fact sentences into sparse tables and look them up with a fixed procedure. The authors interpreted this as associative memory. They then proposed a model that uses a concept table A and a concept-pair table addressed directly by hashing, reading only a fixed number of rows (120 pair rows) per question.

Results

Limitations

The main evaluation is ARC four-choice science questions, and the gap with the pretrained model is large (54.4% vs. 82.1%). The authors also limit the scope of their claim to "what architecture to build when writing weights directly." Results for generative tasks, math and vision extensions are not examined in detail within the provided text. LLM API costs grow with more generations (e.g., $490 for 8 generations of GPT-6 Astra).

Why it matters

The paper suggests that in an era when LLMs "program" other models, architectures suited to gradient descent may not be the best choice. Because the meaning of each weight can be traced and edited, it also has implications for interpretability and knowledge editing.

Key terms

Weight-Space Programming(WSP)
An approach in which an LLM writes the target model's weights directly, without gradient descent or training on data.
Associative memory
A structure that finds keys similar to a query and retrieves the values linked to those keys. This paper compares both Transformers and sparse tables within this framework.
Local edit
The property that changing one row of the weights changes only the predictions that depend on that row. It is needed for debugging a model without training.

Original

Authors: Hyunjong Ok, Jaeho Lee · Posted 2026-11-17 · License arXiv 비독점 배포 라이선스 · Based on the full text

This review is an AI-written summary of the paper. Please treat the original paper as the reference.