Mechanistic interpretability: Looking inside models with sparse autoencoders

Why it was needed: Models that became piles of numbers

When a neural network receives an input, each layer produces a vector with thousands of dimensions, called an activation. The predictions are accurate, but it is hard to know what each number in that vector means. Within interpretability research, mechanistic interpretability is the line of work that tries to reveal, at the level of components, 'what internal computation led the model to its answer'.

The problem is that one neuron does not correspond to one concept. A study of SAEs for symmetric data paper explains this as a phenomenon called superposition. A model has many concepts to handle and fewer dimensions than that, so it packs multiple concepts into a small number of dimensions. As a result, when you look at a single neuron, it often responds to several unrelated concepts at once.

This opacity has real costs. A study of EEG (brain wave) foundation models paper points out that even with high clinical performance, a model whose internal computation cannot be seen has difficulty earning trust in medical settings. A study on code correctness paper argues that, with a substantial amount of AI-suggested code going into production services, we need to know how a model internally judges 'this code is correct' in order to deploy it safely.

Core idea: Splitting a chord into its instruments

Picture a single orchestra recording. The sound picked up by the microphone is a single waveform, but within it violins, cellos and trumpets are mixed together. And at any given moment, only a few instruments in the full ensemble are actually playing. Using these two properties, 'they are mixed' and 'only a few sound at the same time', you can split the waveform back into the sound of each instrument.

A sparse autoencoder (SAE) works on the same idea. Borrowing the formulation of the Expander SAE paper paper, an SAE represents an activation h as the product of a learned dictionary W and a sparse code x (h ≈ Wx). The dictionary is an overcomplete dictionary with more entries than the original number of dimensions, and the code x is mostly zero, with values in only a few positions. Each dictionary entry (a feature, or atom) corresponds to one instrument, and the code value to that instrument's volume.

Why deliberately increase the number of dimensions? Because of superposition, there are more concepts than original dimensions. But increasing dimensions creates too many ways to explain the same activation, so a sparsity constraint, 'only a few may turn on', narrows down the answer. The same paper explains that this inference problem closely resembles the basic setup of compressed sensing (recovering a sparse original signal from a small number of measurements). It is a design that gains human-readable features and accepts reconstruction error and computational cost in return.

How it works: From collection to intervention

First, freeze the model you want to interpret and collect a large number of activations from a specific layer. A study of medical-domain SAEs paper used activations from layer 20 of a Gemma-2 model, and PhysSAE paper used activations from the second-to-last layer of a physics-informed neural network (PINN). Which layer and which data you choose shapes all subsequent results.

Second, an encoder converts activations into sparse codes, and a decoder reconstructs them using the dictionary. Training balances two goals together: 'small reconstruction error' and 'few active features'. Variants differ in how they enforce sparsity. TopK SAEs, which keep only the top k, were used by the EEG study paper and Expander SAE paper; JumpReLU SAEs, which cut off values below a threshold, were used by the medical-domain study paper; and the subspace SAE study paper analyzed the basic objective function with ℓ1 regularization.

Third, give each feature a name describing what it means. Collect the inputs on which that feature activates strongly and have people examine them, or have a language model generate descriptions automatically. The medical-domain study paper used both automated and human evaluation, and the ESMFold study paper used a reasoning language model to interpret a large dictionary.

Fourth, verify whether the names are correct by intervening in the original model. Try removing a feature (ablation) or amplifying it to push the model's output in a desired direction (steering), and see whether the results change as expected. PhysSAE paper went a step further and intervened by subtracting only the component along the feature's direction from the frozen original hidden state, without passing through the SAE decoder. This design is meant to keep SAE reconstruction error from contaminating the intervention results.

How it developed: What each starting-point study fixed

The starting-point studies for this article all appeared in 2025–2026. So rather than a lineage descending from 'the first SAE', it is more accurate to read them as efforts to fix weaknesses of the same tool from different directions. They fall broadly into three strands: data and efficiency, structural assumptions, and applications and validation.

On data and efficiency, the medical-domain SAE study paper found that SAEs trained on broad data spend their fixed feature budget on common but useless patterns. Narrowing training to 195,000 clinical question-answer examples explained up to 20% more variance, reduced the unexplained linear component in the reconstruction error ('dark matter'), and yielded features that matched clinical concepts such as 'infectious mononucleosis'. Expander SAE paper replaced the decoder, which becomes more expensive as the dictionary grows, with a sparse connectivity structure. On Qwen2.5-3B, it cut the number of trainable decoder values by 293x while retaining 84% of the CE loss recovery of a dense decoder.

On structural assumptions, researchers are revising the basic assumption of 'one feature = one direction'. Subspace SAE (SASA, paper) proved that when a real feature has multiple dimensions, a single-direction decoder splits that one feature into several nearly parallel features, and the number of features needed grows exponentially with the feature's dimensionality. It then proposed replacing the decoder with learned subspaces. Equivariant SAE for symmetric data paper argued that symmetries such as rotation worsen the non-identifiability of SAEs (the problem of multiple explanations fitting equally well), and extended the linear representation hypothesis underlying SAE theory to cover symmetries. The SAE neural operator (SAE-NO, paper) represents concepts as functions rather than scalar values, capturing not only whether a concept is active but where and how it appears in the input. It also works at higher resolution than in training, generalizing under conditions where standard SAEs fail.

On applications and validation, researchers extend SAEs beyond LLMs and examine 'whether the features found are real'. The code correctness study paper found a direction in an LLM that predicts incorrect code, and because this direction stays effective after instruction tuning, the authors interpreted it as a mechanism learned during pretraining being reused in fine-tuning. The EEG study paper applied the same procedure to three EEG transformers with different architectures and evaluated features against clinical categories such as abnormal findings, age, sex and medication. PhysSAE paper showed that across six PDE families, SAE features matched independently defined physical quantities with up to |r|=0.951, and that the effect of removing a feature was 1.2–4.2x more spatially concentrated than interventions using PCA or ICA.

Where it is heading: Diagnostic tools and scientific models

Recent research is shifting SAEs from 'building feature catalogs' toward 'diagnostic tools that answer other questions'. A study dissecting Distillation paper compared GPT-2 (Teacher) with DistilGPT-2 (Student), reduced to 6 layers, using CKA, logit lens, attention pattern decomposition, SAE feature statistics and activation patching. The indirect object identification (IOI) circuit collapsed completely in the Student, with accuracy dropping from 94.8% to 35.2%. By contrast, the size comparison (Greater-Than) circuit recovered 87.7% of task performance through a relocation in which the Teacher's layer-9 comparison heads moved to layer 4 in the Student (mean Pearson r = 0.945). The authors conclude that overall metrics alone cannot tell whether circuits are intact.

Another direction is looking inside scientific models. The ESMFold study paper applied SAEs to the folding trunk of a model that predicts 3D protein structure from amino acid sequences. It showed that representations that carry over sequence information and representations that compute the geometry between residue pairs divide the work between them. Using gradient-based attribution, it also traced how concepts are combined across blocks and identified where that computation agrees with and departs from known biochemistry. Taken together with PhysSAE, SAE-NO and the EEG study, SAEs are becoming a general-purpose microscope for models in physics, medicine and biology, beyond language models.

Limitations and open issues

The most fundamental issue is that there is no single 'correct dictionary'. The Equivariant SAE study paper points out that different explanations can fit the data equally well, and reconstruction quality alone cannot tell which one is faithful to the model. On data with symmetries, features can be more useful even when reconstruction is worse, so it warns against using reconstruction quality as the key measure of interpretability.

The second is feature splitting and absorption. The subspace SAE study paper proved that the basic objective function actually prefers splitting multidimensional features into multiple features. The medical-domain study paper also reported features fragmenting or absorbing one another in SAEs trained on broad data. A large number of features does not mean many concepts were found.

The third is the side effects of intervention. The EEG study paper divided steering outcomes into three cases: 'selectively steerable', 'encoded but entangled' and 'not encoded'. It also reported 'wrecking ball' interventions that collapse overall model performance, and entanglement, such as between age and pathology, in which trying to suppress only one damages the other. In the code correctness study paper as well, steering to fix errors came at the cost of breaking correct code. The authors therefore suggest intervening selectively only when an error is expected, rather than intervening all the time.

Criteria for using it in practice

SAEs are a good fit when you want to check whether a model holds a particular concept internally, or when you want to use that concept as a warning signal. The code correctness study paper proposed using the direction that predicts incorrect code as an error alert for developer review. On the other hand, applying always-on steering as a way to permanently fix model behavior calls for caution, given the side effects reported in the studies above.

There are four common mistakes. First, believing an interpretation is correct just because reconstruction error or loss recovery looks good. Second, trying to interpret a specific domain with a single general-purpose SAE built on broad data. The medical domain study paper reported that narrowing the domain works better. Third, stopping at naming features without running intervention experiments. Fourth, looking only at the target concept when measuring intervention effects, without checking whether other concepts or overall performance collapse along with it.

It also helps to decide on the validation method in advance. As in PhysSAE paper, compare against simple baselines such as randomly chosen features or PCA, and check whether results repeat across multiple seeds. As in the EEG study paper, measure effects on both target and non-target concepts. When inspecting a model after compression or Distillation, check at the circuit level rather than by overall scores, as the Distillation dissection study paper shows.

Further study

If you are new to this, start with the medical domain SAE study paper. It covers, in one paper, the typical procedure for applying an SAE to LLM activations, the limits of broad data, and both automated and human evaluation. Next, read the code correctness study paper to see how the directions found can actually be used for prediction and steering, and at what cost.

If you are interested in theory, read the Equivariant SAE for symmetric data paper together with the subspace SAE paper. The former addresses 'reconstruction quality ≠ interpretation quality', and the latter 'feature splitting is caused by the objective function'. Both show where the basic assumptions of SAEs break down. If you want to know the cost of large-scale use, Expander SAE paper lays out the trade-off between storage and fidelity.

To learn validation methodology, PhysSAE paper and the EEG study paper are good choices. They give concrete methods for intervening directly on the original hidden states, setting up random control groups, and measuring unintended effects as well. Finally, to catch up on recent trends, look at how SAEs are used alongside other analysis tools in the Distillation dissection study paper and the ESMFold study paper.