Do SAEs Recover the “True” Features?

The question. Sparse autoencoders decompose neural network activations into interpretable directions. But the features change with SAE width, training seed, and architecture. Which features are “real”? Are any of them the “true” decomposition of what the model is doing?

This question is one of the most actively debated in mechanistic interpretability. We show that it is a view-level question: the answer depends entirely on what you mean by “feature.”

What each view says

Object view — features are parts

Features are concrete, countable components of the model’s computation. There is a fact of the matter about which features exist. An SAE at the “right” width recovers them; an SAE at the wrong width either splits genuine features or merges distinct ones.

The problem. Different SAE widths give different feature sets. Bricken et al. (2023) showed that increasing width progressively splits features into finer-grained variants. If there are true features, which width finds them? The Object view has no principled answer — it requires a ground truth that no one has access to.

Subspace view — the subspace is what’s real

Individual features are basis vectors. What’s real is the subspace they span — a point on the Grassmannian $\mathrm{Gr}(k, d)$. Different SAE widths give different bases for overlapping subspaces. This is expected, not problematic.

The test. Measure Grassmannian distance between SAE feature subspaces across widths and seeds. If the subspaces converge (small $d_{\mathrm{Gr}}$), the SAEs are finding real structure. If they don’t, the decomposition is method-dependent. Which individual features you name is a gauge choice — like choosing a coordinate system for a vector space. The subspace is gauge-invariant; the basis is not.

Perspectival view — features are method projections

SAE features are not discoveries in the model — they are projections of the model through a particular method (sparse dictionary learning with a specific loss function, width, and training procedure). A different method projects different features. Neither is wrong; neither is “true.”

The implication. Asking whether SAE features are “real” is like asking whether a Mercator projection is the “true” shape of a continent. The projection is mathematically valid and practically useful, but the shape is not “in” the territory. Heap et al. (2025) show that SAEs applied to randomly initialized transformers still produce features that score as interpretable — exactly what the Perspectival view predicts.

The resolution

“Do SAEs recover the true features?” is not an empirical question with a single answer. It is three different questions depending on your view:

View	Question	Answer
Object	Are these the real parts?	Underdetermined — no principled way to pick the “right” width
Subspace	Does the subspace converge?	Testable — measure $d_{\mathrm{Gr}}$ across widths and seeds
Perspectival	Are these more than method artifacts?	Requires discriminant validity — trained vs. control comparison

The field argues because researchers at different views mean different things by “true.” The Object-view researcher wants to know which features are real. The Subspace-view researcher wants to know if the subspace is stable. The Perspectival-view researcher wants to know if the method is imposing structure.

The practical fix: stop asking “are these the true features?” and start asking:

1. Subspace convergence — do SAEs at different widths span similar subspaces? (Grassmannian distance)
2. Discriminant validity — do SAEs find qualitatively different structure in trained vs. random models? (V3 Discriminant)
3. Causal relevance — do the features (or subspaces) have causal effects on model behavior? (I2 Sufficiency, I1 Necessity)

These are answerable. “Are these the true features?” is not.

Mechanistic validity criteria

Not in the paper

The following content is not in the paper. The paper’s open-problems appendix records each problem’s status and key views and carries no validity criteria; this table reads the same problem against the companion framework’s criteria.

Criterion	Status	What it tells you
C3 Convergent validity	Central	Do different SAE configurations agree?
M6 Invariance	Central	Are features stable across measurement choices (width, seed)?
I5 Rival mechanism exclusion	Critical	Can you distinguish trained-model features from noise-model features?
I3 Minimality	Relevant	Where does one feature end and another begin?

See also

• Decomposition Identity — the broader open problem this question belongs to
• Superposition — the related question of whether superposition is a problem or a property
• Subspace view — the view that resolves the convergence question
• Perspectival view — the view that predicts method-dependence
• “MI Needs Philosophy” §2.2 — vehicle/content distinction maps onto Object/Role for features (full mapping)

Feature absorption

A concrete instance of this problem: Chanin et al. (2024) showed that safety-relevant SAE features can be absorbed into more general features at higher SAE widths, a phenomenon distinct from the feature splitting Bricken et al. (2023) reported. A “deception” feature at width 4096 may disappear at width 16384 — not because deception stopped being represented, but because the representation was redistributed across finer-grained features.

Under the Object view, this is alarming: a feature you were monitoring vanished. Under the Subspace view, it’s expected: the subspace is stable even as the basis changes. The safety question becomes: is the subspace still detectable, even if the individual feature isn’t? If yes, monitor the subspace, not the feature. If no, the representation genuinely changed — a much more serious concern that requires E2 Prompt generalization evidence.

Resolution status: Clarified

The framework clarifies the question. “True features” is an Object-view concept applied to a domain where the Subspace view is more appropriate. The answerable questions are: do subspaces converge? (Grassmannian distance), are they causally relevant? (DAS/IIA), are they more than artifacts? (discriminant validity). These replace the unanswerable question “which features are true?”