Subspace View

The object and role views both suffer from a coordinate problem: different methods recover different circuits, and rotating the basis moves “the mechanism” to different neurons without changing the computation. The subspace view resolves this by stepping up one level of abstraction — the mechanism is not a component or a role, but a low-dimensional subspace of the residual stream.

Subspaces are coordinate-free. They don’t depend on which basis you use, which means two methods that disagree about which heads matter can agree about which subspace the computation lives in. DAS (distributed alignment search) is the primary tool — it finds the subspace where swapping activations between two inputs makes the model behave as if one causal variable changed.

Thesis

A mechanism is a subspace $S \in \mathrm{Gr}(k, d)$ — the space of all $k$ -dimensional linear subspaces of $\mathbb{R}^d$ — such that intervening on the projection of the residual stream onto $S$ causes the model to behave as if the corresponding high-level variable had changed.

What it explains

Why methods disagree about components but may agree about subspaces. Different circuit-discovery methods project the same underlying subspace onto different component bases. The disagreement at the component level is expected and does not indicate that the methods found different mechanisms — it indicates that the mechanism is more naturally described as a subspace than as a set of components.

How a causal variable can be distributed across many neurons while remaining localized. A $k$ -dimensional subspace is low-dimensional (localized in representation space) even though it may project onto many neurons in any particular basis. Superposition is the natural state of a model packing many variables into a residual stream: each variable gets a low-dimensional subspace, and the subspaces overlap.

Why SAE channels and DAS subspaces can disagree. SAE channels are recovered by optimizing for sparse reconstruction of activations — the directions that best explain activation variance with sparse codes. DAS subspaces are recovered by optimizing for causal interventional consistency (IIA) — the directions that best predict downstream behavior under controlled swaps. A direction can explain activation variance without being causally active (a channel that is expressed but inert for the target task), and a direction can be causally active without being the sparsest activation description (a polysemantic direction).

What this view says

The mechanism is a point on the Grassmannian $\mathrm{Gr}(k, d)$ . DAS identifies a projector $QQ^\top$ that is invariant under $Q \mapsto QR$ for any $R \in O(k)$ (the group of rotations and reflections within the subspace): a subspace, not a specific matrix. Two mechanisms are the same when they are the same point on the Grassmannian — equivalently, when their principal angles (the canonical set of angles measuring the difference between two subspaces) are all zero.

The Grassmannian comes with a canonical distance (principal angles), a notion of averaging (Frechet mean — the generalization of an average to curved spaces), and baseline distributions from random matrix theory (Marchenko-Pastur — the distribution of singular values for random matrices, used as a null model). This gives the subspace view precise, falsifiable identity criteria: you can measure how close two claimed subspaces are, whether convergence across methods is statistically significant, and whether the subspace is distinguishable from random noise.

A nuance: DAS is more precisely a role-view method with a borrowed subspace parameterization. It validates by intervention success (role equivalence), not by Grassmannian distance. The subspace view’s native evidence is convergence of independently recovered subspaces — weight-space SVD and activation-space DAS landing at the same Grassmannian point — not IIA alone.

The subspace view assumes the mechanism is linearly encoded. When that assumption fails — nonlinear or context-switching representations, curved manifolds like the grokking circle — Riemannian geometry is needed instead of Grassmannian geometry. See the stratified view for how linear and nonlinear subspace structure relate.

When it works and when it doesn’t

The subspace view is strongest when the causal variable is linearly encoded in the residual stream. For many known circuits — IOI, gender bias, factual recall — DAS finds subspaces with high IIA (interchange intervention accuracy — the fraction of examples where swapping the subspace’s activations produces the expected output), confirming that the linear encoding assumption holds.

SAE confusion. Sparse autoencoder features are dictionary elements that minimize reconstruction loss; causal subspaces are directions that mediate specific causal paths. These are different objectives and can disagree. An SAE feature is a hypothesis about a one-dimensional subspace mechanism, not a confirmed subspace claim.

Unconstrained alignment vacuousness. Reporting high IIA without specifying the alignment class allows any model to be assigned any causal structure (Sutter et al., 2025). Restricting to linear alignment maps avoids this but is an assumption, not a theorem.

Linearity. The central assumption. When the causal variable is encoded on a curved manifold (as in grokking, where the mechanism lives on $S^1$ ), linear subspace methods will miss the structure. The correct metric comes from the manifold geometry, not from the Grassmannian.

Wrong $k$ . Too small a subspace dimension misses the variable; too large conflates multiple variables. Choosing $k$ requires either cross-validation or an independent estimate of the variable’s intrinsic dimension.

Prompt dependence. A subspace with high IIA on one distribution may not transfer across distributions. A “gender” subspace found on English biographical text may not be the same subspace in French or in dialogue.

Non-surgical interventions or wrong causal graph. Low IIA has two interpretations: (A) the subspace swap disturbs variables other than the claimed one (non-surgical intervention), or (B) the proposed causal graph is incorrect and changing this variable should affect the measured variable (wrong graph). These produce the same IIA signal but require different responses — a better subspace candidate vs. a revised causal graph.

Technical details

Identity criterion

Geodesic distance on $\mathrm{Gr}(k, d)$ :

$d(S_1, S_2) = \left(\sum_{i=1}^k \theta_i^2\right)^{1/2}$

where $\theta_1, \ldots, \theta_k$ are the principal angles between $S_1$ and $S_2$ , computed as $\cos\theta_i = \sigma_i(P_1 P_2)$ .

Evidence

Two native sources, each individually non-injective:

Activation-space interventional: DAS/IIA (with IIA as surgical-intervention test)
Weight-space structural: SVD of $W^{OV} = W^O W^V$ , invariant subspace decomposition
Cross-distribution stability: the geodesic distance on $\mathrm{Gr}(k,d)$ between the subspace recovered on the discovery distribution and the one recovered on a held-out distribution

Training-time subspace estimation (AGOP) bears on when the subspace formed, which is a process-view question rather than a third subspace-evidence domain.

Each domain is individually non-injective on mechanism space, but the joint map is conjectured to be injective under a general position condition.

Interventionism and the surgical intervention condition

The definition is interventionist in the sense of Woodward (2003): a causal variable is identified with the subspace $S$ such that a surgical intervention on $S$ changes the model’s output in the predicted way while leaving everything else unchanged.

IIA is not merely a measurement reliability condition — it is an empirical test bearing on the validity of the causal claim. Low IIA has two distinct interpretations:

(A) Non-surgical intervention. The subspace swap disturbs variables other than the claimed one — the intervention leaks into other parts of the causal graph.

(B) Wrong causal graph. The intervention may be perfectly surgical, but the causal graph assumed by the test is incorrect — the variable being swapped is causally upstream of what is being measured.

IIA should be reported as a diagnostic for intervention quality and graph validity, not just as a scalar accuracy metric.

SAE channels and the subspace view

SAEs trained on residual stream activations learn a dictionary $D \in \mathbb{R}^{d \times n}$ ( $n \gg d$ ) and sparse codes $a \in \mathbb{R}^n_{\geq 0}$ such that $x \approx Da$ . Each column $d_i$ is a candidate one-dimensional subspace — a point in $\mathrm{Gr}(1, d)$ .

Whether a given SAE channel corresponds to a genuine causal variable is a separate question from whether it is needed for reconstruction. SAE recovery should be treated as hypothesis generation: confirm with DAS/IIA that identified directions are causally active, not merely reconstruction-relevant. Divergence between SAE channels and DAS subspaces is informative — it indicates that sparse reconstruction and causal structure are picking out different things.

As dictionary size $n$ increases, some channels split into more specific ones (feature splitting). Whether the split channels are genuinely distinct mechanisms or coordinate choices within the same subspace is an identity question that requires causal testing to resolve.

What it lets you prove

Convergence: weight-space and activation-space subspaces converge on $\mathrm{Gr}(k, d)$
Non-vacuity (theoretical): restricting to transport-respecting alignments (G-SCM) is expected to rule out degenerate solutions (Sutter et al., 2025), though this has not yet been empirically demonstrated
Weight–activation convergence: an SVD of $W^{OV}$ and a DAS run on the same layer land at the same point of $\mathrm{Gr}(k,d)$ , which is what confirms a subspace by its own identity criterion rather than by intervention success
Triangulation necessity: each evidence domain alone is non-injective; convergence across domains is required for unambiguous identification

Formalism

Grassmannian geometry. A structural causal model whose nodes are subspaces and whose edges are weight-induced transport maps is one way to state a constrained alignment class: an alignment that does not respect transport structure violates the model’s causal independence assumptions.