Structural View

You can permute the heads of a transformer, rotate the residual stream basis, and rescale the weights — and the model computes exactly the same function. These are gauge symmetries: transformations that change the parameterization without changing the computation. The structural view says: the mechanism is whatever is invariant under these transformations. Everything else is artifact.

Thesis

A mechanism is an equivalence class of weight configurations under computation-preserving symmetries: two configurations related by such a symmetry — a rescaling, a permutation of neurons, or a rotation of a head’s query–key subspace paired with an inverse rotation of its output — are the same mechanism. This is finer than behavioral equivalence: two configurations can produce identical outputs without lying in the same gauge orbit.

What it explains

Why different methods can find different circuits that are all “correct.” If ACDC and EAP recover different component sets for the same task, the structural view asks: are these component sets related by a gauge transformation? If so, they are different descriptions of the same mechanism, not different mechanisms. Circuit disagreement is expected when the identity criterion is the gauge orbit, not the specific circuit.

Cross-architecture identity. Two models implement the same mechanism if their gauge orbits are isomorphic — no need to match specific heads. Head 9.9 in GPT-2 and whatever implements the same function in Llama can be identified as “the same mechanism” without requiring them to be the same component.

Why the parameterization is not the mechanism. A neuron may look special because of how the weights happen to be arranged, not because it is doing something computationally special. The structural view makes this distinction precise: anything that changes under a symmetry transformation is a property of the parameterization; anything that doesn’t is a property of the mechanism.

What this view says

A gauge orbit is the set of all weight configurations related to each other by function-preserving transformations. Think of it this way: if you take a trained transformer and permute its heads, rotate its residual stream basis, and rescale its weights in any way that preserves the input-output mapping, every configuration you can reach by doing this is in the same gauge orbit as the original. All these configurations compute exactly the same function — they differ only in how that function happens to be written down in the weights. The gauge orbit IS the mechanism: it captures everything about the computation while stripping away the arbitrary choices of how to parameterize it.

Two mechanisms are the same when they lie in the same gauge orbit. This resolves the alternative-exclusion problem that limits the object view: if two component sets are gauge-equivalent, they are not alternative explanations — they are the same explanation expressed in different coordinates.

The structural view also uses holonomy — what happens when you transport a subspace through the network along a closed loop of weight matrices. If the subspace comes back rotated, that rotation (the holonomy) is a gauge-invariant fingerprint characterizing the mechanism. Two mechanisms with different holonomy groups are provably different, regardless of how their components are arranged.

The view’s objects also include cosheaf cohomology classes — topological invariants that track whether a mechanism can be decomposed into locally consistent pieces. When the cohomology is nontrivial ( $H^1 \neq 0$ ), the mechanism has genuinely distributed structure that no local circuit description captures. (Not yet computed for any transformer at scale.)

It is among the most ontologically committed positions in the atlas, ranking above the subspace view and below the stratified view. It addresses the identity, convergence, and invariance problems that the lower views face, though at the cost of computability. Holonomy, cosheaf cohomology, and gauge-orbit comparison are hard to compute at scale. The view also requires specifying the right symmetry group — LayerNorm breaks the full rotation symmetry, and using the wrong gauge group produces wrong identity judgments. In practice, the structural view is more of a theoretical ceiling than a practical toolkit: it tells you what the right answer looks like, even when you can’t compute it yet.

When it works and when it doesn’t

The structural view is strongest when you need cross-architecture comparison, when you need to resolve apparent circuit disagreements, or when you need to prove that two descriptions are actually the same mechanism.

Computability. The main limitation. Computing holonomy requires specifying a connection; computing cosheaf cohomology requires specifying base sections and a linearization. Both are expensive at scale.

Wrong symmetry group. LayerNorm breaks the full $O(d)$ rotation symmetry. The gauge group for a real transformer is more restricted than $O(d)$ , and using the wrong group produces incorrect identity judgments. The correct gauge group for a given architecture must be determined empirically.

Over-abstraction. Too coarse an invariant conflates distinct mechanisms. Cohomology class is coarser than gauge orbit — different gauge orbits can have isomorphic cohomology. Using the wrong level of invariant loses real distinctions.

No dynamics. The structural view characterizes what the mechanism IS at a given checkpoint, not how it formed or why. For questions about training dynamics, phase transitions, or developmental prerequisites, the process view is needed.

Technical details

Identity criterion

Two mechanisms are the same when they lie in the same gauge orbit. Isomorphic holonomy groups are a necessary but not sufficient condition. Same cohomological equivalence class is a coarser criterion than same gauge orbit.

The qua-problem

Why gauge orbits rather than component overlap, behavioral equivalence, or cohomology class? The argument is functional: gauge transformations preserve the input-output function exactly. Any property distinguishing two configurations in the same orbit is a property of the parameterization, not of the computation.

Behavioral equivalence is coarser: two mechanisms can compute differently but have the same output distribution. Cohomology class is also coarser: different gauge orbits can have isomorphic cohomology. The gauge orbit is finer than both and therefore the right choice when the claim is about computational identity.

Evidence

Gauge-invariant measurements: singular values of OV circuits $W^{OV}$ , principal angles, effective rank — invariant under both head permutations and (approximate) orthogonal rotations. Composition scores $\|W^{QK}_v W^{OV}_u\|_F / (\|W^{QK}_v\|_F \|W^{OV}_u\|_F)$ measure how much head $v$ reads from head $u$ independently of basis choice. The composite form is what carries the invariance: the score is defined on $W_{QK}$ and $W_{OV}$ because keys, queries and values are reparameterizable by-products of the low-rank factors, and the normalization is what makes the quantity scale-invariant
Holonomy: estimates of the holonomy group (connection must be specified)
Cohomological tests: $H^0$ and $H^1$ of the circuit cosheaf

What it lets you prove

Gauge-invariant identity: same gauge orbit implies same mechanism
Holonomy as necessary condition: same mechanism implies isomorphic holonomy groups
Cohomological localizability: relative to the cosheaf construction and base sections

Formalism

Fiber bundle $\mathcal{W}/\mathcal{G}$ , principal fiber bundles, parallel transport, holonomy.