Dictionary

A method-level formalism used by sparse autoencoders (SAEs) and dictionary learning.

What it is

A dictionary is a set of vectors $\{d_1, \ldots, d_m\}$ in $\mathbb{R}^d$ (typically $m \gg d$, an overcomplete basis) such that any activation $x$ can be approximated as a sparse linear combination: $x \approx \sum_i c_i d_i$ where most coefficients $c_i$ are zero. Each dictionary element is a feature direction.

The key idea: the model’s activations live in a $d$-dimensional space but the “true” features are more numerous than $d$ — they form a sparse, overcomplete code. The dictionary recovers these features by finding directions that are used sparsely but frequently across the data.

See Sparse dictionary learning on Wikipedia.

How it is used in interpretability

Sparse autoencoders (SAEs) learn a dictionary of features from a model’s activations. Each feature is a direction in activation space, and the sparsity constraint encourages each activation to be explained by a small number of features. The hope is that these features correspond to interpretable concepts.

The limitation: the sparsity objective optimizes for reconstruction quality, not causal relevance. A feature that helps reconstruct activations may not be causally active in the model’s computation. Feature splitting (one concept spread across multiple dictionary elements) and feature absorption (multiple concepts merged into one element) complicate the mapping between dictionary features and functional units. Causal validation — e.g., steering with a feature direction and observing the predicted behavioral change — is needed to establish that a dictionary feature is a mechanism, not just a statistical pattern.

Relationship to other formalisms

Dictionary features are points in $\mathrm{Gr}(1, d)$ — the simplest objects on the Grassmannian. The subspace view generalizes from single directions to $k$-dimensional subspaces. The stratified view places dictionary features at the $\mathcal{M}_1$ stratum and asks when they should be grouped into higher-dimensional mechanisms.

Further reading

• Cunningham et al., “Sparse Autoencoders Find Highly Interpretable Directions in Language Models” (2023) — SAE features as interpretable directions
• Bricken et al., “Towards Monosemanticity” (2023) — scaling SAEs to large models
• See the SAE channels discussion on the subspace view page for when SAE channels and causal subspaces agree or diverge
• See the methods page for how SAE features fit into the broader evidence landscape