sparse crosscoders

Resolve:

• cross-layer features: resolve cross-layer superposition

• circuit simplification: remove redundant features from analysis and enable jumping across training many uninteresting identity circuit connections

• model diffing: produce shared sets of features across models. This also introduce one model across training, and also completely independent models with different architectures.

cross-layer superposition

given the additive properties of transformers’ residual stream, adjacent layers in larger transformers can be thought as “almost parallel”

intuition

In basis of superposition hypothesis, a feature is a linear combinations of neurons at any given layers.

if we think of adjacent layers as being “almost parallel branches that potentially have superposition between them”, then we can apply dictionary learning jointly ¹

Autoencoders and transcoders as special cases of crosscoders.

• autoencoders: reads and predict the same layers
• transcoders: read from layer $n$ and predict layer $n+1$

Crosscoder read/write to many layers, subject to causality constraints.

crosscoders

Let one compute the vector of feature activation $f_(x_j)$ on data point $x_j$ by summing over contributions of activations of different layers $a^l(x_j)$ for layers $l \in L$:

$$\begin{aligned} f(x_j) &= \text{ReLU}(\sum_{l\in L}W_{\text{enc}}^l a^l(x_j) + b_{\text{enc}}) \\[8pt] &\because W^l_{\text{enc}} : \text{ encoder weights at layer } l \\[8pt] &\because a^l(x_j) : \text{ activation on datapoint } x_j \text{ at layer } l \\ \end{aligned}$$

We have loss

and regularization can be rewritten as:

weight of L1 regularization penalty by L1 norm of per-layer decoder weight norms $\sum\limits{l\in L} \|W^l_\text{dec,i}\|$ ³

We use L1 due to

• baseline loss comparison: L2 exhibits lower loss than sum of per-layer SAE losses, as they would effectively obtain a loss “bonus” by spreading features across layers

• layer-wise sparsity surfaces layer-specific features: based on empirical results of model diffing, that L1 uncovers a mix of shared and model-specific features, whereas L2 tends to uncover only shared features.

good to explore:

1. strictly causal crosscoders to capture MLP computation and treat computation performed by attention layers as linear
2. combine strictly causal crosscoders for MLP outputs without weakly causal crosscoders for attention outputs
3. interpretable attention replacement layers that could be used in combination with strictly causal crosscoders for a “replacement model”

see also: model stiching and SVCCA

(Laakso & Cottrell, 2000) proposes compare representations by transforming into representations of distances between data points. ⁴

How do features change over model training? When do they form?

As we make a model wider, do we get more features? or they are largely the same, packed less densely?