Learning Self-Interpretation from Interpretability Artifacts: Training Lightweight Adapters on Vector-Label Pairs

Accepted to ICML 2026

A frozen LLM can describe its own hidden computations, including thoughts it never verbalizes, after training a single bias vector on existing interpretability data. The adapter outperforms the labels it was trained on.

Paper (arXiv) → Read the full paper Code → GitHub repository Data → Wikipedia topic descriptions dataset

Authors: Keenan Pepper¹, Alex McKenzie¹, Florin Pop¹, Stijn Servaes¹, Martin Leitgab¹, Mike Vaiana¹, Judd Rosenblatt¹, Michael S. A. Graziano², Diogo de Lucena¹

Published: 2026

Funded by the AI Alignment Foundation

Key Ideas

Modern interpretability tools can identify directions in a neural network's hidden states that correspond to specific concepts: a "Golden Gate Bridge" direction, a "sarcasm" direction, and so on. Each comes with a vector and a natural-language label. There are different ways to find these directions:

• Contrastive activation vectors: Prompt the model with "Tell me about {{topic}}" for many topics, then subtract the mean activation. What's left isolates the direction for that specific topic.
• SAE features: Sparse autoencoders learn to decompose a model's activations into interpretable building blocks, individual features that fire for recognizable concepts.

Both methods produce large datasets of (vector, label) pairs. What if we could teach the model to write those labels itself? That's the idea behind trained SelfIE. We freeze the LLM entirely and train a tiny adapter (as few as d + 1 parameters) that projects an activation vector back into the model's own embedding space. The model then autoregressively describes what the vector represents, interpreting its own internals.

• Because the LLM stays frozen, the "interpreter" and the "subject" are the same model. The adapter just gives it a better lens.
• On contrastive vectors, the trained adapter reaches 93.7% recall@1 vs. 1.3% untrained.
• On SAE features, generation scoring jumps from 49.1% → 71.4%.
• A dead-simple scalar affine adapter (f(h) = α·h + b) transfers better than fancier alternatives, likely because it has almost no room to memorize. In fact, a learned bias vector alone accounts for ~85% of the improvement.

How it works

We follow the Patchscopes framing: an activation is extracted, transformed by a mapping function f, then injected into a fixed explanation-seeking prompt so the (frozen) model generates a natural-language interpretation. The key change is learning f from artifact data instead of hand-tuning scales.

1. Collect artifacts: vector-label pairs from SAE features and contrastive topic vectors.
2. Train adapter only: learn f (e.g., scalar affine) while keeping the LLM frozen.
3. Autoregressive interpretation: inject f(h) and let the model generate the label text.

Adapter architecture comparison

Scalar affine adapters (d+1 parameters) can outperform more expressive alternatives on behavioral evaluations despite higher validation loss, and a learned bias vector accounts for most of the improvement.

The key takeaway is architectural constraint: simpler adapters transfer better. In particular, the scalar affine form f(h)=α·h+b (d+1 parameters) is a strong default. Much of the lift comes from the learned bias term.

A helpful way to think about it: scalar affine treats all directions the same, so it can’t “memorize” special-case mappings for particular vectors. That makes it harder to overfit and helps it transfer across datasets. In contrast, full-rank adapters can overfit badly on SAE feature datasets.

Empirically, the learned bias term accounts for ~85% of the improvement over untrained self-interpretation.

Adapter architecture comparison (paper Table 1; most notable architectures)

SA = scalar affine. LR = low-rank. Lower validation loss is better. Llama Scope SAEs (Llama-3.1-8B, d=4096).

Appendix G (v1.5): training on ALL-CAPS labels shows the bias vector mostly carries format/style (“prior”), while the input activation supplies the specific semantic content.

These examples use the same SAE latent as our scale-sensitivity evaluation; here we train the adapter to output labels in ALL-CAPS.

Example outputs (Normal vs ALL-CAPS):

→Normal: Event listener and callback function definitions in code →ALL-CAPS: EVENT HANDLING AND CALLBACKS IN PROGRAMMING

Contrastive activation vector results

We train an adapter on Wikipedia contrastive activation vectors and evaluate whether SelfIE retrieves the correct topic label on held-out validation topics.

Recall@1 jumps from ~1% (untrained baseline) to ~94% with the trained adapter (best-of-6 across input scales).

{"title":"Recall@1 (best-of-6)","values":[{"method":"Untrained baseline","value":0.013},{"method":"Trained adapter","value":0.937}]}

Recall@1 (best-of-6)

Δ +92.4 pp

0%25%50%75%100%Untrained baselineTrained adapter

Recall@1: Use the description as a query in an external topic-search system; count how often the correct topic is the #1 result.

Examples (trained vs baseline)

Vector token\nExplore more topics: choose from the examples below. This contrastive activation vector is taken at a newline just before the model begins responding to:

Explore topics

Tell me about Chuuk Lagoon.

Interactive: click to explore

Regular SelfIE (baseline)

Inference-only

"a small bay or inlet of the sea, typically one that is partially enclosed by islands, headlands, or other landforms."

Trained adapter (ours)

Trained adapter

"Chuuk Lagoon, the site of the largest naval battle in history"

Held-out topics: The topics shown above come from the validation split, so the adapter was never trained on them (this tests whether it generalizes).

Dataset split: 49,637 total topics (44,673 train, 4,964 validation shown).

Same prompts, same injection site. Baseline is inference-only; ours trains the adapter.

Scaling with model size

We evaluate Qwen models of increasing size using Recall@100. Trained SelfIE improves with model size, while the Taboo baseline saturates after the smallest model.

We run this scaling experiment on four Qwen models (7B, 14B, 32B, 72B). For trained SelfIE (lower trace), we decode activations from a "Tell me about{{topic}}." prompt into a description, then use semantic search over ~50k topics and report Recall@100 (the correct topic appears in the top 100 results).

To control for general topic knowledge, we include a "Taboo" baseline (upper trace), where the model describes the topic under a no-name constraint, for example: "Describe bananas without using the word ‘banana’, any part of it, or obvious synonyms." We score it with the same Recall@100 pipeline. The Taboo curve flattens quickly, while trained SelfIE keeps improving with model size, suggesting SelfIE scales by better extracting existing knowledge, not just benefiting from bigger models.

[{"model":"7B","paramsB":7,"taboo":0.9336,"tabooErr":0.006,"trained":0.7439,"trainedErr":0.012},{"model":"14B","paramsB":14,"taboo":0.9755,"tabooErr":0.002,"trained":0.7718,"trainedErr":0.012},{"model":"32B","paramsB":32,"taboo":0.9759,"tabooErr":0.002,"trained":0.8137,"trainedErr":0.011},{"model":"72B","paramsB":72,"taboo":0.9774,"tabooErr":0.002,"trained":0.8478,"trainedErr":0.01}]

"Taboo" protocol

Trained SelfIE

Recall@100 = fraction of topics where the correct label appears in the top 100 semantic-search results.

Untrained SelfIE (Recall@100 < 0.02 across all sizes) is off-scale and not shown.

SAE evaluation: scale sensitivity

Different input scales can fall into “ignored”, “just right”, or “fluent but wrong” regimes; training reduces scale sensitivity and improves stability across scales.

Scale sensitivity: Self-interpretation is sensitive to the scale of the injected vector: each vector has a narrow valid range.

Failure mode: outside that valid range, outputs can stay fluent while becoming ungrounded in the injected vector (“fluent but wrong”). Training the adapter reduces sensitivity, but SAE features still benefit from trying multiple scales (best-of-6). Generation scoring is a practical way to verify whether a label actually matches model behavior.

{"totalLatents":3243,"totalScales":6,"buckets":[0,1,2,3,4,5,6],"series":[{"name":"Untrained","values":[1661,436,264,212,185,195,290]},{"name":"Trained","values":[1018,257,207,200,266,391,904]}],"note":"APPROX (TODO): Replace with exact counts from paper Figure 1 (Total: 3243 latents, 6 scales)."}

Distribution of SAE latents by number of scales (out of 6) with accurate labels

Total: 3,243 latents, 6 scales

best-of-6 (evaluated across 6 input scales)

Untrained

Trained

Qualitative example: an SAE latent (three injection scales), before vs. after training.

Decoding implicit reasoning

One two-hop prompt, answer-only: the model replies “Chile” without stating the bridge entity (“Pablo Neruda”). SelfIE tests whether that intermediate concept is still represented internally.

Example (TwoHopFact): “The author of the novel Twenty Love Poems and a Song of Despair was born in the country of” We instruct the model to respond with only the final answer. It answers correctly (“Chile”)—but never says the bridge entity (“Pablo Neruda”). Is it implicitly “thinking” that intermediate step, or using a shortcut?

To probe that, we decode text from the model’s activations and check whether it generates an alias of the bridge entity across layers and token positions (x is aligned so x=0 is the first non-trivial detection). The heatmaps below compare untrained vs. trained, and the interactive version lets you explore where the signal appears token-by-token.

4.8× reduction in undetected bridge entities

91.0% vs 56.4% (trained vs untrained)

455/500 vs 282/500

11× increase in per-generation detection rate: 4.26%±0.24 trained vs 0.38%±0.04 untrained

Bridge entity detection (interactive)

Hover prompt tokens to see where SelfIE surfaces the bridge entity (“Pablo Neruda”) across layers and token positions. Colors show average detection fraction; token positions are aligned so x=0 is the first non-trivial detection (x is relative, not the token’s absolute position).

{"prefix":"","tokens":[{"text":" The","aligned_pos":-6},{"text":" author","aligned_pos":-5},{"text":" of","aligned_pos":-4},{"text":" the","aligned_pos":-3},{"text":" novel","aligned_pos":-2},{"text":" Twenty","aligned_pos":-1},{"text":" Love","aligned_pos":0},{"text":" Po","aligned_pos":1},{"text":"ems","aligned_pos":2},{"text":" and","aligned_pos":3},{"text":" a","aligned_pos":4},{"text":" Song","aligned_pos":5},{"text":" of","aligned_pos":6},{"text":" Des","aligned_pos":7},{"text":"pair","aligned_pos":8},{"text":" was","aligned_pos":9},{"text":" born","aligned_pos":10},{"text":" in","aligned_pos":11},{"text":" the","aligned_pos":12},{"text":" country","aligned_pos":13},{"text":" of","aligned_pos":14},{"text":"<|eot_id|>","aligned_pos":15},{"text":"<|start_header_id|>","aligned_pos":16},{"text":"assistant","aligned_pos":17},{"text":"<|end_header_id|>","aligned_pos":18},{"text":"","aligned_pos":19}],"suffix":"…","selectedAlignedPos":4}

The author of the novel Twenty Love Poems and a Song of Despair was born in the country of[eot_id][start_header_id]assistant[end_header_id][blank]…

Interactive: hover tokens to explore

Untrained: Pablo Neruda

Trained: Pablo Neruda

Conclusion

Interpretability artifacts can be reused as supervision for small, transferable self-interpreters.

By training only a tiny adapter while keeping the LLM frozen, SelfIE turns interpretability artifacts (labeled vectors from SAEs and contrastive directions) into a practical training signal for stable natural-language interpretations across scales. In practice, simple affine mappings generalize best: they reduce brittle scale tuning, transfer across datasets and layers, and can even help recover implicit intermediate concepts during reasoning.

Freezing the model is not just a training trick: it preserves a conservative “privileged access” setup where the interpreter is the exact same model as the subject, rather than a separately fine-tuned copy. That makes it easier to reason about what changed (only the adapter) and what didn’t (the language model).

This reframes interpretability progress as compounding: as the field produces better artifacts, we also get better self-interpretation “for free,” without modifying the underlying model.

Finally, the paper emphasizes learning failure modes: fluent outputs are not evidence of correct grounding. Pairing self-interpretation with verification signals (like generation scoring) is a path toward more trustworthy readouts.

Citation

@misc{selfie2026adapters,
      title={Learning Self-Interpretation from Interpretability Artifacts},
      author={Keenan Pepper and Alex McKenzie and Florin Pop and Stijn Servaes and Martin Leitgab and Mike Vaiana and Judd Rosenblatt and Michael S. A. Graziano and Diogo de Lucena},
      year={2026},
      eprint={2602.10352},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.10352}
}