induction heads pop up everywhere.
why some feature are distributed and some are not.
probing the prefix string to see how different is the representation in the paraphrasis
cosine similarity between the weights of the probes between the verb of the preferix and the actual verbs
if there is a mechanisms like the induction head maybe intervene in the prefix string has an effect.
what about multitokens are encoded the same vs single tokens
Prime LM for dataset for passive - active voice.
Todo:
Probing the prefix string (original sentence) and the generated sentence:
- Calculate cosine similarity between the weights of the probes in the prefix and the one of the actual verb
- Check if there is attention patterns that could indicate an induction head.
- Check if there are induction heads.
- Intervene on the representation of the prefix string to see if it changes the results on the actual generated sentence.
Check out Prime LM to see if it can be used to do passive-active voice flips.
There’s a clean dissociation: the head that most strongly attends to the source verb has the weakest causal effect on tense, and vice versa.
L24H9 is the closest thing to a classical induction head — mean attention weight 0.39, far higher than the other two. But its ablation effect is tiny (0.035) and frac_norm = 2.5%. That last number is the key: despite high attention to src_verb, only 2.5% of its output norm at the generation position comes from that edge. It attends to the source verb but the value vector it reads from there projects almost orthogonally through its W_O — so it’s not writing tense information forward. It likely attends to src_verb for a different purpose (positional, syntactic).
L13H13 sits in the middle — sporadic high attention (the bimodal distribution), moderate edge/full. Occasionally does direct induction but the average effect is weak.
L20H15 is the opposite: barely looks at src_verb, but has the strongest causal effect by far. It reads tense from the residual stream (written by upstream processing) and routes it to the pre-verb generation position.
The overall picture is that tense transfer in this model doesn’t work through a clean induction-head copy of the source verb. The circuit is more indirect: some upstream computation (possibly involving L24H9 and L13H13, plus MLPs) encodes the source tense into the residual stream at the generation position, and L20H15 is the head that amplifies and routes that encoded signal into the final tense prediction. The “reader” and “writer” roles are split across layers, with no single head doing both.
What does Edge/full mean
It’s the ratio of two logit drops:
Edge / Full = edge_knockout_drop / full_ablation_drop
-
Full ablation drop: you zero out the entire contribution of head H at position
pre_verb_gen— everything it writes to the residual stream, regardless of where it attended. This measures the head’s total causal importance for tense. -
Edge knockout drop: you remove only the contribution that comes from head H attending to
src_verb_tokspecifically — i.e.attn_weight × V_h[src_verb] @ W_O_h.T. Everything else the head does is left intact.
So Edge/Full answers: of the head’s total tense effect, what fraction is explained by its attention to the source verb?
- Edge/Full ≈ 1 → the head’s effect is almost entirely mediated by attending to src_verb → induction-likex
- Edge/Full ≈ 0 → the head matters (full drop is large) but it’s not reading from src_verb → reads from the residual stream or other positions
- Edge/Full > 1 (e.g. L20H15 example #1 and #2) → the src_verb edge is actually positive, but the rest of the head’s output is suppressive, so removing the whole head is less damaging than removing just that edge
For L20H15: full_drop = 0.121 (strong), edge/full = 14% → the head strongly affects tense but almost none of that comes from attending to src_verb. For L24H9: full_drop = 0.035 (weak), edge/full = 52% → about half of its small effect does go through src_verb attention, but the effect itself is too weak to matter much.
Ablation and Patching of Attention Heads
To identify which attention heads mediate tense transfer from the source sentence to the generated paraphrase, we applied two causal interventions on the pre-verb generation position (pre_verb_gen) — the token just before the model generates the tensed verb. In ablation mode, for each head H at layer L we subtracted H’s full contribution to the residual stream at pre_verb_gen (computed as preproj_h @ W_O_h.T, the head’s slice of the o_proj input projected through its output weights) and measured the drop in the log-probability of the correct tense verb (clean_logit − ablated_logit); a positive drop means the head was supporting the correct tense. In patching mode, we formed past/present example pairs and replaced head H’s o_proj input slice in the target example with the cached slice from the opposite-tense source example, measuring how much the correct verb’s probability decreased after the injection. Both interventions were run across all heads in selected layers (13, 20, 24) over 100 examples. The heatmaps show mean logit drop per (layer, head) cell, with gold stars marking the top-K heads; red (positive) cells indicate heads that support the correct tense, blue (negative) cells indicate suppressive heads. To further test whether the effect of each candidate head is mediated by direct attention to the source verb, we applied an attention edge knockout: instead of removing the head’s entire output, we subtracted only the contribution of the specific attention edge from pre_verb_gen to src_verb_tok (i.e. attn_weight × V_h[src_verb] @ W_O_h.T), and compared the resulting logit drop to the full ablation drop. The ratio Edge/Full measures what fraction of the head’s tense effect flows through that single attention edge. Results show a clear dissociation: L20H15 has the strongest ablation and patching signal (logit drops of +0.13 and +0.09 respectively) but an Edge/Full of only 14% and a mean attention weight to the source verb of 0.02, indicating it reads tense from the residual stream rather than directly from the source verb. Conversely, L24H9 has the highest mean attention to the source verb (0.39) but a weak ablation effect (0.035) and a negligible value-projection norm through its output weights (frac_norm 2.5%), suggesting it attends to the source verb for reasons unrelated to tense. These results suggest that tense transfer in this model does not follow a simple induction-head copy mechanism; instead, upstream heads (and likely MLPs) encode source tense into the residual stream at the generation position, and L20H15 acts as the primary downstream writer that routes this encoded signal into the final tense prediction.
L20H15 does not attend back to src_verb_tok. Instead it attends to other positions. Since its Q/K/V projections still take the residual stream as input, those other positions it attends to must already carry tense-relevant information that was placed there by earlier heads or MLPs.
So the real claim is about where in the sequence L20H15 is looking, not whether it uses the residual stream. A direct induction head would form Q[pre_verb_gen] · K[src_verb_tok] and get a high attention weight, reading the source tense directly from that token’s representation. L20H15 instead gets high attention weights on other positions — possibly pre_verb_gen itself (self-attention), nearby tokens, or other salient positions — and those positions already have tense information integrated into them by earlier computation.
The paragraph should say something like: “L20H15 does not attend directly to the source verb; instead it attends to other positions whose representations have already been enriched with tense information by upstream heads and MLPs.”
To actually know which positions L20H15 is attending to, you’d want to look at its full attention pattern across the sequence — not just the weight to src_verb_tok but the entire row for pre_verb_gen. That would tell you concretely where it’s reading from.
Ablation
The decomposition of each head’s contribution to the residual stream as preproj_h @ W_O_h.T and measuring the logit drop when it is removed comes from the circuit analysis framework in:
- Elhage et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread, Anthropic. — establishes the per-head OV decomposition and the residual stream view that motivates the technique.
- Michel et al. (2019). Are Sixteen Heads Really Better than One? NeurIPS 2019. — one of the first papers to ablate individual attention heads and measure their contribution to task performance via logit change.
Patching
Replacing a head’s activation with one from a contrastive example to measure causal relevance:
- Meng et al. (2022). Locating and Editing Factual Associations in GPT. NeurIPS 2022. — introduces causal tracing (activation patching between a clean and corrupted run) as the core technique for localizing computations.
- Wang et al. (2022). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. ICLR 2023. — uses activation patching and path patching systematically to identify circuits, and introduces the logit-drop metric used here.
Attention Edge Knockout
Isolating a single attention edge (pre_verb_gen → src_verb_tok) by subtracting only attn_weight × V_h[src_verb] @ W_O_h.T:
- Wang et al. (2022). Interpretability in the Wild (same as above). — introduces path patching at the edge level, patching individual (node, position) → (node, position) edges in the computation graph.
- Conmy et al. (2023). Towards Automated Circuit Discovery for Mechanistic Interpretability. NeurIPS 2023. — formalises edge-level attribution and automates the process of identifying which specific attention edges within a circuit are causally necessary.