Causal convergence in tiny Transformers

PLAIN-LANGUAGE SUMMARY

What we actually tested

Imagine training two small neural networks to add binary numbers. The first network sees one set of addition problems. The second sees a completely different set. They also begin with different random weights. Both eventually learn to answer almost every possible problem correctly.

Correct answers alone do not tell us whether the networks learned similar internal computations. One could use a carry-like procedure while another relies on a different collection of shortcuts that happens to work on this tiny domain. We therefore changed thousands of internal values, one at a time, and measured how each change altered each output bit. That produced a large causal response profile for every model.

We then asked whether two addition models had more similar causal response profiles than an addition model and a matched XOR model. They did. In the hardest condition, addition-to-addition similarity was 0.394, compared with 0.281 for addition-to-XOR. The separation was 0.113.

01 / RESEARCH QUESTION

What should remain the same when almost everything else changes?

Neural networks trained from different random starting points usually end with different numerical weights. Their neurons also have no guaranteed one-to-one correspondence. Directly comparing weights or neuron numbers therefore asks the wrong question.

This experiment instead asks whether the effects of internal interventions converge. If independently trained addition models respond similarly when corresponding internal channels are removed, that is evidence that training produced some shared causal organization. To rule out a trivial explanation, the models in the primary comparison share neither training examples nor initial weights.

02 / DATASET AND PARTITIONS

A complete domain of 256 addition problems

Each input contains two unsigned 4-bit integers. A 4-bit integer can represent 0 through 15, giving 16 possible left operands and 16 possible right operands. Their Cartesian product contains exactly 256 ordered pairs. The target is the correct 5-bit sum, which can represent values from 0 through 30.

For each of 10 partitions, the 256 examples were divided into train_A with 96 examples, train_B with 96 different examples, and a shared test set with 64 examples. The three sets had zero overlap. The exact example IDs were locked and hashed before the confirmatory run.

The partition search considered 2,000 candidate splits and selected matched sets using operand magnitude, output magnitude, carry count, carry position, active input bits, overflow, and chained-carry behavior. This reduced the chance that one training set was simply easier than the other.

03 / MODEL ARCHITECTURE

The same tiny decoder-only Transformer in every run

The model has a 128-number residual representation, four Transformer blocks, four attention heads per block, and a context length of five positions. At each of the first four positions, the model receives the corresponding bit from operand A, the corresponding bit from operand B, and a flag set to zero. The fifth position contains no operand bits and a final-position flag set to one.

An input projection and positional embedding create the initial residual stream. Each block applies causal self-attention and a feed-forward MLP through residual connections. A final normalization and one-channel output head produce one binary logit at each of the five positions, together forming the 5-bit answer.

04 / EXPERIMENTAL DESIGN

Four conditions isolate data and initialization

The primary factorial design changes two variables independently: whether a pair of models receives the same training examples, and whether it begins from the same initial parameters. This creates four addition-to-addition comparisons.

Each seed-and-partition combination produced 14 model roles: eight addition roles for the factorial design, two XOR controls, two random-function controls, and two label-shuffled controls. Across 20 paired seeds and 10 partitions, this produced 2,800 trained checkpoints.

XOR uses exactly the same input encoding and model capacity but implements a different bitwise function. It is the principal negative control for task-specific similarity. Random-function and label-shuffled models help reveal whether the extractor reports high similarity merely because models share an architecture, output format, or optimization process.

05 / TRAINING AND INCLUSION

Successful behavior was required before structural comparison

Models were trained in PyTorch with Adam, a learning rate of 0.0003, full batches of 96 examples, 500 training epochs, dropout of 0.05, and binary cross-entropy with logits. Training ran on Apple's MPS backend. Every planned checkpoint completed; there were no infrastructure or numerical training failures in the confirmatory population.

The primary structural analysis included a model only when it achieved exactly 100% training accuracy and at least 95% exact-answer accuracy on the shared test set. Of 2,800 completed models, 2,346 passed that gate. The main addition conditions averaged approximately 99.6% exact test accuracy. Analyses including all runs were also retained as robustness checks.

06 / CAUSAL MEASUREMENT

How one model became a causal signature

The extractor inspected 14 sites: the embedding; the attention output, MLP output, and residual stream after each of four blocks; and the final residual stream immediately before the output head.

1. 01

   Run every probe example normally and record the model's five output logits.

2. 02

   At one internal site, choose one of five token positions and one of 128 channels.

3. 03

   Set that internal channel to zero, rerun the same examples, and record how all five output logits change.

4. 04

   Repeat across sites, positions, channels, and examples. Flatten the effects into one behavior-conditioned intervention signature.

5. 05

   Normalize signature columns, then remove components that can be explained by output identity and the correct-answer direction.

6. 06

   Compare two output-removed signatures using linear centered kernel alignment, or linear CKA.

This method is causal in the limited experimental sense that it physically changes an internal value and observes the output response. The final similarity score is still an engineered summary of those responses. It is not a direct readout of the model's algorithm.

07 / STATISTICAL PLAN

The analysis was frozen before the full run

The primary signature included every nonzero intervention effect, using a causal-strength threshold of 0.0. A threshold of 0.01 remained a predefined sensitivity analysis. Pairwise results were stratified by seed and partition. Confidence intervals used 2,000 bootstrap resamples; permutation tests used 5,000 permutations; and site-level tests were corrected for multiple comparisons with the Holm procedure.

The central prediction was ordered: same task and same data should be at least as similar as same task with disjoint data, and same-task disjoint-data similarity should exceed the different-task control. The decisive comparison used disjoint data and different initialization.

08 / RESULTS

Same-task similarity survived disjoint data

The primary addition-to-addition estimate was 0.3939 across 185 eligible pairs, with a 95% confidence interval of 0.3746 to 0.4142. The matched addition-to-XOR estimate was 0.2809 across 184 pairs, with a confidence interval of 0.2697 to 0.2921. The intervals are clearly separated.

The result is positive, but two controls prevent a stronger claim. First, a matched random-subspace baseline reached 0.333, only 0.061 below the primary same-task result. Second, independently trained XOR and random-function populations showed very high within-task similarity, approximately 0.809 and 0.798. The extractor is therefore highly sensitive to structure that may be generic to a task and architecture, not specifically to arithmetic algorithms.

The study detected a reproducible task-linked causal signal. It did not cleanly separate that signal from every generic structural effect.

09 / WHERE THE SIGNAL APPEARED

Task-specific separation was strongest late in the network

We repeated the same-task versus different-task comparison at individual intervention sites. The earliest attention and MLP sites did not reliably distinguish addition-to-addition from addition-to-XOR. Separation increased in later residual streams and became strongest in Block 3's MLP and residual output.

This pattern is consistent with early blocks performing broadly reusable input processing while later blocks become more task-specific. It does not prove that interpretation. A late layer can also amplify or reformat computation performed earlier, and the intervention method may be more sensitive near the output.

Carry-stratified probes produced mean within-task similarity of 0.524 on no-carry examples and 0.451 on one-carry examples. More demanding carry behavior did not produce stronger measured convergence in this analysis.

10 / CAUSAL-CORE ANALYSIS

A smaller subspace preserved behavior, but it was not the only usable route

A secondary extractor searched for a lower-dimensional subspace that retained the intervention structure. Across 2,346 successful models, the selected core used an average of 34.5% of the available rank. Preserving that core retained 99.4% accuracy. A matched random subspace retained 93.8%.

The core was sufficient: retaining it preserved nearly all behavior. It was not uniquely necessary: removing it still left 87% accuracy. That is consistent with redundancy, distributed computation, or alternate internal routes. Calling the subspace “the addition circuit” would overstate the evidence.

11 / ARCHITECTURE COMPARISON

The same measurement was less separated in Transformers than in GRUs

This study strictly replicated an earlier calibration experiment performed with GRUs. The Transformer same-task versus different-task convergence margin was 0.130 under the architecture comparison analysis. The GRU margin was 0.216. The Transformer-minus-GRU difference was -0.086, with a paired permutation p-value of 0.0002.

Architecture therefore materially affected what the extractor measured. One plausible reason is that a GRU presents a single recurrent hidden state, while a Transformer distributes computation across positions, attention, MLPs, and residual pathways. Another is that the architectures truly learn differently. The current experiment cannot distinguish those explanations.

12 / PROTOCOL AMENDMENTS AND FAILED GATES

The final protocol was not the first protocol

Two failures occurred before the confirmatory population and remain part of the record. First, the GRU learning rate of 0.01 failed the Transformer's preregistered overfit gate. The Transformer protocol therefore used an architecture-appropriate rate of 0.0003. The original failure was preserved as an optimization control.

Second, Protocol 1.1 used a primary intervention-strength threshold of 0.01. Transformer intervention effects were generally smaller, so the resulting signatures failed the viability gate. Protocol 1.2 changed only the primary threshold to 0.0 and retained 0.01 as a predefined sensitivity analysis. Protocol 1.2.0 was then frozen and hashed before the complete run.

13 / LIMITATIONS

What remains unresolved

1. 01
   The extractor is not the algorithm

   A causal response signature summarizes many interventions. It does not produce a symbolic addition procedure or a graph of the exact operations used by the model.

2. 02
   The random-subspace baseline is high

   The primary 0.394 similarity is only modestly above 0.333 for matched random subspaces, indicating substantial generic geometry in the measurement.

3. 03
   The domain is intentionally tiny

   All 256 input pairs can be enumerated, but many distinct internal implementations can agree perfectly on a finite domain this small.

4. 04
   Coordinate interventions may miss distributed equivalence

   Zeroing one channel at a time can underestimate computations spread across many directions or overemphasize representations close to the output.

5. 05
   No circuit replacement occurred

   We did not extract a compact program, remove the learned path, and demonstrate that the program reproduces every normal and counterfactual behavior.

6. 06
   Architecture and extractor are entangled

   The weaker Transformer margin may reflect a different learned computation, a worse measurement fit, or both.

14 / CONCLUSION

A real signal, not yet an extracted mechanism

Independently trained tiny Transformers solving binary addition from disjoint examples developed intervention-response structures that were more similar to one another than to matched XOR models. The effect survived different initialization, output-direction removal, locked data partitions, and population-level statistics.

That is evidence for reproducible task-linked causal organization. It is not evidence that every model implements one canonical addition circuit. The modest margin over random subspaces, the very high within-task similarity for non-arithmetic controls, and the sufficiency-without-necessity result all show that the current extractor mixes meaningful task structure with generic properties of trained networks.

The actionable result is methodological: future work should stop treating higher similarity as the finish line. A stronger extractor must identify a compact computational object, demonstrate necessity and sufficiency, and survive causal substitution across independent models. Until then, the honest conclusion is narrower: convergence is detectable, but the underlying algorithm remains unrecovered.