The purpose is not to make every study identical. It is to make every important decision visible and every conclusion traceable to evidence.
A model can fail to learn. A tokenizer can encode the wrong sequence. An evaluator can count malformed output as correct. A comparison can accidentally reuse data. An analysis threshold can erase the signal it was meant to measure. The protocol is designed to catch those failures before they become a claim.
Before training begins, we state the exact comparison that would count as evidence and list alternative explanations. Higher similarity might reflect the task, but it might instead reflect shared data, initialization, architecture, output format, or a biased analysis method.
The design changes one factor at a time. Positive controls verify that the measurement can detect a known similarity. Negative, randomized, and shuffled controls estimate how much apparent structure the pipeline creates on its own.
A model first has to overfit a tiny dataset and produce exact answers. The evaluator must parse them correctly. Intervention code, resume logic, checkpoint hashes, and failed-run records are tested before an expensive population run starts.
Dataset partitions, seeds, model roles, inclusion thresholds, intervention settings, and statistical tests are versioned and hashed. Once the confirmatory run starts, scientific parameters do not change in response to the outcome.
Every planned model is accounted for. Infrastructure failures can resume from verified checkpoints. Scientific failures remain in the ledger and are not silently retried with more favorable hyperparameters.
The publication defines technical terms, reports amendments, shows uncertainty, and separates observation from interpretation. A similarity score is not called a recovered algorithm. A sufficient subspace is not called the only circuit.
Two addition models have similar hidden-state geometry.
They may share examples, initial weights, output directions, or generic architecture effects.
Use disjoint datasets, different initialization, XOR controls, shuffled signatures, and locked partitions.
The extractor detects some task-linked causal organization beyond those factors.
The models use one identical, human-readable addition algorithm.
A probe or similarity analysis can show that a pattern exists. It does not show that the model needs the pattern to answer.
Ablation or activation patching can show contribution. Redundant paths can still make necessity difficult to establish.
The strongest test removes the learned route and substitutes an extracted computation that reproduces normal and counterfactual behavior.
Read the complete question, factorial design, intervention method, statistics, amendments, controls, and limitations in one continuous report.
Read the reportRESEARCH DISPATCH
Field notes, protocols, datasets, and experimental releases as they become defensible.
Follow the lab