Model organism is a term mechanistic interpretability borrowed from biology and then quietly changed the meaning of.
In biology, a model organism is a real animal. C. elegans, Drosophila, zebrafish, Arabidopsis. You choose it because it is small, cheap, fast, and tractable, and because the things you learn from it keep being true elsewhere. Nobody chose C. elegans because it was a simplified cartoon of a worm. They chose it because it is a worm, one with 302 neurons and a connectome you can finish.
In interpretability, model organism usually means a toy: a small transformer trained on a synthetic task that was designed to exhibit the phenomenon you already had in mind. Modular addition, induction, a planted backdoor. These are genuinely useful and I do not want to talk anyone out of them. But they have a structural problem that no amount of care fixes: the ground truth is yours by construction. You wrote the generator, so you know the answer, and the model's job was to find the answer you planted. When your method recovers it, what you have shown is that your method can recover something you put there.
Biological foundation models give you the biology version of the term instead. This post is the argument for why, and an account of what we have already got out of treating them that way. It is a companion to Open Problems in Biological Mechanistic Interpretability, which is the list of things to do; this is the argument for the venue.
What actually makes a good model organism
Four properties, translated out of biology:
- You can afford to look at all of it.
- Ground truth exists outside the system, and you did not make it.
- It is doing real work that people depend on, so being wrong has a cost.
- What you learn transfers to the systems you actually care about.
Toy models have the first, and sometimes the fourth. Frontier language models have the third, and arguably the fourth, and neither of the others. Biological foundation models have all four at once, which as far as I can tell makes them the only widely available system that does.
1. You can afford to look at all of it
scGPT and Geneformer sit in the tens to hundreds of millions of parameters. That is small enough that the exhaustive version of an experiment is the version you actually run, rather than the version you describe in the future work section.
Our exhaustive circuit mapping work is the clearest case. Every head ablated, higher-order combinatorial ablations, causal trajectory steering across the whole model. What came back was massive redundancy, a heavy-tailed hub architecture, and layer-dependent control of differentiation directionality. None of those findings are exotic methods. They are what you see when you can afford to run the sweep to completion, and almost nobody can afford to run it to completion on a frontier model.
The same economics let you do things that are otherwise absurd. We ran 141 geometric and topological hypotheses through an automated screen, because screening 141 hypotheses was cheaper than arguing about which three to test. We have measured scaling behaviour for masked-reconstruction transformers on single-cell data directly, rather than inferring it. Cheapness is not a minor convenience here. It changes which experiments are thinkable.
2. The ground truth is not yours
This is the property a toy model cannot have, and it is the important one.
Biology arrives with an answer key that nobody wrote for your benefit. CRISPRi and Perturb-seq screens say which perturbations actually move which genes. Curated regulatory references say which transcription factors are known to regulate which targets. Donor identity, batch structure, tissue of origin, sequencing depth and evolutionary conservation are all recorded, and all of them are inconvenient. None of it was constructed to make an interpretability method look good.
Which means claims can fail against it, and ours have. Our comparative SAE atlas of Geneformer and scGPT found rich, modular, biologically coherent organisation in the representations, and then, on CRISPRi validation, minimal causal regulatory logic. Both halves of that sentence are results. The first half is the kind of thing that gets a paper; the second half is the kind of thing that stops you believing the first half means more than it does. In a toy model the regulatory logic is whatever we wrote into the generator, so the second half of that sentence could not have existed.
Two more in the same shape. We found that attention-derived edge scores add no incremental value over simple gene-level features for perturbation-target prediction, across models, cell types and perturbation modalities, the attention scores encode structured biology, and are still not useful for the thing people use them for. And when we put in-silico perturbation profiles for intelligence-associated genes in human prefrontal cortex through expression-matched null testing, no individual gene survived.
A model organism you cannot be wrong about is not a model organism. It is a demo.
3. The ways the evaluation lies are measurable
A subtler benefit: because the reference data is incomplete in characterised ways, you can quantify how it misleads you instead of just worrying about it.
Incomplete reference labels put a ceiling on apparent performance, and we can compute where that ceiling is, so a mediocre score can be recognised as a saturated one. Benchmark rankings move when you change candidate sets, tissue context or reference network, and we can put a reversal probability on the ordering rather than reporting a rank as if it were stable. Technical leakage, genomic-proximity inflation and degree-distribution artifacts each manufacture edges that look real, and we can audit all three and report how many candidate edges survive.
These are not biology-specific problems. Leaky splits, absent baselines, ceilings nobody reported and rankings that invert under resampling are the ambient failure modes of interpretability everywhere. Biological AI is a place where their size can be measured, and a method that survives the audit here has survived something.
4. Sometimes you can take the mechanism out and keep it
The strongest thing a mechanistic result can do is stop needing the model. Extract the mechanism, delete the network, and find that the mechanism still works.
The hematopoietic manifold work is our clearest instance of this, and I think it is the single best argument in the post. scGPT internally carries a compact, roughly eight to ten dimensional hematopoietic manifold with real developmental branch structure. We exported it: direct operator export from frozen attention weights, a lightweight learned adaptor, and a task-specific readout, with no retraining on the target dataset. The result is a standalone algorithm.
It is also a competitive one. On donor-holdout benchmarks against scVI, Palantir, DPT, CellTypist, PCA and raw expression, the extracted algorithm gives the strongest pseudotime-depth ordering and competitive cell-type classification. Interpreting the exported operator turns up a concentrated four-factor core accounting for 66.2% of ablation impact, and those four factors resolve into explicit T/lymphoid, B/plasma, granulocytic and monocyte/macrophage gene programmes. The operator compresses from three pooled attention heads to one without statistically significant loss, and then further to a rank-64 surrogate.
There is no language-model analogue of this, and the absence is instructive. You cannot export the induction head and ship it as a product that competes with the state of the art on an established benchmark, because there is no established benchmark for what the induction head does that other people were already trying to win. Biology has decades of methods competition on tasks that matter to somebody, which means an extracted mechanism can be entered into a contest it might lose.
Things you cannot do on language models at all
Collecting the argument into the list I actually care about:
Prospective experimental validation. You can take a claim the model's internals imply, walk it into a lab, and find out. There is no wet lab for the model has an induction head. This is the only setting I know of where mechanistic interpretability can be wrong about the world rather than merely wrong about a network.
Controlled attribution of what pretraining bought you. We compared seven single-cell foundation models against a protein language model and a co-expression baseline to ask which signals need single-cell pretraining at all and which were already available from protein sequence. Running that comparison requires several independently trained models on the same underlying reality, with meaningful non-neural baselines. Language modelling has neither.
Cross-model convergence as evidence. Geneformer and scGPT have different tokenisations, objectives and training corpora, and both are modelling the same cells. When causal circuit tracing finds consensus circuits across both, that convergence is evidence about biology rather than about a shared architectural quirk.
Ablations that would be a compute programme elsewhere. Feature-permutation ablation over an entire spatial model, as in our four-category SAE taxonomy for Novae, is a normal experiment here.
Making the model organism easier to read
Everything above takes the architecture as given and works around it. The next move is to stop doing that.
Almost everything we can currently say about a transformer's computation is a statement about activations on a distribution. You interpret what the model did on the data you showed it, and the honest version of any such claim carries a silent on this dataset. Sparse autoencoders, probes, activation patching and circuit tracing are all, in the end, input-conditioned. That is not a criticism of the methods. It is a property of where the computation lives.
Tensor-network layers change where it lives. Tensor-transformer variants are surprisingly performant as a general architectural result, which was the necessary precondition for any of this being interesting: an interpretable architecture that costs you performance is a demonstration, not an option.
We have been training tensor-network foundation models on single-cell data, and multimodal ones over several biological modalities at once. This work is not published yet and I will keep the claims to the shape of the result rather than the numbers. Three things seem to hold. Performance does not suffer, and in places appears to improve. Much more of the computation is carried in structured weights rather than in input-dependent attention, so a far larger share of what the model does can be read directly off the parameters, weight-based interpretability, without the silent on this dataset attached. And the representations that come out are richer and easier to extract, which matters directly for the extraction programme in the previous section.
If that holds up, the model-organism argument gets a good deal stronger. It would mean we are no longer only choosing systems that happen to be tractable. We would be building systems designed to be read, in the one domain where there is external ground truth to check the reading against. That is the version of this programme I am most interested in, and it is also the part I am least able to defend in public right now, so treat it as a statement of direction.
What would change my mind
Three things would substantially damage the argument.
The ground truth could be too weak to carry the weight I am putting on it. CRISPRi screens are noisy, curated references are incomplete and biased toward well-studied genes, and pathway annotations reflect what people have looked at. I have argued that this is manageable because the incompleteness is characterised. If it turns out that the characterisation is itself wrong in the direction that flatters us, a lot of the above weakens at once.
The transfer to language models might not happen. The claim that methods surviving here will survive there is, at present, mostly an argument from the generality of the failure modes. It is not yet an empirical result, and it is the part of the case I would most like someone to attack.
The models might be doing something much shallower than we think. Our own SAE atlas already found much less causal regulatory logic than the organisation of the representations suggested. If that keeps happening, the honest conclusion is that these systems are excellent compressors of expression statistics and poor models of regulation, and the interesting interpretability target is elsewhere.
All three are testable, which is roughly the point.
Summary
Interpretability took a word from biology and dropped the part that made the concept work, which was that the organism is real and can therefore disagree with you. Biological foundation models give that part back. They are small enough to trace exhaustively, they are checked against evidence nobody generated for our convenience, their evaluation pathologies can be measured rather than merely feared, and every so often a mechanism can be lifted out of one and made to stand on its own. We are now trying to build the next generation of them so that reading the weights is the primary way you understand the model, rather than the thing you wish you could do.
The list of specific problems is here. The published work behind the claims above is in Research Outputs.