How capable are biological AI models becoming?

Across a nearly two-thousandfold range of size the score rises about 4 points in total, roughly 1.3 points for every tenfold increase, and most of that arrives below 150 million parameters. Above it the line is flat and noisy: the 3 billion model is the best of the family and the 15 billion model sits below it, back at roughly the 150 million level. The intervals overlap heavily, so read the ordering at the top as unresolved rather than as a ranking.

Every number on this page moved when we fixed our own measurement. Scores on this task fell by 1.5 to 3.6 points once the probe layer stopped being chosen on the test set and the homology split started clustering at the identity threshold the field uses. The earlier, higher numbers were our error, not the models' achievement. What we got wrong, and who found what first →

The flattening at the top is not our discovery either. The Ankh team reported in 2023 that ESM-2 15B did not beat smaller ESM models and behaved inconsistently between runs; a 2025 study in Scientific Reports found every model above 150 million performing comparably.

There is also a confound in the axis: every ESM2 model here saw roughly the same number of training tokens regardless of size, so this ladder does not test scale under a budget that grows with the model. A NeurIPS 2024 study that re-allocated that compute beat ESM-2 3B. So this ladder stops paying; protein models in general have not stopped scaling.

These intervals overlap almost entirely, so this is an ordering of best estimates rather than a demonstrated ranking. Our own Standard says to report how likely a ranking is to flip rather than publish a bare order, and we have not computed that here yet, so by our own rule the claim is underpowered. Read it as "nothing here has clearly moved" rather than "2020 beat 2023".

An earlier build of this page showed the 2023 model far below the others, which would have read as a sharp decline. That was a bug in how its sequences were spelled for its tokeniser, not a property of the model. It is recorded here because it is exactly the kind of plausible artefact this programme exists to catch.

The three viruses disagree with each other. On SARS-CoV-2 the internal layers read more than the model's own output does, on every model. On influenza and HIV they read less. Same protocol, same models, opposite sign — so any single assay would characterise the virus as much as the method, and we report all three rather than the average.

These are published laboratory measurements of mutations other people have already made. Nothing here designs a sequence or proposes one, and the numbers are model-level aggregates rather than per-mutation predictions.

Each number below is six independent repeats of the whole experiment, on model families spanning roughly a thousandfold in size. "Room to improve" is how far the models are from a perfect score — a gap cannot grow where there is nowhere left to go.

The two clear results are the strongest thing on this site. Fold recognition and enzyme function share no data, no categories and no method of splitting train from test, and they agree to within a thousandth. Twelve repeats out of twelve point the same way.

Room to improve explains part of the pattern and not all of it. The task showing nothing is the one nearest a perfect score, where a gap is squeezed shut by arithmetic rather than by anything the model is doing. But the genome task has as much room as fold recognition and points the other way. So something else separates them — modality, architecture, or the particular models — and our current design cannot tell which. We would rather say that than pick the flattering explanation.

Two of the numbers are weak and we mark them so. The genome result and a second protein family both clear zero by less than a hundredth while their repeats disagree in direction. We have twice watched a margin that narrow fail to survive more repeats, in both directions, and see no reason to trust these ones further.

Our single run of fold recognition gave a trend of +0.018 with a margin of −0.011 to +0.046, which we reported as no change. Six runs gave +0.020 with a margin of +0.009 to +0.031. The measurement barely moved; only the uncertainty did. The first number was a statement about how little we had run, dressed as a statement about models.

It holds up to the checks we could think of: dropping the largest model, the smallest, or the strongest performer each leaves a trend still clear of zero, and all six repeats point the same way.

The habit was learned the hard way, in the opposite direction. A second family of models appeared, on one run, to show the gap closing sharply. Run four times, the four disagreed entirely and the effect vanished. A single run misleads in whichever direction it happens to fall, which is why every figure above rests on six.

This matters beyond our own results. Benchmark papers in this field routinely report a single run across four to six model sizes. If our headline could reverse under repetition, so can others.

An earlier build of this page said the gap shrank with scale. It did not. That apparent trend was an artefact of two things: choosing the best internal layer by its score on the very data used to report it, which inflated the gap by about 92% on the one model where we isolated it, and a confidence interval that covered 90% while claiming 95%. Both are fixed, and the trend disappeared with them.

The gap is also much larger where there is room for it. On the easy task it is a fraction of a point; on the hard fold-recognition task it reaches 10 points. That is consistent with the small gaps on the easy task having been squeezed shut by the ceiling rather than reflecting anything about the models, which is why we now report every gap twice, raw and relative to the room remaining.

A benchmark normally scores what a model puts out. SONDE also reads what the model holds internally, through a readout of the same size for both, so a difference is a fact about the model rather than the instrument. How the measurement works →

The third surface, what light fine-tuning recovers, is now measured properly on four models, and it lands where an elicitation measurement should: above both the output and the best internal layer, on every one. An earlier build reported it far below both. That was our error — the budget was too small to move a freshly initialised classifier at all, so the number measured our optimiser rather than the models — and those figures are withdrawn.

The interesting part is how fast. At 25 adaptation steps the models sit near the floor; by 200 the mid-sized one has already matched what the best reading of its frozen internals achieves, and by 400 it has passed it. That speed is the quantity that matters for deciding whether releasing a model's weights is releasing a fixed capability or a starting point.

Whether that recovery gets easier or harder as models grow is the question we most want to answer, and four models is not enough to answer it. The trend across them points upward and is not distinguishable from zero on a properly calibrated interval, so we are not reporting it as a trend. More rungs on the ladder, not a better story, is what settles it.

One task, one family, four sizes. It was chosen by measuring five candidate tasks and taking the one least solvable by cheap tricks: two of the others turned out to be nearly answerable by counting four-letter DNA words, and one was essentially a detector for how GC-rich a sequence is wearing a biological label. So this is the hardest of the five, and the models still barely clear the cheap alternative.

It matters beyond capability. Genome models are where the biosecurity question actually lives, and until now this programme could not measure one at all.

• Every score on this page dropped by 1.5 to 3.6 points when we fixed our own measurement. The corrections are described on the prior-work page; the earlier numbers should not be cited.
• Confidence intervals on the scaling curves overlap heavily, so the ordering of the larger models is unresolved rather than measured.
• The size trend covers one model family. The vintage comparison covers five, enough to be suggestive and not enough to be a trend line.
• The biorisk track has no numbers of ours. It is gated on process, not capability. What is known so far on that side is other people's work, credited above.
• Fine-tuning is measured on four models but under a budget too small to be informative; it currently reads as a lower bound, not a capability.
• Our second task turned out to score seven of its fourteen classes, because a protein family and a homology group are nearly the same thing, so holding homologues out removes whole classes. It is reported as seven.
• The fold task has no genuine ceiling, so we do not know how much of the remaining distance is even reachable.
• Genome language models are not covered yet, and the single-cell track is mid-rerun.
• Coverage is a handful of tasks on one dataset per modality, and the layer result in particular is known to vary by dataset.