Teaching a small model to proofread a brain map

I'm Dan Siroker, and this is the first result we measured ourselves.

When a computer traces neurons through a stack of microscope images, it makes mistakes. It cuts one neuron in two, or it glues two neurons together. Today, people find and fix those mistakes by hand. On the one cubic millimeter of mouse brain that has been mapped, that took about a million corrections. On a whole mouse brain it would take, by Wellcome's 2023 estimate, $7.5 to 21.7 billion, almost all of it wages.

So the question we had to answer before anything else was simple. Can a model do the proofreading? And if it can, how fast does it get better?

What we trained

Every correction a human proofreader made to the fly map and to the mouse cubic millimeter was logged. About 700,000 of those decisions have been curated into a public benchmark, ConnectomeBench2, with two tasks that carry the benchmark's own names:

1. Split correction: here are two pieces. Should they be joined?
2. Merge identification: here is one object. Is it two cells wrongly joined?

We fine-tuned a small image model on those decisions. ConvNeXt-Atto, 7 million parameters. For comparison, the language model you talked to this morning has hundreds of billions. Think of it as the difference between a pocket calculator and a data center. We wanted the calculator, because a factory that maps brains will run this model billions of times, and a model this small costs almost nothing per decision.

What it scored

We trained it on 1,000, 4,000, 16,000, 79,500 and all 531,734 training decisions, three seeds per point, and scored every run on the benchmark's 23,730 held-out decisions. Balanced accuracy, mean of three seeds:

The human figures come from a small sample with wide intervals. So on split correction, call it a match. On merge identification, it is a clear lead. A 56 million parameter version of the same architecture, which we call Proofreader, scores 97.75 and 95.52 percent on the benchmark's full 101,929-decision test split.

That said, the headline number is not the point. The shape of the curve is.

The curve

The error falls as a power law in training data, with exponents of 0.201 on split correction and 0.158 on merge identification, and it has not leveled off. In plain terms: every doubling of labels cuts the error by about 13 percent on one task and 11 percent on the other. A small, predictable gain, every time.

Why does that matter more than the score? Because a factory needs to know what the next dollar buys. If we can say "another million corrections cuts the error by this much," we can plan the labeling budget the way you plan any other line item. If the curve had flattened, no amount of money would get us there. It has not flattened yet. That is the property a factory needs, and not, on its own, the whole plan.

Does it transfer to the new microscope?

Here is the catch. Every one of those 531,734 decisions came from electron microscopy. The maps we intend to build come from light microscopy of expanded tissue, where the images look different. A model trained on one might be useless on the other.

So we tested it. E11 Bio published expansion microscopy data with protein barcodes, which let us derive labels for the same two tasks. We applied our electron-microscopy model to that data with no retraining at all. Chance is 0.50 AUROC. The model scored 0.695 on split correction and 0.758 on merge identification. Not great. Not chance either. It had learned something about how neurons look that survives the change of microscope.

Then we fine-tuned it on the expansion labels, 250 to 3,865 of them, three seeds per point, against a control that started from a generic image model instead of ours. Starting from the connectomics model, 250 labels matched what the generic start needed 2,000 labels to reach on split correction, and 1,000 labels on merge identification. In other words, the electron-microscopy training was worth four to eight times its weight in new labels. The test sets were 564 and 672 decisions, so these are small numbers with real uncertainty, and we say so.

What this means for us is practical. The data from the old method trains the models for the new one. We can start on public electron-microscopy data today and finish on light.

What this does not show

Scoring decisions on a benchmark is not finishing a brain. A benchmark hands you a candidate and asks yes or no. A real map does not hand you anything. The model has to find the mistakes first, in a volume with billions of pieces, and then fix them without making new ones. We tested that separately, on nine held-out cubes of the public mouse cortex volume, and the result is more modest. It is the next post.

Every per-point number, the fit, and the transfer tables are on the scaling page, so you can check the exponents yourself. If you see a mistake in the arithmetic, tell me!

Dan