What happens when you hide the names

On October 16, 2026 we published a result I was proud of. AI agents given the wiring diagram of a fly brain and a simulator built on it predicted 259 published experiments at 0.90 AUROC, against 0.83 for the simulator alone. AUROC is a score from 0.5 (a coin flip) to 1.0 (perfect) for ranking which experiments produced a response. The gain was +0.07, above the +0.05 bar we had set before the first run.

Then I asked a question I did not enjoy asking. How much of that was the map, and how much was the AI remembering the papers?

The honest answer took four more tests and about a day. Here is what they showed.

Test 1. Take away the names

Every neuron in the fly wiring diagram has a name, like MN9 or roundup, and most of the 259 experiments come from papers that use those names. A language model that has read the fly literature can look at a name and recall the result. So we reran the agents with every cell-type name, transmitter and brain region replaced by an opaque code. The agents could still read every connection and run every simulation. They just could not recognize the paper.

The gain fell from +0.07 to +0.04, with a 95 percent interval of +0.01 to +0.07. That is above zero and below our bar. By the rule we wrote down in advance, the test failed.

Where did the gain go? It depends which experiments you look at:

On the 169 experiments the model was built on, the blind agent is the simulation. The named agent's edge there, +0.03, was entirely the names. On the 90 independent experiments, the blind agent still beats the simulation by +0.09 (interval +0.04 to +0.15), about half of the named agent's +0.16.

The agents told us the same thing in their own words. In the named run they flagged reliance on prior knowledge on 136 to 150 of 259 items. Blind, that fell to 40. The celebrated corrections of the named run, where the agent overruled a wrong simulation, turned out to be memory: the blind agent followed the simulation and got them wrong.

Test 2. Hide one thing at a time

A blind test says the names matter. It does not say which part of the names. So we built a ladder with one replicate per level:

1. Hide the paper-identifying text, keep the cell-type names. Score: 0.924 overall, 0.844 on the independent items. Nothing lost.
2. Hide the cell-type names, keep the transmitter, the brain region and the free-text design notes. Score: 0.877 overall, 0.742 independent. That is the blind floor.
3. Hide the annotations too, keep only the design notes. Score: 0.894 overall, 0.800 independent.

The cell-type names carry the independent-item gain. The transmitter and region annotations add nothing we can detect. The design notes are worth something on the 169 validation items, because the text tells the agent the readout is the proboscis motor neuron, and that is legitimate experimental context. That said, one replicate per level means a difference of 0.03 to 0.06 between adjacent levels is inside the noise, and the second replicate was stopped before it finished.

Test 3. Use papers the model has never seen

If memory is the problem, use experiments published after the model's training data was collected. We built a key of 53 experiments from 12 fly papers first public in 2026 and gave a names-only control, no map and no model, a chance to guess them.

It scored 0.815 on the 33 items from papers published since June 2026 and 0.852 on all 52 scorable items. The pass bar was 0.60. Fail.

Why? Because 2026 fly results are guessable from the names and the design. The control's reasons cite papers from 2017 and 2018, never a 2026 source. It was not remembering the new paper. It knew enough biology to predict it. A test like this needs experiments without precedent, and about three times as many.

Test 4. Ask older models

One more question. Is the names-only score recall of specific papers, or general knowledge of fly biology? We ran the same names-only prompt through nine models with training cutoffs from September 2021 to June 2026. The two September 2021 models cannot have read the fly model paper, which first appeared in May 2023.

The scores climbed with the cutoff: 0.557 for GPT-4 (0613) and 0.570 for GPT-3.5, 0.585 for GPT-4o and GPT-4.1, 0.646 to 0.713 for the Claude 4.5 models, 0.825 for the current model. But the old models were just as far behind on the 2026 papers no model has seen. So most of the gap is capability, not recall.

Then we asked the current model directly. For 85 percent of the 259 experiments it named the right source paper. It recalled the right result for 53 percent. It recognizes the paper. It does not reliably remember the answer. Even so, the named agent's gain on independent items sat on the recognized ones: +0.19 on 76 recognized against +0.03 on 14 unrecognized. This test cost $25 in API calls and was not pre-registered.

What I take from this

The map and the model add about +0.09 AUROC on experiments the model was not built on. That is real, it survives every blinding we tried, and it is smaller than the number we published first. The rest of the first result was the model knowing the fly literature.

A benchmark built from published experiments cannot separate memory from mechanism, and newer papers do not fix it, because biology is predictable from the design of the experiment. The clean test is an experiment nobody has published: a compound or a perturbation with no fly data, predicted from the model before the lab runs it, with the prediction written down first.

That is the test we want to run next. If you work with flies and have one, I would love to hear from you!

Every number here, with its intervals and the transcripts, is in the full write-up at braininsights.ai.

Dan