Open science challenges in preclinical biology
Eight thousand papers, forty proposed mechanisms, and room for maybe one. Help us find out which.
Papers and Findings opens November 4th, 2026
Rectangles are claims about the state of the system. The words between them are what gets you from one to the next. Colour is how far the evidence supports that step. Select any of them.
well supported partly supported weakly supported nobody has explained this step
After Figure 1 of the Moult and Darden preprint. The colours record the schema builder's own evidence assessment. Choosing this mechanism as the example does not mean it is better supported than the others.
Why this needs a challenge
The problem is not that we do not know enough. It is that most of the things we think we know are probably wrong.
This is not a story about incompetence or fraud. Known fraud in Alzheimer's research is rare, and great ingenuity has gone into the experiments behind these papers. The problem is volume. Fewer than half of reported preclinical findings replicate, and replication effect sizes average under 20% of those first reported. No person and no lab can hold 8,000 papers in view, let alone weigh them.
The goal
Create AI powered pipelines that generate a comprehensive knowledge representation of a disease: what is known and not known about the mechanisms underlying it, where the key gaps are, what experiments would fill them, and what it all implies for therapy.
The strength of AI here is scale. Humans can critically assess the literature, but the size of the task is overwhelming, and that is the primary reason we have a replication crisis. Preliminary work shows that properly instructed AI systems can handle this kind of complexity. Current systems also make mistakes and cannot be trusted, which is the whole difficulty. How do we build pipelines for this task that hold up?
The form of the AI systems is up to participants. Structure is imposed on the submissions so they can be compared against one another.
The work
Find the papers bearing on a mechanism hypothesis, and extract the relevant primary experimental findings. Tasks a and b, November 4th to December 1st, 2026.
Assess the strength of the evidence for and against each finding. Task c.
Infer the mechanism steps, and assign a confidence to the proposed mechanism. Tasks d and e.
Suggest the experiments most needed to close the evidence gaps. Task f.
Examine what the mechanism implies for treatment. Task g.
Take part
Papers and Findings, Step 1. Five hypotheses about how APOE4 affects endosome function.
We expected the first two tasks to have existing solutions, given how central literature retrieval is to AI-for-science work. As far as we can tell they do not, and tests within the working group confirmed that both are far from trivial.
We ran an agent collab on Hugging Face in which 38 contributed agents worked together on these tasks across five APOE4 hypotheses. They collaborated impressively and retrieved papers better than the working group had. Performance on extracting experimental detail was very poor.
Participants now get a new set of five hypotheses and are asked to find the relevant papers and extract the detail. This is a pilot, to see whether the challenge and assessment designs are sound.
Assessment
What the literature contains is itself part of what is being measured, so the assessment methods break new ground and may develop as the project goes on.
Kaggle predefines one ranking metric. CASP predefines a subset and lets assessors add more once results are in. We set out initial considerations without a ranking formula.
| Cross-evaluation by participants, each scoring ten others | if submissions are many |
| Comparison across groups: which papers, which experiments | identifies the easy results |
| Each finding's relevance, by the directness criteria | |
| Each finding's effect size, P value and system, against the paper | hard for current methods |
| Manual inspection | where necessary |
Background
None of it is needed to take part.
The lack of effective treatments for Alzheimer's remains the major failure of modern medicine. Between 60 and 70% of people with a diagnosis carry a copy of APOE4, so prophylactics built on mechanistic understanding could at least halve the number of cases.
APOE codes for a small protein of 299 amino acids, the primary means of lipid transport in the brain, whose main job seems to be ferrying lipids, cholesterol above all, from astrocytes where they are made to neurons which need a lot and make little. Most people carry two copies of APOE3. APOE4 differs by one base, which puts an arginine at residue 112 where E3 has a cysteine.
Odds ratios for people homozygous for E4, against those homozygous for E3, have been reported as 20 to 30 in neurologically confirmed cohorts. E4 is uncommon in most populations, yet most people with a diagnosis carry at least one copy.
The mechanisms assessed include a set classified by changes in apoE protein properties and interactions, and a biologically broader set. This draws on a Moult and Darden preprint.
The language. Disease mechanisms are usually described in very technical terms. It is all standard material, and a quick query fills any gap. We suggest teams that include biology expertise and help with matchmaking, but it is not essential: the main foundation models are biology-aware. Do not be put off by the vocabulary.
Alzheimer's disease. 6.5 million diagnosed in the US today, 13.6 million expected by 2060. Despite 3.2 billion dollars a year from the NIH alone, there is little consensus on mechanism and no effective treatment. Diagnosis requires specific cognitive impairment plus amyloid plaque and tau tangles. Plaque begins decades before symptoms and does not correlate with impairment; tangles come later and correlate strongly. The form of interest here is late onset, a complex trait disease with variants at 75 known loci, of which APOE4 is by far the strongest.
Genetic variants. Each of us carries about two million variants from the reference genome. Most are benign. Some cause disease directly, as one base change in hemoglobin causes sickle cell anemia. Others shift the odds of a complex trait disease, usually by very little, odds ratios under 1.2. Alzheimer's is a complex trait disease, which is what makes APOE4 so strange.
Lipids. Mostly found in membranes. Most are not water soluble, so moving between cells needs packaging. ApoE is the primary lipid packaging protein in the brain.
Proteins. Chains of amino acids, twenty types. The sequence usually stabilizes one folded shape, whose surface lets the protein interact with other molecules. ApoE recognizes and interacts with lipids in ways that are less well understood.
The unfolded protein response. A folded protein is only weakly stabilized against the unfolded form, so one amino acid can leave a significant fraction unfolded. Unfolded proteins aggregate and cause trouble, and cells answer by slowing many processes, protein production above all. Whether this is happening can be read from the RNA produced for each gene.
Cell types. Five matter most here. Neurons, the network nodes. Astrocytes, which supply them with lipids. Microglia, the brain's immune cell. Oligodendrocytes, which make the myelin around nerve fibres. And pericytes, which bridge the brain and the vascular system at the blood-brain barrier. Each signals to the others.
Organizers
Listed alphabetically by surname. Support: BEACON/SGC.
| Georgia Channing | Hugging Face |
| Lindley Darden | University of Maryland |
| Oishi Deb | Oxford University |
| Abigail Djossou | University of Toronto |
| Nikolas Kalavros | NYU |
| Andriy Kryshtafovych | UC Davis |
| Jeff Lam | Hong Kong University |
| John Moult | University of Maryland |
| Jan Pedersen | Bright Minds |
| Emma Scharfmann | Hugging Face |
| Gustavo Stolovitzky | NYU |
| Yuchen Zhou | BEACON/Conscience |
References