Many Virtual Screening 'Improvements' May Not Be Real, Pre-Registered Study Finds
- The preprint asks a foundational methods question: given the size of a typical structure-based virtual screening benchmark, how large a difference in AUROC or enrichment between two protocols can actually be distinguished from statistical noise.
- Rather than assuming benchmark comparisons are inherently reliable, the authors first directly measure these resolution limits.
- They then pre-registered six candidate interventions to docking/scoring protocols and tested each against the measured resolution limits, rather than reporting post-hoc 'wins.'
- The pre-registration design is meant to guard against selective reporting, a known issue where small, benchmark-specific gains get presented as general improvements.
- This is a methodology/rigor study, not a new-inhibitor or target-specific paper; it addresses how virtual screening improvements should be evaluated.
- Practical implication: many published claims of improved virtual screening performance may fall within the noise floor of typical benchmarks, so hit-finding teams should treat single-benchmark improvement claims cautiously.
Every VS vendor pitch should have to clear this paper's bar. If a 'protocol improvement' can't beat the measured noise floor of AUROC/enrichment on a benchmark our size, it's not an improvement — it's a coin flip dressed up in a webinar slide. Six pre-registered interventions is a high bar; realistically most of us won't pre-register anything before a campaign, but I can at least ask collaborators to report effect sizes against a resolution limit, not just a headline AUROC bump. Sharpest implication for me: the co-folding hype (Boltz-2 and friends) needs the same treatment — if those models are just pattern-matching PDB-adjacent chemotypes, their 'wins' over classic docking/FEP on public benchmarks could be memorization, not generalization. I'm not swapping FEP for a black box on this evidence alone.