Benchmark / challenge
Open download
public
IEDB automated benchmark (MHC class I)
The only prospective, third-party benchmark in the field. Its eight-year summary is sobering: leading methods are statistically indistinguishable, and a new method needs about four years before enough data accumulate to judge it.
At a glance
ProviderLa Jolla Institute for Immunology
AccessOpen download
Licencepublic
Size notesruns continuously on newly deposited data, before it can leak into anyone's training set
FormatsCSV
Labels and annotations
Weekly blinded evaluation of registered prediction servers on newly submitted binding and elution data.
Details
Why it matters
- Nearly every other published comparison was run by the authors of one of the compared tools — and the host tool wins every time.
- PPV means at least three different things in this literature (1:99 vs 1:999 decoys, same-protein vs random decoys, top-N vs top-0.1%), so cross-paper tables are not comparable.
Models trained or evaluated on it
- benchmark NetMHCpan-4.1 Health Tech, Technical University of Denmark (Nielsen lab)
- benchmark MHCflurry 2.0 O'Donnell, Rubinsteyn and Laserson (Mount Sinai / OpenVax)
- benchmark MHCnuggets Karchin lab, Johns Hopkins University
Sources
This page is educational — it is not medical advice and does not replace consultation with an oncologist. Diagnostic and treatment decisions are made solely by specialist physicians.