Benchmarks

Benchmark configs (B_*.yaml) wire a dataset to a sampler, task, and metric. Each config may include description and citation keys for documentation (ignored at runtime). citation may be a string or a list of strings when multiple references apply.

B_AFCATH

Inverse folding on CATH domains from D_AFCATH with recovery and BLOSUM metrics.

Please cite

Li et al. “ProteinInvBench: Benchmarking Protein Inverse Folding on Diverse Structures.” (ProteinInvBench repository; see A4Bio/ProteinInvBench for the current citation).

dataset: D_AFCATH
sampler: MoleculeSampler
task: InverseFoldingTask
metric:
- RecoveryMetric: null
- BlosumScoreMetric: null

B_AFFULL

Inverse folding on AlphaFold Swiss-Prot structures with Foldseek exclusions, with recovery and BLOSUM metrics.

Please cite

Jumper et al. “Highly accurate protein structure prediction with AlphaFold.” Nature 596, 583–589 (2021). https://doi.org/10.1038/s41586-021-03819-2

dataset: D_AFFULL
sampler: MoleculeSampler
task: InverseFoldingTask
metric:
- RecoveryMetric: null
- BlosumScoreMetric: null

B_AFINVF

Inverse folding on AlphaFold Swiss-Prot structures with recovery and BLOSUM metrics.

Please cite

Jumper et al. “Highly accurate protein structure prediction with AlphaFold.” Nature 596, 583–589 (2021). https://doi.org/10.1038/s41586-021-03819-2

dataset: D_AFSP00
sampler: MoleculeSampler
task: InverseFoldingTask
metric:
- RecoveryMetric: null
- BlosumScoreMetric: null

B_AFMR20

Masked residue prediction on 20% of residues in AlphaFold Swiss-Prot structures.

Please cite

Jumper et al. “Highly accurate protein structure prediction with AlphaFold.” Nature 596, 583–589 (2021). https://doi.org/10.1038/s41586-021-03819-2

dataset: D_AFSP00
sampler:
  ResidueSampler:
    num_residues: 0.2
task: MaskedResiduePredictionTask
metric: RecoveryMetric

B_ECINVF

Inverse folding on E. coli AlphaFold structures.

Please cite

Jumper et al. “Highly accurate protein structure prediction with AlphaFold.” Nature 596, 583–589 (2021). https://doi.org/10.1038/s41586-021-03819-2

dataset: D_AFEC00
sampler: MoleculeSampler
task: InverseFoldingTask
metric: RecoveryMetric

B_INVATM

Atom-resolution inverse folding on ProteinInvBench CATH domains.

Please cite

Li et al. “ProteinInvBench: Benchmarking Protein Inverse Folding on Diverse Structures.” (ProteinInvBench repository; see A4Bio/ProteinInvBench for the current citation).

dataset: D_INVC42
sampler: MoleculeSampler
task:
  InverseFoldingTask:
    resolution: atom
metric: RecoveryMetric

B_INVC42

Inverse folding on CATH domains; evaluates sequence recovery and BLOSUM score.

Please cite

Li et al. “ProteinInvBench: Benchmarking Protein Inverse Folding on Diverse Structures.” (ProteinInvBench repository; see A4Bio/ProteinInvBench for the current citation).

dataset: D_INVC42
sampler: MoleculeSampler
task: InverseFoldingTask
metric:
- RecoveryMetric: null
- BlosumScoreMetric: null

B_INVFTR

Cβ position regression as an inverse-folding-related structure task on CATH domains.

Please cite

Li et al. “ProteinInvBench: Benchmarking Protein Inverse Folding on Diverse Structures.” (ProteinInvBench repository; see A4Bio/ProteinInvBench for the current citation).

dataset: D_INVC42
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: CBetaPosition
metric: MeanAbsoluteErrorMetric

B_PROGYM

Protein mutational effect prediction on ProteinGym deep mutational scanning data.

Please cite

Notin et al. “ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design.” NeurIPS Datasets and Benchmarks Track (2023).

dataset: D_PROGYM
sampler: MutationSampler
task: MutationEffectPredictionTask
metric: SpearmansRhoMetric

B_PROGYM_BLAT_ECOLX

Mutational effect prediction on the ProteinGym BLAT_ECOLX assay.

Please cite

Notin et al. “ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design.” NeurIPS Datasets and Benchmarks Track (2023).

Stiffler et al. “Protein Stability Engineering Insights Revealed by High-Throughput Screening.” PNAS 112, E3098–E3106 (2015). https://doi.org/10.1073/pnas.1504567112

dataset: D_PROGYM_BLAT_ECOLX
sampler: MutationSampler
task: MutationEffectPredictionTask
metric: SpearmansRhoMetric
split: random_mutation_split

B_PSAFSP

Inverse folding on ProteinShake AlphaFold Swiss-Prot structures with top-k recovery metrics.

Please cite

Günther et al. “ProteinShake: Building Blocks and Benchmarks for Data-Driven Protein Modeling.” NeurIPS Datasets and Benchmarks Track (2023).

dataset: D_PSAFSP
sampler: MoleculeSampler
task: InverseFoldingTask
metric:
- RecoveryMetric
- TopkRecoveryMetric:
    k: 2
- TopkRecoveryMetric:
    k: 3

B_PSEC00

Enzyme commission (EC) multi-class classification on ProteinShake structures.

Please cite

Günther et al. “ProteinShake: Building Blocks and Benchmarks for Data-Driven Protein Modeling.” NeurIPS Datasets and Benchmarks Track (2023).

dataset: D_PSEC00
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: ec1
    resolution: residue
metric:
- MultiClassAccuracyMetric
- MacroPrecisionMetric
- MacroRecallMetric

B_PSGO00

Gene ontology molecular function prediction (Fmax) on ProteinShake structures.

Please cite

Günther et al. “ProteinShake: Building Blocks and Benchmarks for Data-Driven Protein Modeling.” NeurIPS Datasets and Benchmarks Track (2023).

dataset: D_PSGO00
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: molecular_function
metric:
- FmaxMetric

B_PSLDEC

Virtual screening on protein–ligand decoys with enrichment and mean active rank metrics.

Please cite

Günther et al. “ProteinShake: Building Blocks and Benchmarks for Data-Driven Protein Modeling.” NeurIPS Datasets and Benchmarks Track (2023).

dataset: D_PSLDEC
sampler: MoleculeSampler
task: VirtualScreenTask
metric:
- EnrichmentFactorMetric
- MeanActiveRankMetric

B_PSLINT

Protein–ligand binding site residue prediction on ProteinShake interface data.

Please cite

Günther et al. “ProteinShake: Building Blocks and Benchmarks for Data-Driven Protein Modeling.” NeurIPS Datasets and Benchmarks Track (2023).

dataset: D_PSLINT
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: binding_site
    level: residue
    resolution: residue
metric:
- AurocMetric:
    'on': 2
    per: 1
- AuprcMetric:
    'on': 2
    per: 1

B_PSPFAM

Pfam family multi-class classification on ProteinShake structures.

Please cite

Günther et al. “ProteinShake: Building Blocks and Benchmarks for Data-Driven Protein Modeling.” NeurIPS Datasets and Benchmarks Track (2023).

dataset: D_PSPFAM
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: pfam
metric:
- MultiClassAccuracyMetric
- MacroPrecisionMetric
- MacroRecallMetric

B_PSPPI0

Protein–protein interface contact prediction on ProteinShake structures.

Please cite

Günther et al. “ProteinShake: Building Blocks and Benchmarks for Data-Driven Protein Modeling.” NeurIPS Datasets and Benchmarks Track (2023).

dataset: D_PSPPI0
sampler: InterfacePairSampler
task:
  PairwisePropertyPredictionTask:
    meta: interface_contacts
    level: molecule
metric:
- AurocMetric
- AuprcMetric

B_PSRCSB

Inverse folding on ProteinShake RCSB structures with top-k recovery metrics.

Please cite

Günther et al. “ProteinShake: Building Blocks and Benchmarks for Data-Driven Protein Modeling.” NeurIPS Datasets and Benchmarks Track (2023).

dataset: D_PSRCSB
sampler: MoleculeSampler
task: InverseFoldingTask
metric:
- RecoveryMetric
- TopkRecoveryMetric:
    k: 2
- TopkRecoveryMetric:
    k: 3

B_PSSCOP

SCOP fold classification on ProteinShake structures.

Please cite

Günther et al. “ProteinShake: Building Blocks and Benchmarks for Data-Driven Protein Modeling.” NeurIPS Datasets and Benchmarks Track (2023).

dataset: D_PSSCOP
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: scop_fa
metric:
- MultiClassAccuracyMetric
- MacroPrecisionMetric
- MacroRecallMetric

B_PSTMAL

Pairwise TM-score (lDDT) regression on ProteinShake structural alignments.

Please cite

Günther et al. “ProteinShake: Building Blocks and Benchmarks for Data-Driven Protein Modeling.” NeurIPS Datasets and Benchmarks Track (2023).

dataset: D_PSTMAL
sampler: PairwiseSampler
task:
  PairwisePropertyPredictionTask:
    meta: lddt
    level: molecule
metric:
- MeanAbsoluteErrorMetric
- SpearmansRhoMetric

B_QM9APH

QM9 isotropic polarizability (alpha) regression.

Please cite

Ramakrishnan et al. “Quantum Chemistry Structures and Properties of 134 Kilo Molecules.” Scientific Data 1, 140022 (2014). https://doi.org/10.1038/sdata.2014.22

dataset: D_QNTMA9
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: alpha
metric: MeanAbsoluteErrorMetric

B_QM9CV0

QM9 heat capacity at 298 K (Cv) regression.

Please cite

Ramakrishnan et al. “Quantum Chemistry Structures and Properties of 134 Kilo Molecules.” Scientific Data 1, 140022 (2014). https://doi.org/10.1038/sdata.2014.22

dataset: D_QNTMA9
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: Cv
metric: MeanAbsoluteErrorMetric

B_QM9GAP

QM9 HOMO–LUMO gap regression.

Please cite

Ramakrishnan et al. “Quantum Chemistry Structures and Properties of 134 Kilo Molecules.” Scientific Data 1, 140022 (2014). https://doi.org/10.1038/sdata.2014.22

dataset: D_QNTMA9
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: gap
metric: MeanAbsoluteErrorMetric

B_QM9HOM

QM9 HOMO energy regression.

Please cite

Ramakrishnan et al. “Quantum Chemistry Structures and Properties of 134 Kilo Molecules.” Scientific Data 1, 140022 (2014). https://doi.org/10.1038/sdata.2014.22

dataset: D_QNTMA9
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: homo
metric: MeanAbsoluteErrorMetric

B_QM9LUM

QM9 LUMO energy regression.

Please cite

Ramakrishnan et al. “Quantum Chemistry Structures and Properties of 134 Kilo Molecules.” Scientific Data 1, 140022 (2014). https://doi.org/10.1038/sdata.2014.22

dataset: D_QNTMA9
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: lumo
metric: MeanAbsoluteErrorMetric

B_QM9MU0

QM9 dipole moment (mu) regression.

Please cite

Ramakrishnan et al. “Quantum Chemistry Structures and Properties of 134 Kilo Molecules.” Scientific Data 1, 140022 (2014). https://doi.org/10.1038/sdata.2014.22

dataset: D_QNTMA9
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: mu
metric: MeanAbsoluteErrorMetric

B_RMDASP

Atomic force vector prediction on revised MD aspirin trajectories.

Please cite

Chmiela et al. “Machine Learning Accurate Exchange and Correlation Functionals of the Electronic Density.” Nature Communications 10, 3887 (2019). https://doi.org/10.1038/s41467-019-12827-2

dataset: D_RMDASP
sampler: FrameSampler
task:
  PropertyPredictionTask:
    property: force
    level: atom
metric: MeanAngularErrorMetric

B_RNKRNA

RNA structure quality (RMS) regression on ARES puzzle models.

Please cite

Townshend et al. “Geometric Deep Learning of RNA Structure.” Science 373, 1047–1051 (2021). https://doi.org/10.1126/science.abe5650

dataset: D_ARES00
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: rms
metric: SpearmansRhoMetric

B_SRFC42

Inverse folding on surface-filtered CATH domains with recovery and top-k metrics.

Please cite

Li et al. “ProteinInvBench: Benchmarking Protein Inverse Folding on Diverse Structures.” (ProteinInvBench repository; see A4Bio/ProteinInvBench for the current citation).

dataset: D_SRFC42
sampler: MoleculeSampler
task: InverseFoldingTask
metric:
- RecoveryMetric: null
- TopkRecoveryMetric:
    k: 2
- TopkRecoveryMetric:
    k: 3

B_WNGIR0

Protein–ligand interface RMSD (iRMSD) regression on Weng docking benchmark structures.

Please cite

Weng et al. “Docking Benchmark Version 5.5.” (see Weng lab docking benchmark for the current citation).

dataset: D_WNGDK0
sampler: MoleculeSampler
task:
  PropertyPredictionTask:
    property: irmsd
    resolution: residue
metric:
- MeanAbsoluteErrorMetric
- SpearmansRhoMetric