Benchmark

A benchmark is the central evaluation unit in Bioverse. It wires together four components declared as class attributes:

• a Dataset

• a Sampler

• a Task

• a Metric

loader() drives the full data path: the sampler picks batch indices, the task extracts features and targets from a VirtualBatch, and an optional collater prepares framework-ready batches for the trainer. After inference, update() feeds predictions to the metric.

Benchmark configs (B_*.yaml) select a dataset and override sampler, task, or metric settings. See Benchmarks.

class bioverse.benchmark.Benchmark(root: Path | str = PosixPath('/home/runner/.bioverse/benchmarks'), version: int = 0, split: str = 'default', n_jobs: int | None = None)

Bases: ABC

Orchestrate dataset loading, task extraction, and metric evaluation.

A benchmark binds four components declared as class attributes:

• Dataset

• Sampler

• Task

• Metric

loader() drives the full path from sampled indices to (X, y) pairs (and optionally collated batches). The Trainer iterates loaders and calls update() / result() during evaluation.

Examples

Using a benchmark config (B_*.yaml):

from bioverse.factory import BenchmarkFactory

benchmark = BenchmarkFactory("B_AFCATH")
for (X, y), collated in benchmark.loader("train", batch_size=32):
    ...

Attaching live transforms:

from bioverse.transforms import Random2DRotate

benchmark.live(Random2DRotate())

__init__(root: Path | str = PosixPath('/home/runner/.bioverse/benchmarks'), version: int = 0, split: str = 'default', n_jobs: int | None = None) → None

Parameters:

• root (Path or str, optional) – Root directory for benchmark data storage, defaults to config.benchmarks_path

• version (int, optional) – Version number of the benchmark, defaults to 0

• split (str or None, optional) – Split name for the benchmark data, defaults to None

• n_jobs (int or None, optional) – Number of parallel jobs to run. If None, uses all available cores

Notes

The benchmark class requires the following class attributes to be defined: - dataset: Dataset class, instance, or (class, kwargs) tuple - sampler: Sampler class, instance, or (class, kwargs) tuple - task: Task class, instance, or (class, kwargs) tuple - metric: Metric class, instance, or (class, kwargs) tuple