Dataset

A dataset is a versioned, on-disk collection of sharded batches produced by an adapter (or derived from another dataset). Datasets handle downloading, releasing new versions, applying offline Transform pipelines, and exposing lazy access to shards, splits, and assets.

Each dataset is stored under config.dataset_path as <Name>/v<version>/<transform-hash>/. Offline transforms are materialized to disk; live transforms are applied when batches are loaded inside a benchmark.

Implement release() to define how raw data becomes sharded batches. Dataset configs (D_*.yaml) specify either an adapter plus transforms or a parent dataset plus additional transforms. See Datasets.

class bioverse.dataset.ComposedDataset(*datasets: Dataset | ComposedDataset)

Bases: object

Concatenate multiple datasets into a single shard stream.

Used internally when combining datasets with dataset_a + dataset_b. Splits are not yet merged across constituents (see source TODO).

class bioverse.dataset.Dataset(root: Path | str = PosixPath('/home/runner/.bioverse/datasets'), version: int | None = None, online: bool = True, precomputed: dict | Precomputed | None = None)

Bases: ABC

Versioned, on-disk collection of sharded biomolecular data.

Datasets persist batches under config.dataset_path/<Name>/v<version>/. Offline Transform pipelines are materialized to a content-addressed subdirectory; live transforms run at load time.

Subclasses implement release() to produce batches from an adapter or upstream source. Dataset configs (D_*.yaml) reference adapters and transform lists.

Examples

from bioverse.factory import DatasetFactory

dataset = DatasetFactory("D_AFCATH")
dataset.apply(TokenizeResidues())
print(len(dataset), dataset.split.names)

package(dest: Path | str | None = None) → Path

Archive the current dataset version for upload to a repository.