BioBench: A Blueprint to Move Beyond ImageNet for Scientific ML Benchmarks

Samuel Stevens

Submission Track: Short papers presenting ongoing research or work submitted to other venues (up to 5 pages, excluding references)

Keywords: benchmarking, imagenet

TL;DR: We assemble a suite of computer vision tasks for benchmarking ecological vision models.

Abstract:

ImageNet-1K linear-probe transfer accuracy remains the default proxy for visual representation quality, yet it no longer predicts performance on scientific imagery.
Across 46 modern vision transformer checkpoints, ImageNet top-1 accuracy explains only 34% of variance on ecology tasks and mis-ranks 30% of models above 75% accuracy.
We present BioBench, an open ecology vision benchmark that captures what ImageNet misses.
BioBench unifies 9 publicly released, application-driven tasks, 4 taxonomic kingdoms, and 6 acquisition modalities (drone RGB, web video, micrographs, in-situ and specimen photos, camera-trap frames), totaling 3.1M images.
A single Python API downloads data, fits lightweight classifiers to frozen backbones, and reports class-balanced macro-F1 (plus domain metrics for FishNet and FungiCLEF); ViT-L models evaluate in 6 hours on an A6000 GPU.
BioBench provides new signal for computer vision in ecology and a template recipe for building reliable AI-for-science benchmarks in any domain.
Code and predictions are available at [github.com/samuelstevens/biobench](https://github.com/samuelstevens/biobench) and a results website is at [samuelstevens.me/biobench](https://samuelstevens.me/biobench) .

Submission Number: 55