# JAHS-Bench-201 Leaderboards

The leaderboards as explained in our paper. To add an entry please create a pull request changing `docs/leaderboards.md`.
Make sure to follow our evaluation protocol and to add a URL to material that explains how to reproduce you results.

## Single Objective

### Black-box

#### CIFAR-10

| Rank | Accuracy ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 90.68 ± 0.16 | random search | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Colorectal-Histology

| Rank | Accuracy ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 94.76 ± 0.08 | random search | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Fashion-MNIST

| Rank | Accuracy ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 95.09 ± 0.05 | random search | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


### Cost-aware

#### CIFAR-10

| Rank | Accuracy ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 90.68 ± 0.16 | random search | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Colorectal-Histology

| Rank | Accuracy ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 94.76 ± 0.08 | random search | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Fashion-MNIST

| Rank | Accuracy ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 95.09 ± 0.05 | random search | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


### Multi-fidelity

#### CIFAR-10

| Rank | Accuracy ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 90.78 ± 0.14 | Hyperband | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Colorectal-Histology

| Rank | Accuracy ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 95.27 ± 0.04 | Hyperband | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Fashion-MNIST

| Rank | Accuracy ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 95.13 ± 0.03 | Hyperband | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


### Multi multi-fidelity

#### CIFAR-10

| Rank | Accuracy ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 90.9 ± 0.14 | Hyperband | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Colorectal-Histology

| Rank | Accuracy ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 94.91 ± 0.07 | Hyperband | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Fashion-MNIST

| Rank | Accuracy ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 95.18 ± 0.02 | Hyperband | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


## Multi Objective

### Black-box

#### CIFAR-10

| Rank | Hypervolume ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 1171.15 ± 2.27 | random search | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Colorectal-Histology

| Rank | Hypervolume ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 278.6 ± 2.23 | random search | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Fashion-MNIST

| Rank | Hypervolume ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 231.88 ± 3.86 | random search | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


### Cost-aware

#### CIFAR-10

| Rank | Hypervolume ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 1171.15 ± 2.27 | random search | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Colorectal-Histology

| Rank | Hypervolume ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 278.6 ± 2.23 | random search | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Fashion-MNIST

| Rank | Hypervolume ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 231.88 ± 3.86 | random search | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


### Multi-fidelity

#### CIFAR-10

| Rank | Hypervolume ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 1080.26 ± 1.36 | Hyperband | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Colorectal-Histology

| Rank | Hypervolume ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 347.52 ± 1.96 | Hyperband | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Fashion-MNIST

| Rank | Hypervolume ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 277.52 ± 1.07 | Hyperband | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


### Multi multi-fidelity

#### CIFAR-10

| Rank | Hypervolume ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 899.92 ± 1.54 | Hyperband | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Colorectal-Histology

| Rank | Hypervolume ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 330.76 ± 0.22 | Hyperband | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |


#### Fashion-MNIST

| Rank | Hypervolume ± SE | Name | URL |
| ---- | ----- | ---- | ---- |
| 1 | 306.36 ± 0.19 | Hyperband | [JAHS-Bench-201](https://github.com/automl/jahs_bench_201_experiments) |
