The provided yaml files correspond to the experiments performed in the paper. Each experiment was run multiple times as a sweep over random seeds listed in the files.
The experiments were performed using Weights & Biases to coordinate the sweep and track and store the results.

# Naming convention

Prefix "imc" indicates the IMC cofactor selection experiments and "citeseq" indicates the highly variable gene selection for scRNA-seq experiments.

Word "crossval" in the file title indicates cross-validation experiments.

Word "ablate" or "ablation" in the file title indicates behaviour ablation experiments, with indices "1,2,3" corresponding to which behaviour was being ablated.

Postfixes:
- "ra": random acquisition baseline
- "rs": random scalarization baseline
- "mas" or "ucb_scal_exhaust": MANATEE-AS
- "qnehvi" or "qnhevi_approx": qNEHVI
- "qparego": qNParEGO
- "usemo": USeMO

"citeseq_runs_random_state.yaml" corresponds to the MANATEE-SA method.

"imc_runs.yaml" corresponds to MANATEE-SA, random acquisition, and random scalarization baselines, which were run sequentially.

# Reproducing experiments using yaml files

Running an experiment specified by a yaml file requires Weights & Biases available
in the virtual environment (which it will be as it is specified as a dependency in the `Pipfile`) and the 
user must also be signed in to their account (use command `wandb login --help` for help).

A sweep must be created by running `wandb sweep YAML_FILE.yaml` while located in the directory 
`yaml files`. `YAML_FILE.yaml` specifies the yaml file of your choice. Please note that we were not able
to provide the data files required for running the IMC cofactor selection experiments as they were too large 
for the allowed size of the supplementary materials zip file (50 MB). Therefore the yaml files with "imc" in their
titles will not be possible to run. 

Running the command `wandb sweep YAML_FILE.yaml` will generate an output line starting with `wandb agent`. 
Copy that line to your clipboard.

Executing that line on the command line (with the virtual environment activated and navigated to `automl2023-code`)
will launch a Weights & Biases agent, which will execute the experiment with 
each random seed specified in the yaml file and log the results in the user's 
specified project directory hosted on `wandb.ai`. It will also write files locally in a folder `wandb` and write
output dictionaries of successfully terminated runs in new directories created in `automl2023-code` 
(see README in `automl2023-code`). As there are approximately 100 seeds in all experiments, this can 
take considerable time and generate a lot of outputs.

We performed all experiments for our paper using the Slurm scheduling system. We added an example slurm 
file in `automl2023-code` called `slurm_job.sh` which can be submitted to the scheduler if the user has access
to it.

To do that: navigate to `automl2023-code`, open `slurm_job.sh`, paste the copied line specifying 
the Weights & Biases agent at the bottom of the file, save file and submit the job by running 
`sbatch slurm_job.sh`. 

Please note that `slurm_job.sh` has parameters set according to our system e.g.
partition names, which would have to be edited according to the user's system.
