# Experiments

NOTE: Due to some refactoring, currently only the Memorization-Hyperparameter Relationship (MHPR) experiment in `memorization_hyperparam_rel` is functional.
The other experiments need to be updated to be compatible with the new code structure.


## Code structure (same text as in main README)

There is a folder for each experiment containing the actual code to run it in `experiment.py`, different configurations that run the experiment in different ways in `config.py`, possibly code to generate or load experiment specific data under `data.py`, utilities for loading, visualizing and sharing results in `results.py` and notebooks for specific types of analysis.

Importantly, the `config.py` file of each experiment is supposed to define a handle object that is used by the `main.py` file to register and dispatch experiments.
For example, the Memorization Hyperparameter Relationship (MHPR) experiment defines an `MHPRHandle` object inside its `config.py` file which is imported by `main.py` and added into the `experiment_handles` array, such that it can be invoked from the command line.
The handle objects should define an `id` field, e.g. `mphr`, which is then used to determine which experiment to run.
E.g. if you run `./scripts/run_docker.py --gpus=0 deepspeed src/main.py +sid=0 +mhpr=test`, the code will dispatch the MHPR experiment with the test configuration.

If you need to create a new experiment, the easiest way would be to copy the folder of one of the existing experiments that matches the new one most closely and modify it accordingly to your needs.
In particular, update the `id` field in the experiment handle in the `config.py` file and register the new experiment in `main.py`.
