# Evaluating Robustness of Cooperative MARL: A Model-based Approach

This folder contains the code that was used in the paper: Evaluating Robustness of Cooperative MARL: A Model-based Approach
and includes implementations of the training and adversarial attacks including:
- Random noise
- [Lin et al. (2020)](https://arxiv.org/pdf/2003.03722) + iFGSM
- Variants of our c-MBA

The experiments are conducted using the [multi-agent MuJoCo](https://github.com/schroederdewitt/multiagent_mujoco) and [multi-agent particle](https://github.com/openai/multiagent-particle-envs) environments .

## Installation instructions

1. Create a conda environment with python 3.8

```
conda create -n cmba python=3.8
```

2. Install torch [here](https://pytorch.org/get-started/locally/) (the experiments were run on CPU only but they can work on GPU machines too).

3. Install mujoco-py, follow instructions [here](https://github.com/openai/mujoco-py#install-mujoco) to download mujoco binaries first. Check [here](https://github.com/openai/mujoco-py#troubleshooting) if run into problems during installation/running `mujoco-py`.

4. Install packages in requirements.txt

```
pip install -r requirements.txt
```

## Run an experiment

All the scripts to run are placed in the folder `run_script`.

Configs of each algorithms are in `src/config/algs`.

Pretrained models are in the `results` folder including dynamics models, MARL models and the single adversarial model when using Li et al. (2020) approach. The path to pretrained models can be changed in `src/config/algs` or when running the experiments. See scripts in `run_script/adv_attack` for more info.

When running experiments, log files will be store under `results/sacred/[env_name]/[run_name]/[run_number]` and the models checkpoints (during training) will be stored at `results/models`. In addition, transitions collected to train dynamic models are stored (by default) in `results/collected_data`.

The `notebook` folder contains notebooks to train dynamic models. It also includes a sample notebook to plot the results. Change the respective path when needed. You might need to install `Jupyter Notebook` or `Jupyter Lab` at [https://jupyter.org/](https://jupyter.org/).

### Adversarial attacks

Once we obtain all pre-trained models. We can perform the adversarial attack by running the scripts in `run_script/adv_attack`.

Random noise attack scripts have the template `adv_noise_[norm_type]_norm.sh` where norm_type can be `l1` or `linf`.

For Lin et al. (2020) + iFGSM attack, the script template is:
- `fgsm_atk_[norm_type]_norm_[agent_index].sh`: iFGSM attack on agent=agent_index using adversarial policy trained on the same agent.

For model-based attack (cMBA) variants, the templates of the scripts are:
- `model_atk_fix_[norm_type]_norm.sh`: original cMBA attack on a fixed agent using expert-defined failure state (c-MBA-F).
- `model_atk_data_[norm_type]_norm.sh`: original cMBA attack on a fixed agent using learned failure state (c-MBA-D).
- `model_atk_[norm_type]_norm_opt.sh`: cMBA attack with optimal victim agent selection.
- `model_atk_[norm_type]_norm_opt_new.sh`: cMBA attack with optimal victim agent selection then run the original cMBA attack again on the selected agent(s).
- `model_atk_[norm_type]_norm_opt_bf.sh`: cMBA attack with greedy agent selection.
- `model_atk_[norm_type]_norm_opt_rd.sh`: cMBA attack with random agent selection.

*Example:* Running adversarial noise attack with $\ell_\infty$-norm in *Walker (2x3)* environment:

```
cd run_script/adv_attack/Walker_2x3
sh adv_noise_linf_norm.sh
```

### Plot results

We include a sample notebook to plot the results generated by the code. Users just need to specify the correct path to where sacred stores the output files. Please see more details in the sample notebook in `notebook/Plot-results-Walker_2x3.ipynb`
