
## Code Accompanying "Handling Cost and Constraints in Off-Policy Deep Reinforcement Learning"

----------------------

----------------------

### Building & Installing

 To install this code and get it running, first set up a python virtual environment. We developed and tested this using 
 a conda environment and python 3.6, but one should be able to replicate the results using other configurations.  Create the 
 environment and then do the following:
 
1. Clone this repository.

2.  This code base uses PyTorch.  To ensure that you get the correct version, go to their 
[installation page](https://pytorch.org/get-started/locally/) and enter your system characteristics.

3. If you want to run Safety Gym experiments, clone their repository and place it in the `./envs` folder.
    - Running Safety Gym requires MuJoCo. Further instructions for downloading and installing MuJoco can be found at
      [their website](http://www.mujoco.org).
    - Install Safety Gym by going to `./envs/safety-gym` and typing `pip install -e .`.
 
4. The parallelization in this code base relies on MPI.  We found that the conda-forge version works well;
to get it, type `conda install -c conda-forge mpi4py`.

5. Install risk and its subpackages.  From this home folder, type `pip install -e .`.

----------------------


----------------------

### Training and Testing


Note that the default mode is "train" in all learning code. The random seed can also be configured using the command line.

In general, the .json configuration files should go in the `./configs` folder.
The TensorBoard log files should go in the `./logs` folder. Model files (.pt format) and test results (.pkl format)
should go in the `./output` folder.  Plotting code can be found in `./risk/plotting`.

To view the progress of your runs, go to `./logs` and, in your python environment, call
TensorBoard.  This can be done with a command like `tensorboard --logdir=. --port=7300` (or whatever you want the port 
to be; the default is 6006).  Locally (on a Mac) you can then view your TensorBoard at 
[https://localhost:7300](https://localhost:7300) (or whatever your port number is).  If the localhost prefix doesn't 
work, try replacing it with your IP address.

----------------------
