# QPHIL
This repository is the official repository of the QPHIL project. 

## Presentation
This repository is organized in the following way:
 - algorithms: Contains all the implementations of our algorithms.
 - data: Contains all the general scripts to manage data.
 - docker_builds: Contains docker building files and required dependencies.
 - environments: Contains the scripts of many environments to test.
 - generation: Contains the scripts to generate datasets from diverse agents.
 - models: Contains models saved, used mainly for the generation of datasets.
 - tools: Contains diverse tools for quality of life.
 - visualization: Contains general scripts for data visualization.

## Installation
### Minimal installation
Install miniconda:
```
mkdir -p ~/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm -rf ~/miniconda3/miniconda.sh
```
Change miniconda channel
```
Create a  .condarc in home (or in C:/Users/<you> if you are on Windows)
Edit it with :
channels:
    - conda-forge
```
Create conda environment
```
conda create -n sgcrl python=3.9.19 -y -q
conda activate sgcrl
```
Set channels vars
```
conda config --env --add channels pytorch
conda config --env --add channels nvidia
conda config --env --add channels conda-forge
```
Install diverse requirements with apt-get 
```
sudo apt-get update
sudo apt-get install -y apt-utils
sudo apt-get install -y software-properties-common
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get install -y gnupg2
sudo apt-get install -y wget
sudo apt-get install -y git
sudo apt-get install -y curl
sudo apt-get install -y libosmesa6-dev libgl1-mesa-glx libglfw3
sudo apt-get install -y patchelf
```
Or with conda
```
conda install -c conda-forge mesalib glew glfw (instead or apt-get commands)
conda install libgcc -y (to solve /usr/lib/gcc issue)
conda install patchelf
python -m pip install numpy==1.23.5
export PIP_CACHE_DIR="/home/user/new-pip-cache" (for having a better cache position of pip)
```
Prepare mujoco installation
```
python -m pip install pip --upgrade pip
python -m pip install "cython<3"
```
Install mujoco
```
cd
wget https://github.com/deepmind/mujoco/releases/download/2.1.0/mujoco210-linux-x86_64.tar.gz
mkdir -p .mujoco
tar -xf mujoco210-linux-x86_64.tar.gz --directory /home/user/.mujoco
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/user/.mujoco/mujoco210/bin
export MUJOCO_PY_MUJOCO_PATH=/home/user/.mujoco/mujoco210
```
Install mujoco_py somewhere you like
```
git clone https://github.com/openai/mujoco-py.git
conda install -c conda-forge swig
python -m pip install --no-cache-dir -r ./mujoco-py/requirements.txt
python -m pip install --no-cache-dir -r ./mujoco-py/requirements.dev.txt
python ./mujoco-py/setup.py build 
python ./mujoco-py/setup.py install
```
Install others
```
# Install others
python -m pip install -U torch==2.4.0
python -m pip install -U pandas
python -m pip install -U matplotlib
python -m pip install -U gym==0.23.1
python -m pip install -U gym[mujoco]
python -m pip install git+https://github.com/Farama-Foundation/d4rl@master#egg=d4rl

# Extra command to build mujoco-py
export D4RL_SUPPRESS_IMPORT_ERROR=1
python -c "import mujoco_py"
python -c "import d4rl"
echo 'set editing-mode vi' >> $HOME/.inputrc
echo 'set keymap vi' >> $HOME/.inputrc

# Installing hydra
python -m pip install hydra-core
python -m pip install hydra-submitit-launcher

# Installing dependencies
python -m pip install tensorboard
python -m pip install torchvision
python -m pip install hydra-joblib-launcher --upgrade

# Install jax
python -m pip install "jax[cuda11_cudnn82]==0.4.3" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
conda install -c conda-forge cudatoolkit=11.8 cudnn=8.8
conda install -c "nvidia/label/cuda-11.8.0" cuda-nvcc
```
Git clone the repository with: 
```
git clone repo
```
Go in the folder: 
```
cd qphil_iclr
```
Install the lib in minimal from (antmaze experiments)
```
python -m pip install -e .
```
Also, download the supplementary materials from:
```
https://drive.google.com/drive/folders/1C_NwxC6ruLAnH01wDY1NhRNgmCKGVI_4?usp=sharing
```
and put the folders in the qphil_iclr folder. To use the AntMaze-Extreme datasets, take both datasets rawref=main&private_token=97NvzKm_cG9Xz7ADZWPe&diverse and rawref=main&private_token=97NvzKm_cG9Xz7ADZWPe&play from the drive datasets folder and put them in the .d4rl_ext/datasets folder in your user folder.

Finally, to launch QPHIL training, refer to the:
```
python sgcrl/alorithms/policies/qphil/train.py
```
And to launch evaluation experiments:
```
python sgcrl/alorithms/policies/qphil/experiments/offline_rl_experiment/offline_rl_experiment.py
```