This code was used for the article : "Probability Metrics in Disrtibutional Reinforcement Learning : Why were Gaussian mixtures overlooked ?".

If you want to run experiments with the code, you might need to install the requirements using :
pip install -r requirements.txt

In the code, the following mixture kernel is available :
$k = \theta_0 k_{\rm lap} + \theta_1 k_{\rm mixrbf} + \theta_2 k_{\rm en}$.
To obtain the same kernels as in the paper, one can take $\theta = (1, 0, 0)$ (laplacian), $\theta = (0,1,0)$ (mix rbf), or $\theta = (0, 0, 1)$ (energy).

To run experiments, you can use the example script result.py, the results will be available in the "logs" folder organized as follows :
logs
|-> Environment 1
    |-> run 1
        |-> model
            |-> best (containing the models that gave the best test return)
            |-> final (containing the models that gave the final test return)
        |-> summary (containing the learning curves as a tensorboard file)
    |-> run 2
    etc...
|-> Environment 2
etc...

To see the result, use the following command (replacing name_env by the name of the environment) :
tensorboard --logdir=logs/name_env/

You can choose a certain amount of parameters. For instance, to run the GM-DQM (mixture of gaussians representation) on Asterix in the stochastic setting, you can run the following command in the console :
python result.py --algorithm="MOGN" --env_id="AsterixNoFrameskip-v4"

To run in the deterministic setting, you need to add the option --deterministic

Here are the available algorithms :
_ "MOGN" : Mixtures of Gaussians
_ "QRDQN" : QRDQN
_ "IQN" : IQN
_ "FQF" : FQF

To configure the hyperparameters, modify the corresponding files in the folder config for each algorithm.

Below is a list of Atari environments :
"AirRaidNoFrameskip-v4", 
"AlienNoFrameskip-v4", 
"AmidarNoFrameskip-v4", 
"AssaultNoFrameskip-v4", 
"AsterixNoFrameskip-v4", 
"AsteroidsNoFrameskip-v4", 
"AtlantisNoFrameskip-v4", 
"BankHeistNoFrameskip-v4",
"BattleZoneNoFrameskip-v4", 
"BeamRiderNoFrameskip-v4", 
"BerzerkNoFrameskip-v4", 
"BowlingNoFrameskip-v4", 
"BoxingNoFrameskip-v4", 
"BreakoutNoFrameskip-v4", 
"CarnivalNoFrameskip-v4",
"CentipedeNoFrameskip-v4", 
"ChopperCommandNoFrameskip-v4", 
"CrazyClimberNoFrameskip-v4", 
"DemonAttackNoFrameskip-v4", 
"DoubleDunkNoFrameskip-v4", 
"ElevatorActionNoFrameskip-v4",
"EnduroNoFrameskip-v4", 
"FishingDerbyNoFrameskip-v4", 
"FreewayNoFrameskip-v4", 
"FrostbiteNoFrameskip-v4", 
"GopherNoFrameskip-v4", 
"GravitarNoFrameskip-v4", 
"HeroNoFrameskip-v4",
"IceHockeyNoFrameskip-v4", 
"JamesbondNoFrameskip-v4", 
"JourneyEscapeNoFrameskip-v4", 
"KangarooNoFrameskip-v4", 
"KrullNoFrameskip-v4", 
"KungFuMasterNoFrameskip-v4", 
"MontezumaRevengeNoFrameskip-v4",
"MsPacmanNoFrameskip-v4", 
"NameThisGameNoFrameskip-v4", 
"PhoenixNoFrameskip-v4", 
"PitfallNoFrameskip-v4", 
"PongNoFrameskip-v4", 
"PooyanNoFrameskip-v4", 
"PrivateEyeNoFrameskip-v4",
"QbertNoFrameskip-v4", 
"RiverraidNoFrameskip-v4", 
"RoadRunnerNoFrameskip-v4", 
"RobotankNoFrameskip-v4", 
"SeaquestNoFrameskip-v4", 
"SkiingNoFrameskip-v4", 
"SolarisNoFrameskip-v4", 
"SpaceInvadersNoFrameskip-v4", 
"StarGunnerNoFrameskip-v4", 
"TennisNoFrameskip-v4", 
"TimePilotNoFrameskip-v4", 
"TutankhamNoFrameskip-v4", 
"UpNDownNoFrameskip-v4", 
"VentureNoFrameskip-v4",
"VideoPinballNoFrameskip-v4", 
"WizardOfWorNoFrameskip-v4", 
"YarsRevengeNoFrameskip-v4", 
"ZaxxonNoFrameskip-v4"
