Code base for the Paper: Automaton-Guided Dynamic Task Sampling for Reinforcement Learning Agents


Overview:

Reinforcement Learning (RL) has made significant strides in enabling artificial agents to learn diverse behaviors. However, learning an effective policy often requires a large number of environment interactions. To mitigate sample complexity issues, recent approaches have used high-level task specifications, such as Linear Temporal Logic (LTLf) formulas or Reward Machines (RM), to guide the learning progress of the agent. In this work, we propose a novel approach, called Automaton-Guided Dynamic Task Sampling (AGTS), that learns a set of RL policies to guide an agent from an initial state to a goal state based on a high-level LTLf task objective, while minimizing the number of environmental interactions. Unlike previous work, AGTS does not assume information about the environment dynamics or the Reward Machine, and dynamically samples promising tasks that lead to successful goal policies. We evaluate AGTS on a grid-world and show that it achieves improved time-to-threshold performance on complex sequential decision-making problems compared to state-of-the-art RM and Automaton-guided RL baselines, such as Q-Learning for Reward Machines and Compositional RL from logical Specifications (DiRL). Moreover, we demonstrate that our method outperforms RM and Automaton-guided RL baselines in terms of sample-efficiency, both in a partially observable robotic task and in a continuous control robotic manipulation task.

Experiments for Minigrid can be found in the Minigrid/ folder.
To train the Gridworld-AGTS: $ python Minigrid/AGTS/main.py
To train the Gridworld-AGTS-CT: $ python Minigrid/AGTS_ct/main.py

Experiments for TurlteBot domain can be found in the TurtleBot/ folder.
To train the TurtleBot-AGTS: $ python TurtleBot/main.py

Experiments for Panda arm domain can be found in the Panda/ folder.
To train the Panda-AGTS: 
## Requirements
- python=3.5.2
- openai-gym=0.12.5 (mujoco200 is supported, but you need to use gym >= 0.12.5, it has a bug in the previous version.)
- mujoco-py=1.50.1.56 (~~**Please use this version, if you use mujoco200, you may failed in the FetchSlide-v1**~~)
- pytorch=1.0.0 (**If you use pytorch-0.4.1, you may have data type errors. I will fix it later.**)
- mpi4py

## Instruction to run the code
If you want to use GPU, just add the flag `--cuda` **(Not Recommended, Better Use CPU)**.
1. train the **PandaPush-v2**:
```bash
mpirun -np 1 python -u train.py --env-name='PandaPush-v2' --n-cycles=10 2>&1 | tee reach.log
```