Learning a Hierarchical Recurrent State Space Model in Complicated Environments

Keno Harada, Masahiro Suzuki, Yutaka Matsuo

Abstract:

Temporal abstraction is considered to contribute sample efficiency in model-based reinforcement learning. The
previsously proposed models for temporal abstraction has been experimented in simple environments. However, for
learning behavior policy in real world such as home service robots, it is necessary to test if temporal abstraction can
be accomplished in complicated environments where high-resolution observations can be obtained and where objects
composed of multiple colors and non-plain patterns exist, rather than an existing experimental environment where
only simple and low-resolution observations can be obtained. We believe that temporal abstraction in a complex
environment requires the use of encoders to extract useful information. We train a hierarchical recurrent state-space
model, which is one of the models for temporal abstraction, on a complex environmental data set and show that
VAE pretraining technique for encoder improves the performance of the model in abstracting observation states
and predicting future transitions given contextual data, compared to the case where the model is trained without
the pretraining technique through evaluation experiments.