Separating the World and Ego Models for Self-Driving

Vlad Sobal, Alfredo Canziani, Nicolas Carion, Kyunghyun Cho, Yann LeCun

Keywords: Model-based policy learning, MPC, motion planning, self-driving, kinematic model, reactive simulation, highway driving, world modeling, video prediction model

TL;DR: We split the forward model for self-driving into a simple model of the ego vehicle and a more complex model of the environment, and show improvement on the task of highway-driving using MPC and policy learning.

Abstract:

Training self-driving systems to be robust to the long-tail of driving scenarios is a critical problem.
Model-based approaches leverage simulation to emulate a wide range of scenarios without putting users at risk in the real world.
One promising path to faithful simulation is to train a forward model of the world to predict the future states of both the environment and the ego-vehicle given past states and a sequence of actions.
In this paper, we argue that it is beneficial to model the state of the ego-vehicle, which often has simple, predictable and deterministic behavior, separately from the rest of the environment, which is much more complex and highly multimodal.
We propose to model the ego-vehicle using a simple and differentiable kinematic model, while training a stochastic convolutional forward model on raster representations of the state to predict the behavior of the rest of the environment.
We explore several configurations of such decoupled models, and evaluate their performance both with Model Predictive Control (MPC) and direct policy learning.
We test our methods on the task of highway driving and demonstrate lower crash rates and better stability.

Community Implementations: [ 3 code implementations](https://www.catalyzex.com/paper/separating-the-world-and-ego-models-for-self/code)