
Left: Reconstructions from the camera view; Right: reconstructions of the environment, the agent, and the observer from bird's-eye view. Full results on each video collection: [cat], [human], [bunny], [dog]. |
We use the 4D reconstruction as the training data to learn an agent behavior simulator. Below we show the offline-generated behavior of a cat agent in the 3D environment. Left: birds-eye view; Right: third-person view. |
Environment awareness. We can generates diverse environment-aware motion given the same initial state. |
Observer awareness. By providing different observer motion (red triangles), the cat agent will move differently. |
Long sequence generation. We can generate agent behavior over a long time horizon by conditioning on the environment and the past trajectory. |
User control. We can also control the motion of an agent by manually setting the goal (the blue phere). |
Envoronment code. Removing environment code produces a trajectory penetrating into the wall. |
Past code. Removing past code introduces sudden jumps between adjacent trajectory segments. |
Goal denoising (w/ different conditioning signals)Scenario: Exploring a room
|
Path denoising (w/ different conditioning signals)
|
Body motion denoising |