# PhysWorld: From Real Videos to World Models of Deformable Objects via Physics-Aware Demonstration Synthesis


Abstract: *Interactive world models that simulate object dynamics are crucial for robotics, VR, and AR. However, it remains a significant challenge to learn physics-consistent dynamics models from limited real-world video data, especially for deformable objects with spatially-varying physical properties. To overcome the challenge of data scarcity, we propose PhysWorld, a novel framework that synthesizes physics-aware demonstrations to learn efficient world models. Specifically, we first optimize the physical properties (\eg, friction, elasticity) of an object, and integrate them into a Material Point Method (MPM) simulator. This process effectively distills a real video into a physics-consistent digital twin. Subsequently, we construct extensive and physically plausible demonstration data by sampling diverse control-point trajectories and applying part-aware perturbations to the physical properties within the MPM simulator. Finally, using these demonstrations, we train a GNN-based world model that is embedded with physical properties. The real video can be used to further refine the properties of the world model. PhysWorld achieves accurate and fast future predictions for various deformable objects, and also generalizes well to novel interactions. Experiments show that PhysWorld has competitive performance while enabling inference speeds 20 times faster than the recent state-of-the-art method, \ie, PhysTwin. The code and pre-trained models will be publicly available.*



## Setup

### Python Environment
To prepare the Python environment needed to run PhysWorld, execute the following commands:
```shell
conda create -n PhysWorld python=3.9
conda activate PhysWorld

pip install -r requirements.txt
pip install -e gaussian-splatting/submodules/diff-gaussian-rasterization/
pip install -e gaussian-splatting/submodules/simple-knn/
```
We use pytorch=2.1.0+cu118.

## stage1 Script Description

The `stage1_gradient.py` script is the main entry point for running the first stage of the optimization process using the Gradient Optimizer. 

To run the script, execute the following command in your terminal:

```bash
python stage1_gradient.py
```
---

## stage2 Script Description

The `stage2_gradient.py` script is the main entry point for running the second stage of the optimization process using the Gradient Optimizer.

To run the script, execute the following command in your terminal:

```bash
python stage2_gradient.py
```
---

## evaluate Script Description

The `evaluate.py` script is used to evaluate the optimized parameters from the first and second stages of the optimization process.

To run the script, execute the following command in your terminal:

```bash
python evaluate.py
```

## generating interaction demonstrations
The `generate_training_data.py` script is used to generate interaction demonstrations for training GNN.

To run the script, execute the following command in your terminal:

```bash
python generate_training_data.py

```

### Config Json File
A single `.json` file should detail all data preprocessing and simulation parameters for each scene. Key parameters include:

---

#### Data Preprocessing Parameters
- **`opacity_threshold`**: `0.00`  
  Filters out Gaussian kernels with opacity below this threshold. A value of `0.00` means no filtering is applied.

- **`rotation_degree`**: `[0]`  
  The rotation angle of the scene. A value of `[0]` means no rotation is applied.

- **`rotation_axis`**: `[1]`  
  The axis of rotation for the scene. A value of `[1]` typically corresponds to the Y-axis.

- **`sim_area`**: `[-10, 10, -10, 10, -5, 5]`  
  Defines the bounding box for selecting particles for simulation. The format is `[xmin, xmax, ymin, ymax, zmin, zmax]`. This ensures only particles within this region are included in the simulation.

---

#### Simulation Parameters
- **`material`**: `"jelly"`  
  The material type. Supported materials include `jelly`, `metal`, `sand`, `foam`, `snow`, and `plasticine`. Each material has unique physical properties. Specifically, when the material is selected as`cloth`, the parameter content will undergo some changes. For details, please refer to default_cloth.json

- **`E`**:
  Young's modulus, which measures the stiffness of the material. A higher value indicates a stiffer material.

- **`nu`**:
  Poisson's ratio, which describes the material's tendency to expand in directions perpendicular to compression. A value of `0.4` is typical for many soft materials.

- **`density`**:
  Material density, which affects the mass and inertia of the simulated particles.

- **`g`**:
  Gravity acceleration vector. A value of `[0, 0, 1]` means gravity acts along the Z-axis.

- **`substep_dt`**:
  Simulation time step size. Smaller values increase simulation accuracy but require more computational resources.

- **`n_grid`**:
  MPM (Material Point Method) grid size. A higher value increases the resolution of the simulation but also increases computational cost.

- **`yield_stress`**:
  Yield stress, which determines the stress level at which the material begins to deform plastically. A high value like `1000000` indicates a very stiff material.

- **`friction`**: 
  Friction coefficient, which affects interactions between particles and boundaries. A value of `0.1` indicates low friction.

- **`softening`**:
  Softening parameter, which controls how much the material softens under deformation. A value of `1` means full softening is applied.

- **`hardening`**:
  Hardening parameter, which controls how much the material hardens under deformation. A value of `0` means no hardening is applied.

Please see sample config files under the `config` folder for reference. 
