DMesh: A Differentiable Mesh Representation

Submission ID: 4957

Overview


"In this work, we present DMesh, a differentiable representation for general 3D mesh."

Point cloud of DMesh (Left) and extracted mesh (Right) during optimization.

DMesh is an explicit shape representation that encodes every information into point cloud.
Then, we determine existence probability of faces based on the point cloud to extract mesh.
Therefore, we optimize the point attributes to reconstruct the target 3D mesh.

Point cloud reconstruction (Left) / First (Middle) and last (Right) epoch of multi-view image reconstruction.

DMesh handles mesh connectivity in a differentiable manner.
Therefore, it admits dynamic topology change.

Open surfaces (DeepFashion3d)
Closed surfaces (Thingi32)
Mixed surfaces (Objaverse)
Geometry of diverse topology

DMesh is very general representation, which can handle closed and open surfaces together.

Experiment: Mesh to DMesh

Since DMesh handles mesh connectivity in a differentiable manner, it can optimize point attributes to recover the ground truth connectivity as much as possible, with only small perturbations to the vertices of the mesh.




Ground truth mesh (Left) and DMesh (Right) during optimization.
Even though there are slight perturbations to the vertex positions, DMesh can restore 99% of connectivity of the ground truth mesh, while only having less than 1% false positive ratio. The periodical flickering of DMesh is induced by additional point insertion.

Experiment: Point Cloud Reconstruction


Here we assume each point cloud is comprised of 100K points. From there, we sample 10K points to initialize DMesh. Then, we optimize DMesh by minimizing the expected Chamfer Distance loss to the given point cloud.

Point cloud of DMesh (Left) and extracted mesh (Right) during optimization.
In the point cloud rendering, the color of each point represents its real value.
Note that some points disappear because they lose weights and thus discarded during optimization.
Extracted mesh without edges (Left) and with edges (Right) during optimization.
Since we already have sample points, we can use their subset to initialize our mesh.
Therefore, it converges fast to the target shape in several optimization steps.
Note that connectivity keeps changing mainly due to additional regularizations.

Experiment: Multi-view Image Reconstruction


Here we assume that we are given diffuse and depth rendering of ground truth mesh from 64 viewpoints. We use differentiable renderer to render the object, and optimize mesh based on L1 loss to the given images. Unlike point cloud reconstruction, we start optimization from random state, as we do not have sample points. We take a coarse-to-fine approach, and optimize for 4 epochs. At the start of each epoch, we sample points from the previous mesh and use them to initialize the mesh. The number of sample points increase to get better, fine-grained results.

Epoch 0: Start optimization from 8000 regularly distributed points.
Epoch 1: Sample 1000 points from previous mesh to initialize mesh, and optimize.
Epoch 2: Sample 3000 points from previous mesh to initialize mesh, and optimize.
Epoch 3: Sample 10000 points from previous mesh to initialize mesh, and optimize.