Efficient Structure-Aware 3D Gaussians via Lightweight Information Shaping

Yunchao Zhang, Guandao Yang, Leonidas Guibas, Yanchao Yang

Keywords: Mutual Information Maximization; 3D Reconstruction; 3D Editing

Abstract:

3D Gaussians,
as an explicit scene representation,
typically involve
thousands to millions of elements per scene.
This makes it
challenging to control
the scene in ways
that reflect the underlying semantics,
where the number of independent entities
is typically much smaller.
Especially,
if one wants to animate or edit objects in the scene,
as this requires coordination among the many Gaussians
involved in representing each object.
To address this issue,
we develop a mutual information shaping technique
that enforces resonance and coordination
between correlated Gaussians
via a Gaussian attribute decoding network.
Such correlations can be learned
from putative 2D object masks in different views.
By approximating the
mutual information with
the gradients concerning the network parameters,
our method ensures consistency
between scene elements
and enables efficient scene editing
by operating on network parameters rather than massive Gaussians.
In particular,
we develop an effective contrastive learning pipeline
with lightweight optimization to shape the attribute decoding network,
while ensuring that the shaping (consistency) is maintained during continuous edits,
avoiding re-shaping after parameter changes.
Notably,
our training only touches
a small fraction of all Gaussians in the scene
yet attains the desired correlated behavior
according to the underlying scene structure.
The proposed technique
is evaluated on challenging scenes and demonstrates significant performance improvements
in 3D object segmentation and promoting scene interactions,
while inducing
low computation and memory requirements.
Our code and trained models
will be made available.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4693