3D-GSR: 3D Super-Resolution with Multi-View Adapter and Gaussian Splatting
Keywords: Super-Resolution, 3D Gaussian Splatting, Multi-view, Novel View Synthesis, High-Resolution Novel View Synthesis
TL;DR: 3D-GSR: a 3D super-resolution model with 3D GS that leverages the proposed multi-view adapter to use arbitrary single-image super-resolution model for consistent upscaling of many frames.
Abstract: The rapid advancements in 2D generative models and efficient 3D reconstruction techniques are fueling the growth of the 3D visual generative AI field. In this work, we address the challenge of 3D super-resolution by developing a consistent multi-view super-resolution model and leveraging 3D reconstruction techniques to obtain high-quality 3D models from the ones of lower quality. Our approach is versatile, generalizing to various inputs, including renderings of digital 3D assets and turn-around videos of real-world objects. Its adaptability also allows it to serve as a post-processing step for any existing 3D generative method, regardless of the underlying geometry representation. We introduce 3D-GSR, a model that integrates 2D super-resolution with 3D reconstruction to enhance the 3D generation process. Central to our approach, we propose the Multi-View Super-Resolution (MV-SR) adapter, designed to leverage arbitrary single-image SR model to generate consistent multi-view frames of the same object. We tested our MV-SR technique with four common SR models: R-ESRGAN, SwinIR, HAT-L, and SD-XL based, showing how our adapter increases each model's accuracy in the multi-view scenario. The geometry of the asset is reconstructed using a novel 3D Gaussian Splatting model. Our 3D-GSR model is adept at both geometric transformations and appearance enhancements, producing assets with intricate geometry and exceptional fidelity, excelling in capturing high-fidelity details of the high-frequency components in particular.
Submission Number: 20
Loading