1.78mJ/Frame 373fps 3D GS Processor Based on Shape-Aware Hybrid Architecture Using Earlier Computation Skipping and Gaussian Cache Scheduler

Xiaoyu Feng, Hedi Wang, Chen Tang, Tongda Wu, Huazhong Yang, Yongpan Liu

Abstract: 3D rendering plays a crucial role in emerging applications like virtual reality and embodied AI. Unlike traditional Neural Radiance Fields (NeRF) [1], the novel 3D Gaussian Splatting approach (3D GS) [2] circumvents NeRF's frequent sampling and intensive network inference. Hence, it demonstrates substantial accuracy and frame rate advantages on high-performance GPUs. However, the variability in Gaussian distribution shapes poses significant challenges on resource-constrained edge devices, as in Fig. 2.6.1. For example, 3D GS achieves only 6.4fps on edge Xavier NX. First, a complicated projection + rasterization dataflow is required to render the vast and diverse Gaussian distributions. Consequently, a 3D GS processor demands more kinds of operators and higher bitwidth than NeRF or networks. Additionally, the diversity in shapes leads to numerous input-dependent inefficient computations, escalating power consumption and latency. Lastly, a considerable off-chip memory footprint is required for storing the parameters of millions of Gaussians, particularly the Spherical Harmonics (SH) coefficients. The frequent irregular access patterns further exacerbate the off-chip transfer challenge.