Multimodal 3D Monitoring and Visual Analytics via Dynamic Frequency Residual Splatting

YONGFENG SHAN, Christy Jie Liang, Daming Luo, Chenxuan Zhou, Xiaoru Yuan

Abstract: Multimodal packet streams, including video synchronized with telemetry, pose estimates, and runtime events, increasingly support real-time monitoring in robotics, digital twins, and sensor networks. This paper presents Coherent Frequency Packet Splatting for Dynamics (CFPS-DYN), a visual analytics methodology that computes an Explainable Frequency Heatmap through a tiled, phase-sensitive spectral residual between the incoming observation and the current prediction. The framework evaluates pixel residual, FFT amplitude, phase coherence, and packet-based diagnostic representations under both previous-frame proxy settings and real 3D Gaussian Splatting renders. The results show that proxy evaluation can create an oracle advantage, while frequency-domain packet energy remains predictive of reconstruction error under real 3DGS renders. A linked-view visual analytics interface exposes heatmap overlays and timeline prioritization to support monitoring and intervention.