Neuro-Symbolic Representations of 3D Scenes using Universal Scene Description Language

M Shifat Hossain, Md Rubel Ahmed, Laura Pullum, Sumit Jha, Rickard Ewetz

Keywords: NeuroSymbolicAI, USDL, 3D

TL;DR: Using USD language for 3D object representation in NeuroSymbolicAI.

Abstract: Developing efficient and expressive representations of 3D scenes is a pivotal problem within 3D computer vision. The state-of-the-art approach is based on utilizing 3D point clouds, which is inefficient in data utilization. In this paper, we propose a neuro-symbolic approach leveraging the Universal Scene Description (USD) language. The approach is based on representing 3D scenes using a combination of known objects (symbolic) and 3D point clouds (neural) for the background. We also propose a framework called neuro-symbolic conversion (NSC) for automatically converting 3D scenes into the proposed neuro-symbolic representation. The NSC framework first locates candidate objects in the 3D point cloud representation. Next, the objects are substituted with their compact symbolic representation while considering translations and rotations. The correctness of the substitution is verified by rendering the neuro-symbolic representation and comparing the visual similarity with the original point cloud representation (or RGB-D view). The experimental results demonstrate that our framework is highly accurate in object identification and objection substitution. The neuro-symbolic representations are expected to be useful for downstream tasks such as entity identification, activity recognition, and object tracking.

Submission Number: 21