Structure recovery from single omnidirectional image with distortion-aware learning

Ming Meng, Yi Zhou, Dongshi Zuo, Zhaoxin Li, Zhong Zhou

Abstract: Recovering structures from images with 180∘<math><msup is="true"><mrow is="true"></mrow><mrow is="true"><mo is="true">∘</mo></mrow></msup></math> or 360∘<math><msup is="true"><mrow is="true"></mrow><mrow is="true"><mo is="true">∘</mo></mrow></msup></math> FoV is pivotal in computer vision and computational photography, particularly for VR/AR/MR and autonomous robotics applications. Due to varying distortions and the complexity of indoor scenes, recovering flexible structures from a single image is challenging. We introduce OmniSRNet, a comprehensive deep learning framework that merges distortion-aware learning with bidirectional LSTM. Utilizing a curated dataset with optimized panorama and expanded fisheye images, our framework features a distortion-aware module (DAM) for extracting features and a horizontal and vertical step module (HVSM) of LSTM for contextual predictions. OmniSRNet excels in applications such as VR-based house viewing and MR-based video surveillance, achieving leading results on cuboid and non-cuboid datasets. The code and dataset can be accessed at https://github.com/mmlph/OmniSRNet/.