Unpaired Panoramic Image-to-Image Translation Leveraging Pinhole Images

Soohyun Kim, Junho Kim, Taekyung Kim, Hwan Heo, Seungryong Kim, Jiyoung Lee, Jin-Hwa Kim

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Image-to-Image-Translation, Image Synthesis, Panorama

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

In this paper, we tackle the challenging task of unpaired panoramic Image-to-Image translation (Pano-I2I) for the first time. This task aims to learn a mapping between unpaired panoramic source and non-panoramic target domains to modify naive 360 panoramic images. However, it is difficult to use existing I2I methods due to two main challenges. Firstly, panoramas inherit geometric distortions, which pose challenges for methods based on a narrow field-of-view. Secondly, accessing panoramic datasets encompassing various weather conditions or times for training purposes is severely limited. To address these challenges, we propose a novel I2I model tailored for mitigating panoramic distortion that harnesses readily obtainable pinhole images as the target domain for training. We introduce a versatile encoder and distortion-free discrimination that efficiently bridges the large domain gap between panoramic and pinhole images by simultaneously encoding them in a consolidated structure. It allows our model to learn style mappings while overcoming significant geometric differences between the source and target domains. Moreover, we carefully design spherical position embedding, sphere-based rotation augmentation, and its ensemble to address the discontinuities at the panorama edges. Comprehensive experiments verify that our framework effectively translates panoramic street views from daytime to night, rainy, and twilight scenes by referring to the holistic style of pinhole data.
Our method also shows superior results in both maintaining structural coherence and rotation equivariance, clearly surpassing the existing I2I methods in qualitative and quantitative results.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 581