Panoptically guided Image Inpainting with Image-level and Object-level Semantic Discriminators

Haitian Zheng, Zhe Lin, Jingwan Lu, Scott Cohen, Eli Shechtman, Jianming Zhang, Sohrab Amirghodsi, Qing Liu, Jiebo Luo

Keywords: Generative model, image inpainting, image manipulation

TL;DR: Guided Image Inpainting and image inpainting with a novel discriminator design.

Abstract: Recent image inpainting methods have made great progress. However, the existing approaches often struggle to hallucinate realistic object instances in natural scenes. Such a limitation is partially due to the lack of semantic-level constraints inside the hole as well as the lack of a mechanism to enforce the realism of local objects. To tackle the challenging object inpainting task, we propose a new panoptically guided image inpainting task that leverages a panoptic segmentation map to guide the completion of object instances. To enforce the realism of the generated objects, we propose a semantic discriminator that leverages pretrained visual features to improve the generated semantics. Furthermore, we propose object-level discriminators that take aligned instances as input to enforce the realism of individual objects. Experiments on the large-scale Places2 dataset demonstrate the significant improvement by our method on object completion, verified in both quantitative and qualitative evaluation. Furthermore, our framework is flexible and can be generalized to other inpainting tasks including segmentation-guided inpainting, edge-guided inpainting, as well as standard image inpainting without guidance. Consequently, our approach achieves new state-of-the-art performance on the various inpainting tasks and impressive results on object completion.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Generative models