Image inpainting with edge-guided learnable bidirectional attention maps

Dongsheng Wang, Chaohao Xie, Shaohui Liu, Zhenxing Niu, Wangmeng Zuo

Abstract:

For image inpainting, the convolutional neural networks (CNN) in previous methods often adopt standard convolutional
operator, which treats valid pixels and holes indistinguishably. As a result, they are limited in handling irregular holes and tend to
produce color-discrepant and blurry inpainting result. Partial convolution (PConv) copes with this issue by conducting masked
convolution and feature re-normalization conditioned only on valid pixels, but the mask-updating is handcrafted and independent with
image structural information. In this paper, we present an edge-guided learnable bidirectional attention map (Edge-LBAM) for
improving image inpainting of irregular holes with several distinct merits. Instead of using a hard 0-1 mask, a learnable attention map
module is introduced for learning feature re-normalization and mask-updating in an end-to-end manner. Learnable reverse attention
maps are further proposed in the decoder for emphasizing on filling in unknown pixels instead of reconstructing all pixels. Motivated by
that the filling-in order is crucial to inpainting results and largely depends on image structures in exemplar-based methods, we further
suggest a multi-scale edge completion network to predict coherent edges. Our Edge-LBAM method contains dual procedures,
including structure-aware mask-updating guided by predict edges and attention maps generated by masks for feature re-normalization.
Extensive experiments show that our Edge-LBAM is effective in generating coherent image structures and preventing color
discrepancy and blurriness, and performs favorably against the state-of-the-art methods in terms of qualitative metrics and visual
quality. The source code and pre-trained models are available at https://github.com/wds1998/Edge-LBAM.