FaPN: Feature-aligned Pyramid Network for Dense Image Prediction

Shihua Huang

Abstract:

Recent advancements in deep neural networks have
made remarkable leap-forwards in dense image prediction.
However, the issue of feature alignment remains as neglected by most existing approaches for simplicity. Direct
pixel addition between upsampled and local features leads
to feature maps with misaligned contexts that, in turn, translate to mis-classifications in prediction, especially on object boundaries. In this paper, we propose a feature alignment module that learns transformation offsets of pixels
to contextually align upsampled higher-level features; and
another feature selection module to emphasize the lowerlevel features with rich spatial details. We then integrate
these two modules in a top-down pyramidal architecture
and present the Feature-aligned Pyramid Network (FaPN).
Extensive experimental evaluations on four dense prediction tasks and four datasets have demonstrated the efficacy
of FaPN, yielding an overall improvement of 1.2 - 2.6 points
in AP / mIoU over FPN when paired with Faster / Mask RCNN. In particular, our FaPN achieves the state-of-the-art
of 56.7% mIoU on ADE20K when integrated within MaskFormer.