Neighborhood Encoding Network for Semantic Segmentation

Xiaotian Lou, Xiaoyu Chen, Lianfa Bai, Jing Han

Abstract: With recent advances of deep neural networks, semantic segmentation algorithms are in rapid development. However, as pixel-level semantic segmentation is often treated as pixel-wise classification task where the neighbor correlation is ignored during inference, the entirety of results is inevitably impaired. In order to increase the correlation ship among the pixels in neural networks, we propose neighborhood encoding network (NENet) to extract the semantics and encode the pixel-level correlation of inputs in a backbone network. In NENet, we use neighborhood prediction module (NPM) to decode the pixel-level correlation and get the result. The NPM can also help the backbone network encode the correlation during training phase. We also design a stage-wise training strategy with NPM for correlation transmission, which eases the training process and increases the performance effectively. The structure of NENet can be expanded to other encoder-decoder network. We evaluate the proposed NENet on CamVid and Cityscpaes datasets, and the NENet achieves impressive results.