Global Boundary Refinement for Semantic Segmentation via Optimal Transport

Feng Dai, Shuaibin Zhang, Hao Liu, Yike Ma, Qiang Zhao

Abstract: Semantic boundary prediction is an important but challenging problem in semantic segmentation. Previous methods usually regard the boundary prediction as a pure classification problem using binary cross-entropy loss, which does not consider the spatial distance between the predicted boundary and ground truth. To address this issue, we formulate semantic boundary prediction as the optimal transport problem where the minimum transport cost reflects the spatial distance. Specifically, the predicted boundary and the true boundary are formulated as source distribution and target distribution, respectively. Then, we calculate the spatial distance between source pixels and target pixels as the transport cost matrix. Finally, we solve the transport problem and obtain the minimum transport cost for effective boundary supervision. Additionally, our method does not explicitly fuse boundary and semantic feature, which only provides supervision for the predicted boundary during the training. And the boundary branch can be discarded during the inference, thus will not bring extra computations and parameters. Experiments show that our method can significantly and consistently improve the segmentation accuracy of various models on public segmentation benchmarks: Cityscapes and CamVid.