When Semantic Segmentation Meets Frequency Aliasing

Linwei Chen, Lin Gu, Ying Fu

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Aliasing; Hard Pixel; Semantic Segmentation

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Despite recent advancements in semantic segmentation, where and what pixels are hard to segment remains largely unexplored.
Existing research only separates an image into easy and hard regions and empirically observes the latter are associated with object boundaries.
In this paper, we conduct a comprehensive analysis of hard pixel errors, categorizing them into three types: false responses, merging mistakes, and displacements.
Our findings reveal a quantitative association between hard pixels and aliasing,
which is distortion caused by the overlapping of frequency components in the Fourier domain during downsampling.
To identify the frequencies responsible for aliasing, we propose using the equivalent sampling rate to calculate the Nyquist frequency, which marks the threshold for aliasing.
Then, we introduce the aliasing score as a metric to quantify the extent of aliasing.
While positively correlated with the proposed aliasing score, three types of hard pixels exhibit different patterns.
Here, we propose two novel de-aliasing filter (DAF) and frequency mixing (FreqMix) modules to alleviate aliasing degradation by accurately removing or adjusting frequencies higher than the Nyquist frequency.
The DAF precisely removes the frequencies responsible for aliasing before downsampling,
while the FreqMix dynamically selects high-frequency components within the encoder block.
Experimental results demonstrate consistent improvements in semantic segmentation and low-light instance segmentation tasks.
The code is at: \url{https://github.com/Linwei-Chen/Seg-Aliasing}.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Primary Area: representation learning for computer vision, audio, language, and other modalities

Submission Number: 661