Adaptive Sensitivity Analysis for Robust Augmentation against Natural Corruptions in Image Segmentation

Laura Yu Zheng, Wenjie Wei, Tony Wu, Jacob Clements, Shreelekha Revankar, Andre Harrison, Yu Shen, Ming Lin

TL;DR: We estimate sensitivity curves for different perturbation functions at intervals during train-time, and sample these curves for sensitivity-based augmentation.

Abstract: Achieving robustness in image segmentation models is challenging due to the fine-grained nature of pixel-level classification. These models, which are crucial for many real-time perception applications, particularly struggle when faced with natural corruptions in the wild for autonomous systems. While sensitivity analysis can help us understand how input variables influence model outputs, its application to natural and uncontrollable corruptions in training data is computationally expensive. In this work, we present an adaptive, sensitivity-guided augmentation method to enhance robustness against natural corruptions. Our sensitivity analysis on average runs 10 times faster and requires about 200 times less storage than previous sensitivity analysis, enabling practical, on-the-fly estimation during training for a model-free augmentation policy. With minimal fine-tuning, our sensitivity-guided augmentation method achieves improved robustness on both real-world and synthetic datasets compared to state-of-the-art data augmentation techniques in image segmentation.

Lay Summary: Edge cases like adverse weather are difficult to account for in perception tasks like segmentation, especially in real-time applications like robotics. We propose a solution via data augmentation, where we sample "evenly difficult" synthetic perturbation intensities with respect to model performance. Our approach is similar to how students may choose to practice on more difficult practice questions and less on easier questions across different sections, relative to their current knowledge. Our approach helps improve convergence and generalization of segmentation models, especially in unseen real world scenarios.

Application-Driven Machine Learning: This submission is on Application-Driven Machine Learning.

Link To Code: https://github.com/laurayuzheng/SensAug

Primary Area: Applications->Computer Vision

Keywords: image segmentation, augmentation, robustness, heuristic, natural corruptions, adverse weather

Submission Number: 7990