Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification

Sicong Li, Qianqian Xu, Zhiyong Yang, Zitai Wang, Linchao Zhang, Xiaochun Cao, Qingming Huang

Abstract: Real-world datasets often follow a long-tailed distribution, making generalization to tail classes difficult. Recent methods resorted to long-tail variants of Sharpness-Aware Minimization (SAM), such as ImbSAM and CC-SAM, to improve generalization by flattening the loss landscape. However, these attempts face a trade-off between computational efficiency and control over the loss landscape. On the one hand, ImbSAM is efficient but offers only coarse control as it excludes head classes from the SAM process. On the other hand, CC-SAM provides fine-grained control through class-dependent perturbations but at the cost of efficiency due to multiple backpropagations. Seeing this dilemma, we introduce Focal-SAM, which assigns different penalties to class-wise sharpness, achieving fine-grained control without extra backpropagations, thus maintaining efficiency. Furthermore, we theoretically analyze Focal-SAM's generalization ability and derive a sharper generalization bound. Extensive experiments on both traditional and foundation models validate the effectiveness of Focal-SAM.

Lay Summary:

In real-world image datasets, some categories such as "cats" have thousands of examples, while others like rare animals may have only a few. This imbalance, known as a long-tailed distribution, makes it difficult for AI models to learn rare categories effectively.

Recent methods have tried to address this using a technique called Sharpness-Aware Minimization (SAM), which helps models generalize better by avoiding sharp regions in the loss landscape. However, existing approaches either place too much emphasis on rare classes, which harms performance on common ones, or they require intensive computation that slows down training.

This paper proposes Focal-SAM, a new method that improves model performance across both common and rare categories. Focal-SAM efficiently adjusts sharpness penalty applied to each class, giving more attention to rare ones without ignoring the common ones.

Experiments on several benchmark datasets show that Focal-SAM consistently outperforms other methods. This helps AI systems make better predictions across a wide range of categories, including those with limited data.

Primary Area: General Machine Learning->Supervised Learning

Keywords: Long-tailed Learning

Submission Number: 922