Making Reliable and Flexible Decisions in Long-tailed Classification
Abstract: Long-tailed classification is challenging due to its heavy imbalance in class probabilities. While existing methods often focus on overall accuracy or accuracy for tail classes, they overlook a critical aspect: certain types of errors can carry greater risks than others in real-world long-tailed problems. For example, misclassifying patients (a tail class) as healthy individuals (a head class) entails far more serious consequences than the reverse scenario. To address this critical issue, we introduce Making Reliable and Flexible Decisions in Long-tailed Classification (RF-DLC), a novel framework aimed at reliable predictions in long-tailed problems. Leveraging Bayesian Decision Theory, we introduce an integrated gain to seamlessly combine long-tailed data distributions and the decision-making procedure. We further propose an efficient variational optimization strategy for the decision risk objective. Our method adapts readily to diverse utility matrices, which can be designed for specific tasks, ensuring its flexibility for different problem settings. In empirical evaluation, we design a new metric, False Head Rate, to quantify tail-sensitivity risk, along with comprehensive experiments on multiple real-world tasks, including large-scale image classification and uncertainty quantification, to demonstrate the reliability and flexibility of our method.
Submission Length: Regular submission (no more than 12 pages of main content)
Previous TMLR Submission Url: https://openreview.net/forum?id=6xwqONp6KK
Changes Since Last Submission: We have made the following changes to our paper based on the suggestions from previous reviewers and the editor:
1. We improve the experimental results of our method on iNaturalist by further tuning the learning rates. Our method now outperforms all baselines and demonstrates consistent improvements across all datasets.
2. We updated the related works, adding comparisons with cost-sensitive classification. We highlighted our novelty in enabling optimal decision-making in long-tailed problems, which has not been explored by previous cost-sensitive methods.
3. We added detailed ablation studies on the design of utility matrices to the appendix, highlighting the robustness of utility designs to the overall performances.
4. We corrected typos and modified the paper structure for clarity, according to the reviewers’ suggestions.
Assigned Action Editor: ~Aditya_Menon1
Submission Number: 3556
Loading