Reweighting Improves Conditional Risk Bounds
Abstract: In this work, we study the weighted empirical risk minimization (weighted ERM) schema, in which an additional data-dependent weight function is incorporated when the empirical risk function is being minimized. We show that under a general ``balanceable" Bernstein condition, one can design a weighted ERM estimator to achieve superior performance in certain sub-regions over the one obtained from standard ERM, and the superiority manifests itself through a data-dependent constant term in the error bound. These sub-regions correspond to large-margin ones in classification settings and low-variance ones in heteroscedastic regression settings, respectively. Our findings are supported by evidence from synthetic data experiments.
Submission Length: Long submission (more than 12 pages of main content)
Changes Since Last Submission: # Outline of Changes
The following major changes have been made based on the comments from the reviewers.
1. Section 3: we have shortened the overview of results, to remove some of those that was briefly touched upon in Section 1. (in response to Reviewer ZpQh)
2. Section 4: we have streamlined theorem statements and added some additional remarks.
3. Section 6: we expanded on the discussion pertaining to the assumption of a well-specified setting
4. Other changes: fixed typographical errors and typos in the proofs in the Appendix.
Assigned Action Editor: ~Nishant_A_Mehta1
Submission Number: 2679
Loading