Controlling Neural Network Generalization via Constraint-Guided Weight Transformations

TMLR Paper9335 Authors

Abstract:

Despite the success of neural networks (NN), models often reach a plateau during training where they converge to a suboptimal region. In these cases, standard gradient-based optimization often fails to escape or recover, leading to overfitting. We show that generalization can be improved by deliberately perturbing a converged model in a constraint-guided, minimal way, and resuming training. To that end, we present Controlled Misclassification (CMC), a framework that identifies a small subset of training points whose predicted labels are intentionally flipped through minimal weight perturbations. Our approach uses mixed-integer linear programming (MILP), to ensure that model changes are minimal, while enforcing the desired label changes and preserving the model’s overall structure. The key insight is that targeted, constraint-guided perturbations push the model out of sharp or overfitted regions of the loss landscape. When training is resumed from this modified state, the model converges to solutions with improved generalization. We evaluate our approach on 10 multiclass image datasets and 5 binary tabular datasets; we show that CMC improves test accuracy by up to 2.8%. By using constraint optimization for generalization, our method enables more precise and interpretable model edits than gradient-based fine-tuning,
offering a verifiable way to enhance performance.

Submission Type: Regular submission (no more than 12 pages of main content)

Previous TMLR Submission Url: https://openreview.net/forum?id=XNNKUboAiV

Changes Since Last Submission: The submission was desk-rejected for using the wrong stylefile. The current submission uses the correct stylefile.

Assigned Action Editor: ~Qing_Qu2

Submission Number: 9335