AdaCubic: An Adaptive Cubic Regularization Optimizer for Deep Learning

TMLR Paper6482 Authors

Abstract:

A novel regularization technique called AdaCubic is proposed that adapts the weight of
the cubic term. The heart of AdaCubic is an auxiliary optimization problem with cubic
constraints that dynamically adjusts the weight of the cubic term in Newton’s cubic regular-
ized method. We utilize Hutchinson’s method to approximate the Hessian matrix, thereby
reducing computation costs. We demonstrate that AdaCubic inherits the cubically regular-
ized Newton method’s local convergence guarantees. Our experiments in Computer Vision,
Natural Language Processing, and Signal Processing tasks demonstrate that AdaCubic out-
performs or competes with several widely used optimizers. Unlike other adaptive algorithms
that require fine-tuning of hyperparameters, AdaCubic is evaluated with a pre-fixed set of
hyperparameters, making it a highly attractive optimizer in situations where fine-tuning is
not feasible. This makes AdaCubic an attractive option for researchers and practitioners
alike. To our knowledge, AdaCubic is the first optimizer to leverage the power of cubic
regularization for large-scale applications. The code of AdaCubic will be publicly released
upon paper acceptance.

Submission Type: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Yi_Zhou2

Submission Number: 6482