A Second Order Majorant Algorithm for Nonnegative Matrix Factorization

TMLR Paper5917 Authors

Abstract:

Nonnegative Matrix Factorization (NMF) is a fundamental tool in unsupervised learning, widely used for tasks such as dimensionality reduction, feature extraction, representation learning, and topic modeling. Many algorithms have been developed for NMF, including the well-known Multiplicative Updates (MU) algorithm, which belongs to a broader class of majorization-minimization techniques.
In this work, we introduce a general second-order optimization framework for NMF under both quadratic and $\beta$-divergence loss functions.
This approach, called Second-Order Majorant (SOM), constructs a local quadratic majorization of the loss function by majorizing its elementwise nonnegative Hessian matrix.
It includes MU as a special case, while enabling faster variants. In particular, we propose mSOM, a new algorithm within this class that leverages a tighter local approximation to accelerate convergence. We provide a convergence analysis, showing linear convergence for individual factor updates and global convergence to a stationary point for the alternating version, AmSOM. Numerical experiments on both synthetic and real datasets demonstrate that AmSOM is a promising algorithm for NMF.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission:

Here is a list of modifications on the manuscript:

- Experiments:
- Adding clarifications on the experimental claims: mSOM is
competitive with the state-of-the-art for Frobenius norm (both
sparse and dense), performs favorably wrt MU for KL in the
synthetic experiments, but struggles with realitic dataset (the
audio experiment still to be further understood when it fails in
that setting; this application is reported as special for
algorithmic performance also by e.g. Hien and
Gillis.[@hien2021algorithms]).
- Adding experiments to support these claims: 1. HSI with the
KL-divergence, which a dense, large-scale experiment for KL 2.
Audio with the Frobenius norm, which is a sparse, medium-scale
experiment for the Frobenius norm. 3. Sparse + unbalanced
Frobenius loss simulated experiments. These experiments complete
the simulations already provided and hopefully better highlight
the strengths of the mSOM algorithm.
- Additional information: winner plots (customized performance
plots) and error bands on convergence plots.
- The plots are now larger; they can be further improved (although
this is quite time-consuming) before sending the final
manuscript if required.
- The complexity of mSOM updates is now explicitly written in the
manuscript.
- The link between NNLS solutions, separable constraints and diagonal
Hessian approximations is now more detailed.
- Clarifications around the choice of the Hessian majorant. In
particular, we identify that mSOM approximates explicitly the
Hessian with the $\ell_1$ norm.
- More context around the use of mirror descent for NMF with
$\beta$-divergences.
- The fact that AmSOM is an approximate Alternating Optimization
algorithm is now more clearly stated.
- Various typos were corrected.
- The introduction is now structured with a contributions section
- A sentence about normalization of the columns/rows has been added to
acknowledge this option.

Assigned Action Editor: ~Kejun_Huang1

Submission Number: 5917