Towards Practical Alternating Least-Squares for CCA

Zhiqiang Xu, Ping Li

Abstract: Alternating least-squares (ALS) is a simple yet effective solver for canonical correlation analysis (CCA). In terms of ease of use, it is arguably the first choice for practitioners. Despite recently proposed fast stochastic variants, however, the performance still remains unsatisfactory. In this work, we propose the truly and inexact alternating least-squares (TALS). Instead of approximately solving two independent linear systems, in each iteration, it simply solves two coupled linear systems of half the size. It turns out that the coupling is able to bring significant performance improvements in practice. Moreover, inspired by the accelerated power method, on top of TALS, we propose the faster alternating least-squares (FastTALS) where momentum terms are introduced into the update equations. Both algorithms enjoy linear convergence. To be more practical, we put forward an adaptive version of FastTALS, named AdaFastTALS, to avoid tuning the momentum parameter. It is as easy-to-use as ALS while retaining advantages of FastTALS. Experiments on real datasets demonstrate the empirical superiority of the proposed algorithms to recent variants of ALS.

CMT Num: 8364