Boosting as Frank-Wolfe

Ryotaro Mitsuboshi, Kohei Hatano, Eiji Takimoto

Keywords: Boosting, Frank-Wolfe, Soft margin optimization

TL;DR: First, we give a unified view of boosting for soft margin optimization and then we propose a new boosting scheme that can incorporate any heuristics without losing the convergence guarantee.

Abstract:

Some boosting algorithms, such as LPBoost, ERLPBoost, and C-ERLPBoost, aim to solve the soft margin optimization problem with the $\ell_1$-norm regularization.
LPBoost rapidly converges to an $\epsilon$-approximate solution in practice,
but it is known to take $\Omega(m)$ iterations in the worst case, where $m$ is the sample size.
On the other hand, ERLPBoost and C-ERLPBoost are guaranteed to converge to an $\epsilon$-approximate solution in $O(\frac{1}{\epsilon^2} \ln \frac{m}{\nu})$ iterations. However, the computation per iteration is very high compared to LPBoost.

To address this issue, we propose a generic boosting scheme that combines the Frank-Wolfe algorithm and any secondary algorithm
and switches one to the other iteratively. We show that the scheme retains the same convergence guarantee
as ERLPBoost and C-ERLPBoost. One can incorporate any secondary algorithm to improve in practice.
This scheme comes from a unified view of boosting algorithms for soft margin optimization.
More specifically, we show that LPBoost, ERLPBoost, and C-ERLPBoost are instances
of the Frank-Wolfe algorithm. In experiments on real datasets,
one of the instances of our scheme exploits the better updates of the second algorithm
and performs comparably with LPBoost.