Exponential Machines

Alexander Novikov, Mikhail Trofimov, Ivan Oseledets

Nov 04, 2016 (modified: Jan 17, 2017) ICLR 2017 conference submission readers: everyone
  • Abstract: Modeling interactions between features improves the performance of machine learning solutions in many domains (e.g. recommender systems or sentiment analysis). In this paper, we introduce Exponential Machines (ExM), a predictor that models all interactions of every order. The key idea is to represent an exponentially large tensor of parameters in a factorized format called Tensor Train (TT). The Tensor Train format regularizes the model and lets you control the number of underlying parameters. To train the model, we develop a stochastic Riemannian optimization procedure, which allows us to fit tensors with 2^160 entries. We show that the model achieves state-of-the-art performance on synthetic data with high-order interactions and that it works on par with high-order factorization machines on a recommender system dataset MovieLens 100K.
  • TL;DR: A supervised machine learning algorithm with a polynomial decision function (like SVM with a polynomial kernel) that models exponentially many polynomial terms by factorizing the tensor of the parameters.
  • Conflicts: phystech.edu, skoltech.ru, bayesgroup.ru, msu.ru, hse.ru
  • Keywords: Supervised Learning, Optimization