Sparse Hyperbolic Representation Learning

Atsushi Suzuki, Atsushi Nitanda, jing wang, Feng Tian, Kenji Yamanishi

Keywords: hyperbolic space, Riemannian manifold, sparse learning, sparse regularization, iterative shrinkage-thresholding algorithm, Riemannian gradient descent

TL;DR: The paper proposes sparse regularization for hyperbolic-space-based machine learning.

Abstract: Reducing the space complexity of representations while minimizing the loss of information makes data science procedures computationally efficient. For the entities with the tree structure, hyperbolic-space-based representation learning (HSBRL) has successfully reduced the space complexity of representations by using low-dimensional space. Nevertheless, it has not minimized the space complexity of each representation since it has used the same dimension for all representations and has not selected the best dimension for each representation. Hence, this paper aims to minimize representations' space complexity for HSBRL. For minimizing each representation's space complexity, sparse learning has been effective in the context of linear-space-based machine learning; however, no sparse learning has been proposed for HSBRL. It is non-trivial to propose sparse learning for HSBRL because (i) sparse learning methods designed for linear space cause non-uniform sparseness in hyperbolic space, and (ii) existing Riemannian gradient descent methods fail to obtain sparse representations owing to an oscillation problem. This paper, for the first time, establishes a sparse learning scheme for hyperbolic space, overcoming the above issues with our novel sparse regularization term and optimization algorithm. Our regularization term achieves uniform sparseness since it is defined based on geometric distance from subspaces inducing sparsity. Our optimization algorithm successfully obtains sparse representations, avoiding the oscillation problem by realizing the shrinkage-thresholding idea in a general Riemannian manifold. Numerical experiments demonstrate that our scheme can obtain sparse representations with smaller information loss than traditional methods, successfully avoiding the oscillation problem.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics

Submission Guidelines: Yes

Please Choose The Closest Area That Your Submission Falls Into: Deep Learning and representational learning