Abelian Neural Networks

Kenshin Abe, Takanori Maehara, Issei Sato

Keywords: algebra, Abelian group, word analogy, invertible neural networks, permutation invariant, size generalization

Abstract: In several domains such as natural language processing, it has been empirically reported that simple addition and subtraction in a somehow learned embedding space capture analogical relations. However, there is no guarantee that such relation holds for a new embedding space acquired by some training strategies. To tackle this issue, we propose to explicitly model analogical structure with an Abelian group. We construct an Abelian group network using invertible neural networks and show its universal approximation property. In experiments, our model successfully learns to capture word analogies from word2vec representations and shows better performance than other learning-based strategies. As a byproduct of modeling Abelian group operations, we furthermore obtain its natural extension to permutation invariant models with theoretical size-generalization capability.

One-sentence Summary: We propose a neural network architecture for modeling Abelian groups.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2102.12232/code)