Breaking the Softmax Bottleneck via Learnable Monotonic Pointwise Non-linearities

Octavian Ganea, Sylvain Gelly, Gary Bécigneul, Aliaksei Severyn

Abstract: The Softmax function on top of a final linear layer is the de facto method to output probability distributions in neural networks. In many applications such as language models or text generation, t...