Open Peer Review. Open Publishing. Open Access. Open Discussion. Open Directory. Open Recommendations. Open API. Open Source.
Efficient Softmax Approximation for GPUs
Édouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, Hervé Jégou
Nov 04, 2016 (modified: Dec 13, 2016)ICLR 2017 conference submissionreaders: everyone
Abstract:We propose an approximate strategy to efficiently train neural network based language models over very large vocabularies. Our approach, called adaptive softmax, circumvents the linear dependency on the vocabulary size by exploiting the unbalanced word distribution to form clusters that explicitly minimize the expectation of computational complexity. Our approach further reduces the computational cost by exploiting the specificities of modern architectures and matrix-matrix vector operations, making it particularly suited for graphical processing units. Our experiments carried out on standard benchmarks, such as EuroParl and One Billion Word, show that our approach brings a large gain in efficiency over standard approximations while achieving an accuracy close to that of the full softmax.
Keywords:Natural language processing
Conflicts:fb.com, kaust.edu.sa, columbia.edu
Enter your feedback below and we'll get back to you as soon as possible.