Improving Transformer Optimization Through Better Initialization

Xiao Shi Huang, Felipe Pérez, Jimmy Ba, Maksims Volkovs

Abstract: The Transformer architecture has achieved considerable success recently; the key component of the Transformer is the attention layer that enables the model to focus on important regions within an i...