The Evolved Transformer

David So, Quoc Le, Chen Liang

Abstract: Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search (NAS) has begun to outperform human-designed models....