Transformers: Statistical Interpretation, Architectures and Applications

Fanfei Meng, Yuxin Wang

Keywords: Transformers, natural language processing, computer vision, speech processesing

TL;DR: Overview of mechanisms of transformer and its application.

Abstract:

Transformers have been widely recognized as powerful tools to analyze multiple tasks due to its state-of art multi-head attention spaces, such as Natural Language Processing (NLP), Computer Vision (CV) and Speech Recognition (SR). Inspired by its abundant designs and strong functions on analyzing input data, I would like to start from the various architectures, further proceed to the investigation on its statistical mechanism and inference and then introduce its applications on dominant tasks. The underlying
statistical mechanisms arouse my interests and intrigue me to investigate it in a higher level, and this surveys will focus on its mathematical foundations and then use the principles to try to analyze the reasons for its excellent performance on many recognition scenarios.