Analysis of the Attention in Tabular Language ModelsDownload PDF

Published: 21 Oct 2022, Last Modified: 16 May 2023TRL @ NeurIPS 2022 OralReaders: Everyone
Keywords: attention, tabular language models
Abstract: Recent transformer-based models for learning table representation have reported state-of-the-art results for different tasks such as table understanding, question answering and semantic parsing. The various proposed models use different architectures, specifically different attention mechanisms. In this paper, we analyze and compare the attention mechanisms used by two different tabular language models. By visualizing the attention maps of the models, we shed a light on the different patterns that the models exhibit. With our analysis on the aggregate attention over two tabular datasets, we provide insights which might help towards building more efficient models tailored for table representation learning.
Slides: pdf
0 Replies