Beyond Layout Embedding: Layout Attention with Gaussian Biases for Structured Document Understanding
Submission Type: Regular Long Paper
Submission Track: Efficient Methods for NLP
Submission Track 2: Speech and Multimodality
Keywords: Structured Document Understanding, Layout Attention, Spatial Relationships, Polar Coordinates
TL;DR: We propose a novel layout attention with Gaussian biases in transformer for structured document understanding, that modifies self-attention by the bias reflecting spatial relationships via Gaussian kernels.
Abstract: Effectively encoding layout information is a central problem in structured document understanding. Most existing methods rely heavily on millions of trainable parameters to learn the layout features of each word from Cartesian coordinates. However, two unresolved questions remain: (1) Is the Cartesian coordinate system the optimal choice for layout modeling? (2) Are massive learnable parameters truly necessary for layout representation? In this paper, we address these questions by proposing Layout Attention with Gaussian Biases (LAGaBi): Firstly, we find that polar coordinates provide a superior choice over Cartesian coordinates as they offer a measurement of both distance and angle between word pairs, capturing relative positions more effectively. Furthermore, by feeding the distances and angles into 2-D Gaussian kernels, we model intuitive inductive layout biases, i.e., the words closer within a document should receive more attention, which will act as the attention biases to revise the textual attention distribution. LAGaBi is model-agnostic and language-independent, which can be applied to a range of transformer-based models, such as the text pre-training models from the BERT series and the LayoutLM series that incorporate visual features. Experimental results on three widely used benchmarks demonstrate that, despite reducing the number of layout parameters from millions to 48, LAGaBi achieves competitive or even superior performance.
Submission Number: 5027
Loading