The Double Helix inside the NLP Transformer

Anonymous

TL;DR: This study presents a framework for investigating information processing in NLP Transformers and obtains novel results in attention mechanism, positional encoding and syntactic clustering.

Abstract: This study introduces a novel framework for exploring the information processing within NLP Transformers. We categorize information into four distinct layers: positional, syntactic, semantic, and contextual. Challenging the conventional integration of positional data into semantic embeddings, we propose a more effective “Linear-and-Add” method. Our analysis uncovers an intrinsic separation of positional elements in deeper layers, revealing that these components form a helix-like pattern in both encoder and decoder stages. Notably, our approach enables the identification of Part-of-Speech (PoS) clusters within conceptual dimensions. These insights offer a new perspective on information processing in the complex architecture of NLP Transformers, potentially guiding future developments in the field.

Paper Type: long

Research Area: Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability

Languages Studied: Portuguese, English