Unveiling Transformer Perception by Exploring Input Manifolds

Alessandro Benfenati, Alfio Ferrara, Alessio Marta, Davide Riva, Elisabetta Rocchetti

Keywords: transformers interpretability, input space exploration, geometric deep learning

TL;DR: This paper introduces a general method for the exploration of equivalence classes in the input space of Transformer models.

Abstract:

This paper introduces a general method for the exploration of equivalence classes in the input space of
Transformer models. The proposed approach is based on sound mathematical theory which describes the internal layers of a Transformer architecture as sequential deformations of the input manifold. Using eigendecomposition of the pullback of the distance metric defined on the output space through the Jacobian of the model, we are able to reconstruct equivalence classes in the input space and navigate across them. We illustrate how this method can be used as a powerful tool for investigating how a Transformer sees the input space, facilitating local and task-agnostic explainability in Computer Vision and Natural Language Processing tasks.

Primary Area: Interpretability and explainability

Submission Number: 11093