TENSORIZED ATTENTION MODEL

Makoto Nakatsuji, Yasuhiro Fujiwara, Narichika Nomoto, Atsushi Fukayama

Primary Area: representation learning for computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Attention model, Tensorized Transformer, Encoder model

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

In recent years, attention mechanisms have played a crucial role in the
success of Transformer models, as seen in platforms like OpenAI's
ChatGPT. However, these models often struggle to compute attention
weights across various object types, such as 'comments,' 'replies,' and
specific 'subjects,' which naturally express relationships in many
real-world scenarios. This limitation can potentially impact prediction
accuracy.
To overcome this limitation, we introduce the Tensorized Attention Model
(TAM). By leveraging Tucker decomposition, TAM calculates attention
weights across a diverse array of objects and seamlessly integrates them
into Transformer outputs.
We have implemented TAM within the Transformer encoder and have
showcased its effectiveness in response selection tasks. Our model takes
into account relationships based on 'the current context in the
dialogue', 'the entire dialogue history', and 'the subject matter of the
dialogue'. Evaluation using the Reddit dataset across a wide variety of
topics indicates that TAM significantly outperforms existing
Transformer-based methods in terms of accuracy.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2141