Investigating Psychometric Predictive Power of Syntactic Attention

Ryo Yoshida, Yushi Sugimoto, Yohei Oseki

Keywords: attention, syntactic language models, psychometric predictive power

Abstract: In recent computational psycholinguistics, Merkx and Frank (2021) showed that surprisals from Transformers demonstrate a closer fit to measures of human reading effort than those from Recurrent Neural Networks (RNNs), suggesting that Transformers may capture the cue-based retrieval-like operations in human sentence processing. On the other hand, explicit incorporation of syntactic structures has been shown to improve LMs' predictive power for human cognitive load---for example, Hale et al. (2018) demonstrated that Recurrent Neural Network Grammars (RNNGs), which integrate RNNs with explicit syntactic structures, account for aspects of human brain activity that vanilla RNNs cannot. In this paper, we test the psychometric predictive power of Composition Attention Grammars (CAGs), the integration of Transformers with explicit syntactic structures, to investigate whether they can provide better fit to human gaze durations than vanilla Transformers and RNNGs by capturing cue-based retrieval-like operations on syntactic structures, which could potentially be involved in human sentence processing. The results of our controlled experiment demonstrate that surprisals from CAGs outperformed those from Transformers and RNNGs, suggesting that syntactic attention in CAGs may serve as a mechanistic implementation of human retrieval from syntactically-constructed memory representations.

Submission Number: 167