Transformers Can Model Human Hyperprediction in Buzzer Quiz
Abstract: Humans are thought to predict the next words during sentence comprehension, but under unique circumstances, they demonstrate an ability for longer coherent word sequence prediction. In this paper, we investigate whether Transformers can model such hyperprediction observed in humans during sentence processing, specifically in the context of Japanese buzzer quizzes.
We conducted eye-tracking experiments where the participants read the first half of buzzer quiz questions and predicted the second half, while we modeled their reading time using the GPT-2. The results showed that the GPT-2 can partially capture human hyperprediction. When the language model was fine-tuned with quiz questions, the perplexity value decreased. Lower perplexity corresponded to higher psychometric predictive power; however, excessive data for fine-tuning led to a decrease in perplexity and the fine-tuned model exhibited a low psychometric predictive power.
Overall, our findings suggest that a moderate amount of data is required for fine-tuning in order to model human hyperprediction.
Paper Type: Long
Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics
Research Area Keywords: cognitive modeling,computational psycholinguistics,fine-tuning
Contribution Types: NLP engineering experiment
Languages Studied: Japanese
Submission Number: 3391
Loading