Semantic Update as a Predictor of Reading Time: Moving Beyond Word-Level Surprisal

Siddharth Gupta, Alessandro Lopopolo, Milena Rabovsky

Keywords: N400, sentence processing, semantic prediction, sentence gestalt model, eye tracking, reading times

Abstract:

Surprisal has been ubiquitous in testing psycholinguistic hypotheses about lexical predictability and its effects on
reading measures [1]. While surprisal captures predictability at the lexical-level, to predict per-word total fixation
duration (gaze duration) we here test a complementary measure derived from an artificial neural network model
of sentence comprehension that works at the level of sentence meaning: the Sentence Gestalt (SG) model [2].
This measure, called semantic update (SU), calculates how the introduction of each new word in a sentence
updates the SG model’s internal, predictive representation of sentence meaning [3, 4]. Mathematically SU,
as reflected by the arrival of a new word wn, is defined as: SUn = ∑k
i=1 |ai(wn) − ai(wn−1)|, where ai(wn) is
the activation at the i-th unit in the gestalt layer (with k units) as the network (see Fig 1 for the visualisation
of encoder-decoder architecture in SG model) encounters the n-th word in the sentence. Hence, SU is the
absolute error between the layer’s activation before and after encountering a word. SU has previously been
successfully applied to predict the N400 event-related potential (ERP) component [3, 4], the most widely used
ERP component in research on language comprehension [5]. We hypothesize that reading behavior reflects
both lexical-level prediction of individual word forms and the incremental integration of meaning at the sentence
level - operationalised by word surprisal and SU respectively. As each word is encountered, it not only has
a context-dependent predictability but also modifies the evolving representation of sentence meaning. We
therefore expect that reading times reflect both how predictable a word is and how strongly it updates ongoing
semantic representations.
In the current study, we examined the effect of SU as predictor of total word fixation duration (gaze duration)
using an eye-tracking dataset of 46 native English-speaking participants who read 12 English texts [6]. To
predict gaze duration, we fitted a linear mixed-effects model (lmer) that included SU, surprisal derived from
a language model (LM) trained on the same dataset as the SG model, word length, word position, and word
frequency, with random intercepts and slopes for SU and LM-based surprisal, grouped by both participant and
word. We included LM-based surprisal [4] to assess whether the SU provides explanatory power beyond this
information-theoretic predictor.
The results (Table 1) indicate that SU significantly predicts gaze duration (β = 0.03, t = 2.81, p =< 0.001)
beyond the influence of LM-based surprisal, which as expected also has a significant impact (β = 0.02, t =
2.27, p = 0.024). Note-worthily, the effects of the predictors of interest are specific to content words as seen in
lmer models run on the content and functional subsets as well as an interaction model with word category as
the interaction term (Table 2). Besides, we also conducted an ANOVA comparing the first model to a model
excluding SU and found that including SU significantly improves model fit (p < 0.001).
The analysis reveals a significant relationship between gaze duration and the size of semantic updates. These
findings suggest that gaze duration reflects the cognitive effort of updating semantic representations, empha-
sizing the role of sentence meaning-not just lexical-prediction in reading behavior.

Submission Number: 29