Revisiting Entropy Rate Constancy in Text

Vivek Verma, Nicholas Tomlin, Dan Klein

Submission Type: Regular Long Paper

Submission Track: Linguistic Theories, Cognitive Modeling, and Psycholinguistics

Keywords: Uniform Information Density, Entropy Rate, Large Language Models, Linguistic Theories

Abstract: The uniform information density (UID) hypothesis states that humans tend to distribute information roughly evenly across an utterance or discourse. Early evidence in support of the UID hypothesis came from Genzel and Charniak (2002), which proposed an entropy rate constancy principle based on the probability of English text under $n$-gram language models. We re-evaluate the claims of Genzel and Charniak (2002) with neural language models, failing to find clear evidence in support of entropy rate constancy. We conduct a range of experiments across datasets, model sizes, and languages and discuss implications for the uniform information density hypothesis and linguistic theories of efficient communication more broadly.

Submission Number: 4210