Resource-Rational Noisy-Channel Language Processing: Testing the Effect of Algorithmic Constraints on Inferences

ACL ARR 2025 May Submission4906 Authors

Abstract: Human language use is robust to errors: comprehenders can and do mentally correct utterances that are implausible or anomalous. How are humans able to solve these problems in real time, picking out alternatives from an unbounded space of options using limited cognitive resources? And can language models trained on next-word prediction for typical language be augmented to handle language anomalies in a human-like way? Using a language model as a prior and an error model to encode likelihoods, we use Sequential Monte Carlo with optional rejuvenation to perform incremental and approximate probabilistic inference over intended sentences and production errors. We demonstrate that the model captures previously established patterns in human sentence processing, and that a trade-off between human-like noisy-channel inferences and computational resources falls out of this model. From a psycholinguistic perspective, our results offer a candidate algorithmic model of rational inference in language processing. From an NLP perspective, our results showcase how to elicit reasoning-like behavior from a relatively small LLM while controlling the amount of computation available during inference. Our model is implemented in the Gen.jl probabilistic programming language, and our code is available at \url{https://osf.io/4zyd5/?view_only=54ebfb788ceb4f139f675130e7161111}.

Paper Type: Long

Research Area: Linguistic theories, Cognitive Modeling and Psycholinguistics

Research Area Keywords: neurosymbolic approaches,cognitive modeling,computational psycholinguistics

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 4906