Natural Language Systematicity from a Constraint on Excess Entropy
Keywords: language evolution, excess entropy, systematicity, linguistics, statistical complexity
TL;DR: Codes that are systematic in a natural-language-like way have lower excess entropy than matched nonsystematic codes.
Abstract: Natural language is systematic: utterances are composed of individually meaningful parts which are typically concatenated together. I argue that natural-language-like systematicity arises in codes when they are constrained by excess entropy, the mutual information between the past and the future of a process. In three examples, I show that codes with natural-language-like systematicity have lower excess entropy than matched alternatives.
Submission Number: 15
Loading