Mathematically Modeling the Lexicon Entropy of Emergent Language

Brendon Boldt, David R Mortensen

Keywords: emergent language, emergent communication, stochastic process, modeling

TL;DR: We introduce a stochastic process which as a mathematical model of the lexicon entropy of emergent languages.

Abstract: We formulate a stochastic process, FiLex, as a mathematical model of lexicon entropy in deep learning-based emergent language systems. Defining a model mathematically allows it to generate clear predictions which can be directly and decisively tested. We empirically verify across four different environments that FiLex predicts the correct correlation between hyperparameters (training steps, lexicon size, learning rate, rollout buffer size, and Gumbel-Softmax temperature) and the emergent language's entropy in $20$ out of $20$ environment-hyperparameter combinations. Furthermore, our experiments reveal that different environments show diverse relationships between their hyperparameters and entropy which demonstrates the need for a model which can make well-defined predictions at a precise level of granularity.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/mathematically-modeling-the-lexicon-entropy/code)