Action chunking as policy compression

Lucy Lai, Ann Huang, Samuel Gershman

Abstract:

Many skills in our everyday lives are learned by sequencing actions toward a desired goal. The action sequence can become a \chunk" when individual actions are grouped together and executed as one unit, making them more efficient to store and execute. While chunking has been studied extensively across various domains, a puzzle remains as to why and under what conditions action chunking occurs. To tackle these questions, we apply the concept of policy compression|the reduction in cognitive cost by making policies simpler to action selection in order to explain the origin of chunking. We argue that chunking is a result of optimizing the trade-off between reward and policy
complexity. Chunking compresses policies when there is structure in the mapping from environment states to optimal action sequences, reducing the amount of memory necessary to encode the policy. We experimentally con rm our model's predictions, showing that chunking reduces policy complexity and reaction time. Chunking also increases with working memory load, consistent with the hypothesis that the degree of policy compression scales with the scarcity of cognitive resources.