Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability

Published: 04 Mar 2024, Last Modified: 10 Apr 2024ME-FoMo 2024 PosterEveryoneRevisionsBibTeXCC BY 4.0
Keywords: Large Language Model; in-context learning; compositional ability
TL;DR: We investigate how Large Language Model behaves in composite tasks empirically and theoretically.
Abstract: Large language models (LLM) have emerged as a powerful tool exhibiting remarkable in-context learning (ICL) capabilities. In this study, we delve into the ICL capabilities of LLMs on composite tasks, with only simple tasks as in-context examples. We develop a test suite of composite tasks that include logical and linguistic challenges and perform empirical studies across different LLM families. We observe that models exhibit divergent behaviors: (1) For simpler composite tasks that contains different input segments, the models demonstrate decent compositional ability, while scaling up the model enhances this ability; (2) for more complex composite tasks that involving sequentiaL reasoning, models typically underperform, and scaling up provide no improvements. We offer theoretical analysis in a simplified setting. We believe our work sheds new light on the capabilities of LLMs in solving composite tasks regarding the nature of the tasks and model scale. Our dataset and code is available at {\small \url{https://github.com/OliverXUZY/LLM_Compose}}.
Submission Number: 69
Loading