Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability

Zhuoyan Xu, Zhenmei Shi, Yingyu Liang

Keywords: Large Language Model; in-context learning; compositional ability

TL;DR: We investigate how Large Language Model behaves in composite tasks empirically and theoretically.

Abstract:

Large language models (LLM) have emerged as a powerful tool exhibiting remarkable in-context learning (ICL) capabilities.
In this study, we delve into the ICL capabilities of LLMs on composite tasks, with only simple tasks as in-context examples. We develop a test suite of composite tasks that include logical and linguistic challenges and perform empirical studies across different LLM families. We observe that models exhibit divergent behaviors: (1) For simpler composite tasks that contains different input segments, the models demonstrate decent compositional ability, while scaling up the model enhances this ability; (2) for more complex composite tasks that involving sequentiaL reasoning, models typically underperform, and scaling up provide no improvements. We offer theoretical analysis in a simplified setting.
We believe our work sheds new light on the capabilities of LLMs in solving composite tasks regarding the nature of the tasks and model scale. Our dataset and code is available at {\small \url{https://github.com/OliverXUZY/LLM_Compose}}.

Submission Number: 69