On Task Description of In-context Learning: A Study from Information Perspective

Meidai Xuanyuan, Tao Yang, Jingwen Fu, Yuwang Wang

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: in-context learning

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract: Large language models (LLMs) have demonstrated remarkable performance in a wide range of applications, making in-context learning an essential technique. Although the in-context learning has been widely applied, our understanding of its underlying processes still remains limited. In-context learning in LLMs primarily relies on two types of information: in-context samples and task descriptions. While previous research has extensively investigated the influence of in-context samples on learning behavior, the role of task descriptions has not been adequately explored, despite their practical significance. In this paper, we present a study examining the impact of task descriptions on the in-context learning performance of LLMs. We devise a synthetic experiment setting, making the information of task description controllable. Through a series of well-designed experiments, we systematically vary task description information and assess the resulting effects on model performance across multiple tasks. Our findings reveal complex roles of task descriptions: task description will suppress the model to learn from in-context examples; task description will increase the lower bound of the in-context learning performance. This study contributes to a deeper understanding of the in-context learning mechanism in LLMs, paving the way for more effective real-world applications of these powerful models

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 2250