Are LLMs Aware that Some Questions are not Open-ended?
Primary Area: representation learning for computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Large Language Model, Text Generation
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
Abstract: Large Language Models (LLMs) have shown the impressive capability of answering questions in a wide range of scenarios. However, when LLMs face different types of questions, it is worth exploring whether LLMs are aware that some questions have limited answers and have to respond more deterministically but some do not. We refer to the ability as question awareness that LLMs know to adjust the determinacy of the answers according to the questions. The lack of question awareness leads to two contradictory issues: (1) Too casual to answer non-open-ended questions. (2) Too boring to answer open-ended questions. In this paper, we first evaluate the question awareness ability of LLMs. The experimental results show that LLMs have the above issues of lacking the awareness of questions in certain domains, e.g. factual knowledge. To mitigate these issues, we propose a method called Question Awareness Temperature (QAT) sampling. This method enhances the question awareness ability of LLMs by dynamically adjusting the answer distributions based on question features. The automatic adjustment in QAT eliminates the need for manual temperature tuning in text generation. These findings underscore the potential of QAT sampling to enhance LLMs' question-awareness capabilities, thereby advancing their performance in various LLM benchmarks.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 7732
Loading