Are LLMs Aware that Some Questions are not Open-ended?

Anonymous

Abstract: Large Language Models (LLMs) have shown the impressive capability of answering questions in a wide range of scenarios. However, when LLMs face different types of questions, it is worth exploring whether LLMs are aware that some questions have limited answers and have to respond more deterministically but some do not. We refer to the ability as question awareness that LLMs know to adjust the determinacy of the answers according to the questions. The lack of question awareness leads to two contradictory issues: (1) Too casual to answer non-open-ended questions. (2) Too boring to answer open-ended questions. In this paper, we first evaluate the question awareness of LLMs. The experimental results show that LLMs have the above issues of lacking the awareness of questions in certain domains, e.g. factual knowledge. To mitigate these issues, we propose a method called Question Awareness Temperature Sampling (QuATS). This method enhances the question awareness of LLMs by dynamically adjusting the output distributions based on question features. The automatic adjustment in QuATS eliminates the need for manual temperature tuning in text generation and improves model performance in various benchmarks.

Paper Type: long

Research Area: Generation

Contribution Types: Model analysis & interpretability, NLP engineering experiment

Languages Studied: English