Can ChatGPT understand the implicit meaning of language? Discussion of ChatGPT's ability to generate metaphorical samples
Abstract: The effectiveness of large-scale language modeling (LLM) in generating data samples has been widely proven, especially in question answering and textual entailment tasks. However, these tasks are primarily concerned with surface semantics and usually require the model to learn only information about lexical and syntactic structures. In contrast, generating metaphorical samples requires LLMs to develop a deeper understanding of the implicit meanings in the text. Therefore, the aim of this paper is to explore the ability of ChatGPT to generate metaphorical samples. First, we propose two prompt enhancement methods based on definitions and multiple word meanings. The former introduces a metaphor definition, and the latter requires LLM to generate the corresponding metaphorical or literal sample content based on each word sense. Experimental results show that the SPE method performs slightly lower than manually labeled samples in terms of fine-tuning performance (3.5\% lower than the average F1 value for the three metaphorical datasets), but at 1/250th the cost of the latter. Since most of work focuses on zero- or few-shot methods, we use it as a baseline. We provide an in-depth discussion of the differences between the four sample generation methods mentioned above through manual evaluation, automated evaluation, and example analysis. To enhance the reliability of the study, we introduce ChatGPT, LLaMA3, and Mixtral to further explore the differences in generating implicit semantic content across LLMs.
Paper Type: Long
Research Area: Efficient/Low-Resource Methods for NLP
Research Area Keywords: Efficient/Low-Resource Methods for NLP, Interpretability and Analysis of Models for NLP, Language Modeling
Contribution Types: Approaches to low-resource settings, Approaches low compute settings-efficiency
Languages Studied: English
Submission Number: 324
Loading