A Closer Look into Using Large Language Models for Automatic Evaluation

Cheng-Han Chiang, Hung-yi Lee

Submission Type: Regular Short Paper

Submission Track: Theme Track: Large Language Models and the Future of NLP

Submission Track 2: Resources and Evaluation

Keywords: LLM, automatic evaluation, LLM evaluaiton

TL;DR: We compare two widely used LLM evaluation methods and improves the correlation between ChatGPT's rating and ground truth human ratings.

Abstract:

Using large language models (LLMs) to evaluate text quality has recently gained popularity.
Some existing prior works explore the idea of using LLMs for evaluation, while they differ in some details of the evaluation process.
In this paper, we analyze *LLM evaluation* and *G-Eval*, and we discuss how those details in the evaluation process change how well the ratings given by LLMs correlate with human ratings.
We find that the auto Chain-of-Thought (CoT) used in G-Eval does not always make G-Eval more aligned with human ratings.
We also show that forcing the LLM to output only a numeric rating, as in G-Eval, is suboptimal.
Last, we reveal that asking the LLM to explain its own ratings consistently improves the correlation between the ChatGPT and human ratings and pushes state-of-the-art (SoTA) correlations on two meta-evaluation datasets.

Submission Number: 4841