Semantic-KG: Using Knowledge Graphs to Construct Benchmarks for Measuring Semantic Similarity

Qiyao Wei, Edward Morrell, Lea Goetz, Mihaela van der Schaar

Keywords: Large language model, semantic similarity benchmark, knowledge graph, automation, scalability

TL;DR: This paper introduces a novel method for generating benchmarks to evaluate semantic similarity methods for LLM outputs, achieving cross-domain scalability and not replying on human judgment

Abstract: Evaluating the open-form textual responses generated by Large Language Models (LLMs) typically requires measuring the semantic similarity of the response to a (human generated) reference. However, there is evidence that current semantic similarity methods may capture syntactic or lexical forms over semantic content. While benchmarks exist for semantic equivalence, they often suffer from high generation costs due to reliance on subjective human judgment, limited availability for domain-specific applications, and unclear definitions of equivalence. This paper introduces a novel method for generating benchmarks to evaluate semantic similarity methods for LLM outputs, specifically addressing these limitations. Our approach leverages knowledge graphs (KGs) to generate pairs of natural-language statements that are semantically similar or dissimilar, with dissimilar pairs categorized into one of four sub-types. We generate benchmark datasets in four different domains (general knowledge, biomedicine, finance, biology), and conduct a comparative study of semantic similarity methods including traditional natural language processing scores and LLM-as-a-judge predictions. We observe that the sub-type of semantic variation, as well as the domain of the benchmark impact the performance of semantic similarity methods, with no method being consistently superior. Our results present important implications for the use of LLM-as-a-judge in detecting the semantic content of text. Code is available at \url{https://github.com/QiyaoWei/semantic-kg} and the dataset is available at \url{https://huggingface.co/datasets/QiyaoWei/Semantic-KG}.

Dataset URL: https://huggingface.co/datasets/QiyaoWei/Semantic-KG

Code URL: https://github.com/QiyaoWei/semantic-kg

Primary Area: Datasets & Benchmarks for applications in language modeling and vision language modeling

Flagged For Ethics Review: true

Submission Number: 1667