OSLLM: A Retrieve-Reason-Refine Framework for Multi-Domain Relation Extraction with Large Language Models

Jie Zhou, Yongxue Shan, Meihan Wu, Fei Hu, Li Zheng, Xiaodong Wang

Abstract: Relation Extraction (RE) aims to identify relations between entities in text. Despite the potential of Large Language Models (LLMs) in RE, they struggle with low relevance of relations in retrieved demonstrations and inconsistent responses to identical queries. Therefore, we introduce OSLLM, a novel retrieve-reason-refine framework for RE in Open Source Intelligence with LLMs. OSLLM leverages relation embeddings for accurate retrieval, performs better in-context reasoning, and refines outputs via template self-optimization and answer self-evaluation. Extensive experiments demonstrate that our method outperforms other LLM-based methods in traditional, temporal, and open RE tasks. Additionally, leveraging OSLLM to assist fine-tuned models in handling boundary samples significantly boosts the performance of these smaller models.

External IDs:dblp:conf/icmcs/ZhouSWHZW25