Elementary: Pattern-aware Evidence Discovery with Large Language Models

Jinpeng Zhang, Ruifang He, Junjie Wang

Keywords: Evidence Discovery, Large Language Model

Abstract: The remarkable success of rationale generation provokes precise Evidence Discovery, which aims to identify a small subset of the inputs sufficient to support a given claim. However, existing general extraction methods still fall short in quantifying the support of evidence and ensuring its completeness. This paper introduces a heuristic search framework, Elementary, which formulates the Evidence Discovery as a multi-step prompt construction process. Specifically, we offer a clear perspective that the LLMs prompted with \emph{according to}, without fine-tuning on domain-specific knowledge, can serve as an excellent reward function to assess sufficiency. Based on this, Elementary explores various potential reasoning patterns and uses future expected rewards, including independent and pattern-aware rewards, to find the optimal prompt as evidence. Experiments on three common task datasets demonstrate that the proposed framework significantly outperforms previous approaches, additional analysis further validates that Elementary has advantages in extracting complex evidence.

Primary Area: applications to computer vision, audio, language, and other modalities

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 9766