Report-based Recommendations for Policy Making and Agency Operations: Dataset and LLM Evaluation

ACL ARR 2025 May Submission2522 Authors

Abstract: Large Language Models (LLMs) have provided incredible tools when it comes to text generation. These generative capabilities bring us to a point where LLMs could potentially provide useful insights in policy making or agency operations. In this paper, we introduce a new task consisting of generating recommendations which can be used to inform future actions and improvements of agencies work within private and public organisations. In particular, we present the first benchmark and coherent evaluation for developing recommendation systems to inform organisation policies. This task is clearly different from usual product or user recommendation systems, but rather aims at providing a basis to suggest policy improvements based on the conclusions drawn from reports. Our results demonstrate that state-of-the-art LLMs have the potential to emphasize and reflect on key issues and learning points within generated recommendations.

Paper Type: Long

Research Area: Special Theme (conference specific)

Research Area Keywords: Special Theme Track: Interdisciplinary Recontextualization of NLP, Resources and Evaluation,NLP Applications

Contribution Types: Approaches to low-resource settings, Data resources, Data analysis

Languages Studied: English

Submission Number: 2522