The Diamonds and Rusts of LLMs as Guardian Angels

ACL ARR 2026 January Submission9174 Authors

Keywords: Large Language Models, Agents, Planning, Safety, Evaluation, Benchmark, Human-Centered AI

Abstract:

Large Language Models (LLMs) have become increasingly powerful tools for complex planning tasks, yet most research remains confined to closed-world environments, limiting their effectiveness in dynamic, real-world scenarios. This paper introduces a novel framework inspired by Peter Szolovits' "Guardian Angel" concept, which leverages LLMs to manage daily tasks and control physical devices in safety-critical, open-world environments. Our approach aims to bridge the gap between traditional planning solutions and the need for adaptive, real-time decision-making in human-centered applications. We present a custom-curated dataset featuring real-world use cases, such as autonomous driving and automated insulin delivery systems, to evaluate the strengths and limitations of LLM-based planning. Furthermore, we introduce a new benchmark for assessing LLMs in these environments, along with an LLM-based evaluation method to improve the accuracy and cost-effectiveness of plan assessments. Our results highlight both the advantages and challenges of using LLMs as "Guardian Angels" for real-world planning, offering insights into their future potential and application in safety-critical domains.
The benchmark dataset, simulation environments, and evaluation scripts are provided in the supplementary material to support reproducibility.

Paper Type: Long

Research Area: AI/LLM Agents

Research Area Keywords: Autonomous agents, LLM agents, planning in agents, agent evaluation, safety and alignment for agents

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources

Languages Studied: English

Submission Number: 9174