Enforcing Temporal Constraints for LLM Agents

Adharsh Kamath, Sishen Zhang, Changming Xu, Shubham Ugare, Gagandeep Singh, Sasa Misailovic

Track: long paper (up to 8 pages)

Keywords: Tool-Calling Agents, Temporal Logic, Runtime Monitoring

TL;DR: Agent-C is a novel framework that provides run-time guarantees ensuring LLM agents adhere to temporal and state-dependent constraints

Abstract:

We present Agent-C, a novel framework that provides run-time guarantees ensuring LLM agents adhere to temporal and state-dependent constraints. Agent-C introduces a domain-specific language for expressing temporal properties,
translates specifications to first-order logic, and interleaves SMT solving with constrained generation techniques to detect non-compliant agent actions during token generation.
If the agent generates a non-compliant action, Agent-C backtracks the generation and samples a compliant alternative. We evaluate Agent-C on two real-world applications: retail customer service and airline ticket reservation, with multiple language models (open-weight and closed-source) as agents. Our results demonstrate that Agent-C achieves perfect safety (100% conformance, 0% harm) in both benign and adversarial scenarios, while improving task utility compared to state-of-the-art guardrails and unrestricted agents. On state-of-the-art closed-source models, Agent-C improves conformance (from 77.4% to 100% for Claude Sonnet 4.5 and 83.7% to 100% for GPT-5), while increasing utility (from 66.1% to 69.3% and 61.0% to 64.3%, respectively), setting a new state-of-the-art frontier for reliable agentic systems. More details about the framework (e.g., code repository, demo) can be found on the project website: https://go.illinois.edu/agentc.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 15