Business as Rulesual: A Benchmark and Framework for Business Rule Flow Modeling with LLMs

ACL ARR 2026 January Submission2120 Authors

Keywords: Process Modeling, Information Extraction, Interpretability and Analysis of Models for NLP, Language Modeling, NLP Applications

Abstract:

Extracting structured procedural knowledge from unstructured business documents is a critical yet unresolved bottleneck in process automation. While prior work has focused on extracting linear action flows from instructional texts (e.g., recipes), it has insufficiently addressed the complex logical structures—such as conditional branching and parallel execution—that are pervasive in real-world regulatory and administrative documents. Furthermore, existing benchmarks are limited by simplistic schemas and shallow logical dependencies, restricting progress toward logic-aware large language models (LLMs). To bridge this ``Logic Gap'', we introduce \textbf{BREX}, a carefully curated benchmark comprising 409 real-world business documents and 2,855 expert-annotated rules. Unlike prior datasets centered on narrow service scenarios, BREX spans over 30 vertical domains, covering scientific, industrial, administrative, and financial regulations.

We further propose \textbf{ExIde}, a structure-aware reasoning framework that investigates five distinct prompting strategies, ranging from implicit semantic alignment to executable grounding via pseudo-code generation, enabling explicit modeling of rule dependencies and providing an out-of-the-box framework for different business customers without finetuning their own LLMs. We benchmark ExIde using 13 state-of-the-art LLMs. Our extensive evaluation reveals that: (1) Executable grounding serves as a superior inductive bias, significantly outperforming standard prompts in rule extraction; and (2) Reasoning-optimized models demonstrate a distinct advantage in tracing long-range dependencies and non-linear rule dependencies compared to standard instruction-tuned models.

Paper Type: Long

Research Area: Information Extraction and Retrieval

Research Area Keywords: Information Extraction, Interpretability and Analysis of Models for NLP, Resources and Evaluation, Language Modeling, NLP Applications

Contribution Types: Model analysis & interpretability, NLP engineering experiment, Data resources, Data analysis

Languages Studied: Chinese

Submission Number: 2120