Legal-ISA: A Modular Framework for Systematic Legal AI Evaluation

Dai-Song Gong

Keywords: Legal AI, Modular Architecture, Neuro-Symbolic Systems, Explainable AI, Cross-Jurisdictional Reasoning

Paper Type: Full papers

TL;DR: Legal-ISA: A modular framework enabling plug-and-play composition of legal AI components through standardized operation interfaces with mandatory provenance tracking.

Abstract: Current legal AI faces critical fragmentation across specialized tasks and diverse jurisdictions. We introduce Legal-ISA, a modular integration framework that addresses this by systematically composing mature techniques—retrieval, verification, and reasoning—via standardized interfaces, inspired by the Instruction Set Architecture principle. The framework defines a comprehensive set of operations covering the majority of tasks within major legal benchmarks, enabling mandatory provenance tracking and fine-grained failure attribution. Comprehensive evaluation across four diverse legal benchmarks spanning multiple jurisdictions validates Legal-ISA's design, demonstrating: True Modularity achieved through configuration-only substitution across multiple component combinations; Systematic Attribution achieving significantly higher error attribution coverage than pure neural baselines; Quantified Transparency evidenced by low calibration error for reliable uncertainty assessment; and Performance Gains showing substantial improvement over the best Retrieval-Augmented Generation baseline. However, cross-jurisdictional tests revealed a performance drop, exposing a critical reliance on manual legal knowledge engineering for concept mapping; this dependency precludes direct comparison with automated model-learning approaches. Our core contribution is engineering-focused: a systematically validated, modular architecture for standardized legal AI evaluation and comparative risk assessment via composition and human knowledge integration.

Submission Number: 15