Accuracy Isn’t Everything: Understanding the Desiderata of AI Tools in Legal-Financial Settings

Sudhan Chitgopkar, Noah Dohrmann, Stephanie Monson, Jimmy Mendez, Finale Doshi-Velez, Weiwei Pan

Keywords: HCI, Human Computer Interaction, IE, Information Extraction, ML, Machine Learning, Financial IE, Financial AI, Legal IE, Legal AI, Legal LLMs, AI Design, AI Tool Design, AI HCI, XAI

Abstract: Modern financial analysts' workflows often include significant manual information extraction (IE) from legal financial documents. Recent advances in large language models have sparked an interest in the automation of such workflows using ML. While research and commercial tools exist for legal IE, this work often focuses exclusively on maximizing extraction accuracy rather than supporting actual analysts' workflows. To fill this gap, we develop an AI-enabled tool for legal IE as a probe for interviews with domain experts in finance. We aim to understand how IE tools should be designed for safe and effective use in financial settings. Our interviews underscore a number of expected desiderata for future design of IE tools (e.g. designs should enable users to easily validate results), as well as a number of important unexpected implications (e.g. little value is placed on an AI tool's self-reported uncertainty).

Submission Number: 64