PRISM: Prompt-Refined In-Context System Modeling for Financial Retrieval

Chun Chet Ng, Jia Yu Lim, Low Wei Zeng

Track: Main Papers Track (6 to 9 pages)

Keywords: Prompt Engineering, In-Context Learning, Multi Agent System, Financial Information Retrieval, Financial Question Answering

TL;DR: PRISM delivers simple, powerful, production-ready financial intelligence that requires no training and can deliver top tier results.

Abstract: With the rapid progress of large language models (LLMs), financial information retrieval has become a critical industrial application. Extracting task-relevant information from lengthy financial filings is essential for both operational and analytical decision-making. We present PRISM, a training-free framework that integrates refined system prompting, in-context learning (ICL), and lightweight multi-agent coordination for document and chunk ranking tasks. Our primary contribution is a systematic empirical study that characterizes when each component provides value; prompt engineering delivers consistent performance with minimal overhead, ICL enhances reasoning for complex queries when applied selectively, and multi-agent systems show potential primarily with larger models and careful architectural design. Extensive ablation studies across FinAgentBench, FiQA-2018, and FinanceBench reveal that simpler configurations often outperform complex multi-agent pipelines, providing practical guidance for practitioners. Our best configuration achieves an NDCG@5 of 0.71818 on FinAgentBench, ranking third while being the only training-free approach in the top three. We provide comprehensive feasibility analyses covering latency, token usage, and cost trade-offs to support deployment decisions. Code is ready for public release.

Submission Number: 52