CPUQ: Categorical Perplexity based Uncertainty Quantification with Language Models

Anonymous

Abstract: Agent-Based Modelling (ABM) for Economic Allocation (EA) analyzes interactions between economic agents and indicators over time, aiding policymakers and decision analysts in scenario analysis for complex systems like government spending or financial market contagion. These EA-ABMs, when graphed, often have limited datasets (timesteps) but a large number of nodes (agents or indicators) and edges (relationships), which hinders statistical network estimation methods. Additionally, statistical relationship estimation methods lack interpretability for non-technical users. To address these issues, we introduce the CPUQ framework, compatible with any Language Model (LM) and utilizing a LM's reasoning to generate predictive hurdle distributions that quantify relationship strength between agents/indicators, coupled with textual explanations for each prediction to enhance interpretability for non-technical audiences. CPUQ also includes a novel post-hoc calibration approach for network estimation. Evaluation on a real EA dataset demonstrates CPUQ's alignment with expert opinions and its superior forecasting capability over existing statistical and LM methods in assessing relationships in EA-ABMs.

Paper Type: long

Research Area: Computational Social Science and Cultural Analytics

Contribution Types: Model analysis & interpretability

Languages Studied: English