CPUQ: Categorical Perplexity Based Uncertainty Quantification with Language Models

Anonymous

Abstract: Agent Based Modelling (ABM) algorithms for Economic Allocation (EA) systems model interactions between economic agents and indicators. These EA-ABMs provide important insight for policy makers and decision analysis as they can be used to model complex systems such as Government Spending or Financial Market Contagion. However, the utility of EA-ABM's depends on the quality and interpretability of the underlying graph’s estimated edge weights. Statistical network estimation methods perform poorly due to these datasets often having limited timesteps of data but a large number of nodes (economic actors or indicators) and edges (causal relationships). We propose a structured method to use Large Language Models (LLM) to produce predictive hurdle distributions for the edge weights; enhancing interpretation through uncertainty quantification and textual reasoning. Our approach, Categorical Uncertainty based Uncertainty Quantification (CPUQ) decouples the modelling of causal relationships into separately modelling existence and causal relationship strength. Through evaluation on a real Economic Allocation dataset, we show that CPUQ produces probabilistic predictions well aligned with experts opinions, and achieves better EA-ABMs forecasting ability than existing statistical and LLM based methods. We also motivate a solution for the issues of conflating a language model's uncertainty with syntactical uncertainty as opposed to semantic uncertainty.

Paper Type: long

Research Area: Computational Social Science and Cultural Analytics

Contribution Types: Model analysis & interpretability

Languages Studied: English

Consent To Share Submission Details: On behalf of all authors, we agree to the terms above to share our submission details.