Two steps to risk sensitivity

Christopher Gagne, Peter Dayan

Keywords: Risk measures, Conditional value-at-risk, Two-step task, Time-consistency, Distributional reinforcement learning, Anxiety

Abstract: Distributional reinforcement learning (RL) – in which agents learn about all the possible long-term consequences of their actions, and not just the expected value – is of great recent interest. One of the most important affordances of a distributional view is facilitating a modern, measured, approach to risk when outcomes are not completely certain. By contrast, psychological and neuroscientific investigations into decision making under risk have utilized a variety of more venerable theoretical models such as prospect theory that lack axiomatically desirable properties such as coherence. Here, we consider a particularly relevant risk measure for modeling human and animal planning, called conditional value-at-risk (CVaR), which quantifies worst-case outcomes (e.g., vehicle accidents or predation). We first adopt a conventional distributional approach to CVaR in a sequential setting and reanalyze the choices of human decision-makers in the well-known two-step task, revealing substantial risk aversion that had been lurking under stickiness and perseveration. We then consider a further critical property of risk sensitivity, namely time consistency, showing alternatives to this form of CVaR that enjoy this desirable characteristic. We use simulations to examine settings in which the various forms differ in ways that have implications for human and animal planning and behavior.

Code Of Conduct: I certify that all co-authors of this work have read and commit to adhering to the NeurIPS Statement on Ethics, Fairness, Inclusivity, and Code of Conduct.

TL;DR: We use a conditional value-at-risk (CVaR) measure to show that many human subjects are highly risk averse in the popular two-step task; and use simulations in a novel domain to suggest how to distinguish time-consistent/inconsistent CVaRiants.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2111.06803/code)