Multi-Objective Agency Requires Non-Markovian Rewards
Keywords: reward modeling, multi-objective agency, intertemporal fairness
TL;DR: We show that non-Markovian rewards are necessary for agents whose multiple objectives have varying time preference, and propose a practical non-Markovian aggregation scheme
Abstract: As the capabilities of artificial agents improve, they are being increasingly deployed to service multiple diverse objectives and stakeholders. However, the composition of these objectives is often performed ad hoc, with no clear justification. This paper takes a normative approach to multi-objective agency: from a set of intuitively appealing axioms, we show that Markovian aggregation of Markovian reward functions is not possible when the time preference (discount factor) for each objective may vary. It follows that optimal multi-objective agents must admit rewards that are non-Markovian (history dependent) with respect to the individual objectives. To this end, we propose a practical non-Markovian aggregation scheme that overcomes the impossibility with only one additional parameter for each objective. Our work offers new insights into sequential, multi-objective agency and intertemporal choice, and has practical implications for the design of AI systems deployed to serve multiple generations of principals with varying time preference.
Submission Number: 62
Loading