Select to Perfect: Imitating desired behavior from large multi-agent data

Tim Franzmeyer, Edith Elkind, Philip Torr, Jakob Nicolaus Foerster, Joao F. Henriques

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: multi-agent systems, imitation learning, credit assignment, learning from human data

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

AI agents are commonly trained with large datasets of demonstrations of human behavior.
However, not all behaviors are equally safe or desirable.
Desired characteristics for an AI agent can be expressed by assigning desirability scores, which we assume are not assigned to individual behaviors but to collective trajectories.
For example, in a dataset of vehicle interactions, these scores might relate to the number of incidents that occurred.
We first assess the effect of each individual agent's behavior on the collective desirability score, e.g., assessing how likely an agent is to cause incidents.
This allows us to selectively imitate agents with a positive effect, e.g., only imitating agents that are unlikely to cause incidents.
To enable this, we propose the concept of an agent's \textit{Exchange Value}, which quantifies an individual agent's contribution to the collective desirability score.
The Exchange Value is the expected change in desirability score when substituting the agent for a randomly selected agent.
We propose additional methods for estimating Exchange Values from real-world datasets, enabling us to learn desired imitation policies that outperform relevant baselines. The project website can be found at https://tinyurl.com/select-to-perfect.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Primary Area: general machine learning (i.e., none of the above)

Submission Number: 3394