Customizing Reinforcement Learning Agent with Multi-Objective Preference Control

Yucheng Yang, Tianyi Zhou, Meng Fang, Mykola Pechenizkiy

Keywords: reinforcement learning, multi-objective optimization, deep reinforcement learning

TL;DR: We develop “Preference control (PC) RL”, which aims to train a meta-policy that takes user preference as input controlling the generation of a trajectory on the Pareto frontier adhering to the preference.

Abstract:

Practical reinforcement learning (RL) usually requires agents to be optimized for multiple potentially conflicting criteria, e.g. speed vs. safety.
Although Multi-Objective RL (MORL) algorithms have been studied in previous works, their trained agents often lack precise controllability of the delicate trade-off among multiple objectives. Hence, the resulting agent is not versatile in aligning with customized requests from different users.
To bridge the gap, we develop ``Preference control (PC) RL'', which aims to train a meta-policy that takes user preference as input controlling the generation of a trajectory on the Pareto frontier adhering to the preference. To this end, we train a preference-conditioned meta-policy by our proposed preference-regularized MORL algorithm. The achieved meta-policy performs as a multi-objective optimizer that can produce user-desired solutions on the Pareto frontier. The proposed algorithm is analyzed and its convergence and controllability are theoretically justified.
Experiments from discrete toy examples to higher-dimension robotic control tasks and experiments with more than two objectives are conducted to show its performance. In these experiments, PCRL-trained policies show significantly better controllability than existing approaches and can generate Pareto optimal solutions with better diversity and utilities.

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 5131