Universal Off-Policy Selection for Human-Centric Systems via Participant Sub-grouping

Ge Gao, Xi Yang, Qitong Gao, Song Ju, Miroslav Pajic, Min Chi

Primary Area: reinforcement learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Off-policy selection (OPS), Participants subgrouping for OPS, OPS for human-involved systems

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Abstract:

Human-centric tasks like healthcare and education are characterized by heterogeneity among patients and students, resulting in different disease trajectories and learning styles that require personalized treatments or instructional interventions for specific subgroups. When deploying reinforcement learning (RL) for such tasks, off-policy selection (OPS) is essential, since it it closes the loop by selecting and
evaluating RL-induced policies offline, without the need for any online interaction with the participants. Many pre-existing OPS methods, however, do not consider the heterogeneity among the participants. In this work, we introduce a universal off-policy selection (UOPS) approach to address the issue of participant heterogeneity by taking a multi-step approach. Initially, it divides the participants into sub-groups, grouping together those who exhibit similar behaviors. Subsequently, it acquires OPS criteria tailored to each of these sub-groups. Consequently, when new participants come, they will receive policy recommendations based on the sub-groups
they align with. This methodology enhances the adaptability and personalization of the RL system, ensuring that policy selections align more closely with the unique characteristics of each participant or group of participants. We evaluate UOPS’ effectiveness through two applications: an intelligent tutor system that has been used in classrooms for over eight years, as well as a healthcare application for
sepsis treatment and intervention. In both applications, UOPS shows significant improvements in students’ learning and patient outcomes.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6782