Physics-Informed RL for HVAC Controls: Imitation to Online Fine-Tuning

Ozan Baris Mulayim, Mario Berges, Kevin J. Kircher, Guannan Qu

Keywords: physics informed RL, reinforcement learning, HVAC controls

TL;DR: We tackle real-world challenges in deploying RL for HVAC control, addressing system modeling, behavior alignment, and reward design while proposing physics-informed and calibration strategies to improve safety, interpretability, and reliability.

Abstract: Reinforcement learning (RL) deployment in real-world, safety-critical systems remains a significant challenge despite advancements in the field. This work analyzes practical obstacles to deploying RL for HVAC systems through a case study on residential heat pump control, utilizing real-world data to apply and extend a model-based RL algorithm. To enhance interpretability and safety, we incorporated a physics-informed system model and faced challenges in parameter estimation due to local minima, even in idealized settings with sufficient exploration. Given the complexities of reward shaping without a simulator, inverse RL was employed to learn cost parameters from an existing controller. We then introduced two calibration strategies to impose user-defined control characteristics via a simple, generic reward function with a single thermal discomfort price parameter. Ultimately, this research aims to advance the practical application of RL in safety-critical systems, offering insights for bridging the gap between simulation and real-world deployment.

Submission Number: 9