Mitigating Short Board Effect via Dynamic Reward Balancing in Multi-reward LLM Optimization

Nuo Chen, Yufei Gao, Yongnan Jin, Yan Hu, Anningzhe Gao, Lingyong Yan, Benyou Wang

Keywords: Multi-reward Optimization, Large Language Models

TL;DR: Our framework mitigates the short-board effect in multi-reward settings by adjusting reward weights, achieving balanced optimization across multiple metrics and improving overall model performance.

Abstract: In the current landscape of large language models (LLMs), many evaluation metrics have been developed and used as rewards during training to improve specific metrics. However, balancing these metrics and dynamically adjusting reward weights remains challenging, as current approaches often fail to enhance weaker metrics. To address this, we empirically propose a Dynamic Reward Balancing Optimization framework (DRBO) to mitigate the "short-board effect" by measuring performance, adjusting reward weights to prioritize weaker metrics, and optimizing the model via reinforcement learning. We apply DRBO to both single-task and multi-type task scenarios, validating its effectiveness in generation with citations and online shopping conversation tasks. The results demonstrate improved overall performance and balanced optimization across multiple metrics, effectively overcoming the diversity and complexity inherent in LLMs.

Submission Number: 20