Mitigating Short Board Effect via Dynamic Reward Balancing in Multi-reward LLM Optimization
Abstract: In the current landscape of large language models (LLMs), many evaluation metrics have been developed and used as rewards during training to improve specific metrics. However, balancing these metrics and dynamically adjusting reward weights remains challenging, as current approaches often fail to enhance weaker metrics. To address this, we empirically propose a Dynamic Reward Balancing Optimization framework (DRBO) to mitigate the "short-board effect" by measuring performance, adjusting reward weights to prioritize weaker metrics, and optimizing the model via reinforcement learning. We apply DRBO to both single-task and multi-type task scenarios, validating its effectiveness in generation with citations and online shopping conversation tasks. The results demonstrate improved overall performance and balanced optimization across multiple metrics, effectively overcoming the diversity and complexity inherent in LLMs.
Paper Type: Long
Research Area: NLP Applications
Research Area Keywords: Multi-reward Optimization, Large Language Models
Contribution Types: NLP engineering experiment, Data analysis
Languages Studied: English
Submission Number: 881
Loading