CANDID DAC: Leveraging Coupled Action Dimensions with Importance Differences in DAC

Philipp Bordne, M Asif Hasan, Edward Bergman, Noor Awad, André Biedenkapp

Keywords: Dynamic Algorithm Configuration, Reinforcement Learning, high-dimensional action spaces

TL;DR: Using sequential policies to control high-dimensional action spaces for Dynamic Algorithm Configuration.

Abstract:

High-dimensional action spaces remain a challenge for dynamic algorithm configuration (DAC).
Interdependencies and varying importance between action dimensions are further known key characteristics of DAC problems. We argue that these Coupled Action Dimensions with Importance Differences (CANDID) represent aspects of the DAC problem that are not yet fully explored. To address this gap, we introduce a new white-box benchmark within the DACBench suite that simulates the properties of CANDID. Further, we propose sequential policies as an effective strategy for managing these properties. Such policies factorize the action space and mitigate exponential growth by learning a policy per action dimension. At the same time, these policies accommodate the interdependence of action dimensions by fostering implicit coordination. We show this in an experimental study of value-based policies on our new benchmark. This study demonstrates that sequential policies significantly outperform independent learning of factorized policies in CANDID action spaces. In addition, they overcome the scalability limitations associated with learning a single policy across all action dimensions. The code used for our experiments is available under https://github.com/PhilippBordne/candidDAC.

Submission Checklist: Yes

Broader Impact Statement: Yes

Paper Availability And License: Yes

Code Of Conduct: Yes

Optional Meta-Data For Green-AutoML: All questions below on environmental impact are optional.

Submission Number: 14