Graph reinforcement learning resistant to sparsity scaling

Abstract:

We investigate the impact of graph sparsity on the NP-hard combinatorial optimization problem of delivery route optimization, by combining proximity policy optimization with graph convolutional neural networks.
Sparsity poses a critical challenge for consistent routing policies, as limited connectivity can significantly impact solution strategies by changing the intrinsic structure of possible paths.
In order to address such challenge, we relate robustness in different graph sparsity regimes to learning dynamics in a sequential decision-making and graph-structured environment.
Our experiments systematically consider graphs with up to 20 nodes, demonstrating the robustness of the algorithm to various densities of graph sparsity, from low-degree nodes that impose inefficient paths to highly connected structures that require extensive exploration.
To ensure consistent evaluation across different graph topologies, we introduce the normalization of the return function based on the length of the DRL episode and the number of nodes in the graph.
The algorithm is evaluated based on metrics such as cumulative reward, path length, and the number of steps required to complete a DRL episode.
Learning maintains stable performance across a wide range of graph densities, thus revealing its effectiveness.
By systematically characterizing the role of sparsity in graph-based reinforcement learning for route optimization, this work provides insights into the challenges posed by real-world transportation logistics networks.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Christopher_Morris1

Submission Number: 7056