Exploring Search for FPGA Placement using RL

Shang Wang, Owen Randall, Matthew E. Taylor

Keywords: Reinforcement learning, Search, Combinatorial search, FPGA placement

Abstract:

Reinforcement learning (RL) has become increasingly popular, but applying it successfully to real-world problems remains challenging.
In contrast, search has been a powerful yet underused tool in real-world settings, despite achieving notable successes in several domains.
We believe the potential of search-based approaches in real-world applications has not yet been fully explored, as many real-world applications are combinatorial in nature.

Our work explores the use of reinforcement learning guided Monte Carlo tree search (MCTS) algorithms, using AlphaZero-style agents in the challenging real-world task of field-programmable gate array (FPGA) component placement.
Our preliminary results show that MCTS can significantly improve a reinforcement learning agent's performance in this task. This, in itself, is not surprising.
However, additional results show that by integrating Gumbel-enhanced MCTS, the policy converges faster and achieves better performance, demonstrating the utility of search-based approaches in this real-world application.
Our results highlight a broader insight: search is not an outdated baseline --- when used in combination with RL methods, it could be an under-used approach for solving real-world decision problems.

Submission Number: 9