Self-Attention-Guided Genetic Programming: Leveraging BERT for Enhanced Tree-Structured Data Operations

Shanshi Mao, Fangfang Zhang, Yi Mei, Mengjie Zhang

Keywords: BERT, Tree Data Structure, Optimization

Abstract: This study investigates the application of BERT to tree-structured data which presents a significant challenge due to its lack of explicit sequential order and complex topological dependencies. While BERT has demonstrated strong performance in learning rich representations from sequential and grid-based inputs like natural language and images, its extension to non-sequential topologies remains an open research question. In this paper, we integrate BERT with genetic programming whose classic data representation is tree data structure to solve the dynamic flexible job shop scheduling (DFJSS) problem. The DFJSS problem's inherent computational complexity and highly dynamic, uncertain nature provide a rigorous testbed for our methodology. Our experiments demonstrate that BERT can effectively capture and integrate the structural information embedded in these tree-based representations. This finding highlights the versatility and adaptability of the self-attention mechanism, extending its utility beyond conventional sequential or grid-based data structures to a broader class of complex, non-sequential topologies.

Primary Area: optimization

Submission Number: 3852