PACE: A Parallelizable Computation Encoder for Directed Acyclic Graphs

Zehao Dong, Muhan Zhang, Fuhai Li, Yixin Chen

Keywords: DAG encoder, graph neural network, Transformer

Abstract: Optimization of directed acyclic graph (DAG) structures has many applications, such as neural architecture search (NAS) and probabilistic graphical model learning. Encoding DAGs into real vectors is a dominant component in most neural-network-based DAG optimization frameworks. Currently, most popular DAG encoders use an asynchronous message passing scheme which sequentially processes nodes according to the dependency between nodes in a DAG. That is, a node must not be processed until all its predecessors are processed. As a result, they are inherently not parallelizable. In this work, we propose a Parallelizable Attention-based Computation structure Encoder (PACE) that processes nodes simultaneously and encodes DAGs in parallel. We demonstrate the superiority of PACE through encoder-dependent optimization subroutines that search the optimal DAG structure based on the learned DAG embeddings. Experiments show that PACE not only improves the effectiveness over previous sequential DAG encoders with a significantly boosted training and inference speed, but also generates smooth latent (DAG encoding) spaces that are beneficial to downstream optimization subroutines.

One-sentence Summary: This paper introduces a novel DAG encoder based on Transformer to encode the computation structure defined by DAGs in a fully parallelizable manner.

Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2203.10304/code)