Stochastic Training of Graph Convolutional Networks

Jianfei Chen, Jun Zhu

Abstract: Graph convolutional networks (GCNs) are powerful deep neural networks for graph-structured data. However, GCN computes nodes' representation recursively from their neighbors, making the receptive field size grow exponentially with the number of layers. Previous attempts on reducing the receptive field size by subsampling neighbors do not have any convergence guarantee, and their receptive field size per node is still in the order of hundreds. In this paper, we develop a preprocessing strategy and two control variate based algorithms to further reduce the receptive field size. Our algorithms are guaranteed to converge to GCN's local optimum regardless of the neighbor sampling size. Empirical results show that our algorithms have a similar convergence speed per epoch with the exact algorithm even using only two neighbors per node. The time consumption of our algorithm on the Reddit dataset is only one fifth of previous neighbor sampling algorithms.

TL;DR: A control variate based stochastic training algorithm for graph convolutional networks that the receptive field can be only two neighbors per node.

Keywords: Graph convolutional networks, stochastic gradient descent, variance reduction, control variate

Data: [PPI](https://paperswithcode.com/dataset/ppi), [Reddit](https://paperswithcode.com/dataset/reddit)

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/stochastic-training-of-graph-convolutional/code)