Parallel Large-Scale Neural Network Training For Online Advertising

Quanchang Qi, Guangming Lu, Jun Zhang, Lichun Yang, Haishan Liu

Abstract: Neural networks have shown great successes in many fields. Due to the complexity of the training pipeline, however, using them in an industrial setting is challenging. In online advertising, the complexity arises from the immense size of the training data, and the dimensionality of the sparse feature space (both can be hundreds of billions). To tackle these challenges, we built TrainSparse (TS), a system that parallelizes the training of neural networks with a focus on efficiently handling large-scale sparse features. In this paper, we present the design and implementation of TS, and show the effectiveness of the system by applying it to predict the ad conversion rate (pCVR), one of the key problems in online advertising. We also compare several methods for dimensionality reduction on sparse features in the pCVR task. Experiments on real-world industry data show that TS achieves outstanding performance and scalability.