Can Semi-Supervised Learning Improve Prediction of Deep Learning Model Resource Consumption?

NeurIPS 2023 Workshop MLSys Submission15 Authors

Keywords: Performance Model, Deep Learning, Graph Neural Network, Semi-Supervised Learning

Abstract: With the increasing computational demands of Deep Learning (DL), predicting training characteristics like training time and memory usage is crucial for efficient hardware allocation. Traditional methods rely solely on supervised learning for such predictions. Our work integrates a semi-supervised approach for improved accuracy. We present TraPPM, which utilizes a graph autoencoder to understand representations of unlabeled DL graphs, then combined with a supervised graph neural network training to predict the metrics. Our model significantly surpasses standard methods in prediction accuracy, with MAPE values of 9.51\% for training step time and 4.92\% for memory usage. The code and dataset are available at https://github.com/karthickai/trappm

Submission Number: 15