Unveiling Source of Performance Variance on Search-based Compiler Optimization

Pranati Modumudi, Xiyou Zhou, Sunghyun Park

Keywords: Compiler Optimization, Machine Learning, Performance Variance, Cost Model

Abstract:

To squeeze out the performance in the post-Moore era, the compiler auto-tuning approach has been widely studied and productized. Despite its superior efficiency in compiler optimization problems, performance variance in final tuning output has long been an issue for search-based auto-tuning methods. It poses a challenge to research reproducibility and production stability. In general, the causes of such performance variance come from many aspects across different system layers. In addition to generic causes, we observe that auto-tuners add unique sources of variance, including the use of different search methods and cost models.
In this work, we specifically focus on the performance variance originating from the nature of auto-tuning. Based on our observation, we set three major hypotheses on the search method, cost model, and hardware characteristics. Then, we validated our hypotheses through experiments with a production auto-tuner and a representative set of machine learning workloads. Our preliminary result suggests impactful factors to consider in future investigations.

TL;DR: This work is the preliminary work to attack the performance variance problem in the popular search-based optimization methods.