  To squeeze out the performance in the post-Moore era, the compiler auto-tuning approach has been widely studied and productized. Despite its superior efficiency in compiler optimization problems, performance variance in final tuning output has long been an issue for search-based auto-tuning methods. It poses a challenge to research reproducibility and production stability. In general, the causes of such performance variance come from many aspects across different system layers. In addition to generic causes, we observe that auto-tuners add unique sources of variance, including the use of different search methods and cost models.
  In this work, we specifically focus on the performance variance originating from the nature of auto-tuning. Based on our observation, we set three major hypotheses on the search method, cost model, and hardware characteristics. Then, we validated our hypotheses through experiments with a production auto-tuner and a representative set of machine learning workloads. Our preliminary result suggests impactful factors to consider in future investigations.

Unveiling Source of Performance Variance on Search-based Compiler Optimization

Recent work introduced DiffTune, a neural network-based technique to automatically learn the microarchitecture-specific parameters of basic block CPU simulators. The authors apply their approach to the llvm-mca simulator. They show that the learned parameter values achieve an accuracy on the BHive benchmark suite that is comparable, and in some cases even better than that of the original, expert-provided llvm-mca parameter values.

In this paper, we show that an accuracy in this range is actually trivial to achieve: We propose a simple set of parameter values that outperforms the values learned by DiffTune. In fact, our set of parameter values is so simple that it can be fully described within this abstract: We set the dispatch width to 4, the reorder buffer size to 100, the latencies and μop counts of all instructions to 1, and all other parameters to 0. These parameter values lead to more accurate predictions than DiffTune's values on all four microarchitectures that were considered in the DiffTune paper.

We then develop a simple learning algorithm for the llvm-mca parameters. We show that the parameter values learned by our algorithm lead to an average error on the BHive benchmark suite that is between 29% and 47% lower compared to DiffTune's values.

DiffTune Revisited: A Simple Baseline for Evaluating Learned llvm-mca Parameters

In the recent years, cloud computing is going though a major transformation throughout its system stack, from its application to hardware. Its services are increasingly shifting from large monolithic applications to complex graphs with many single-purpose microservices, which offer many advantages in terms of deployment and development. On the other hand, cloud datacenters are becoming increasingly heterogeneous as they host more GPUs, FPGAs, and ASICs. While this heterogenous hardware can not only accelerate but also expand the capability of microservices, they further complicate the complex action space in microservices scheduling. Importantly, the convergence of the changes in both applications and hardware brings up unique challenges in datacenter scheduling for microservices. Recent innovations has shown that data-driven Machine Learning (ML) approaches leveraging neural networks can improve both end-to-end latency of the applications and probability of QoS violations. However, these works have focused on a rather homogeneous clusters which may become prohibitive as the scheduling problem gets more complex as the datacenters become heterogeneous. This paper first analyzes the potential limitations of the previous approaches and explores a new dimension of efficiency in the development of schedulers for microservices by incorporating a light-weight ML-based model. To this end, the paper develops a prototype light-weight ML-based scheduler dubbed Octopus that harbors a decision tree to efficiently schedule microservices on heterogeneous clusters. Comparisons against conventional scheduling techniques including First-Fit, Random, and Kubernetes-like schedulers show that Octopus provides up to 6.35x faster end-to-end latency.

Exploring Efficient ML-based Scheduler for Microservices in Heterogenous Clusters

Compilers are equipped with optimization passes that can be applied to improve the quality of a program. The selection and ordering of these passes is a classic NP-hard problem known as the phase ordering problem. Traditionally, compilers use expert-picked sequences to optimize for performance (e.g. -O2, -O3), or code size (e.g. -Os, -Oz). However, not all programs respond positively to all optimizations, and prior work has shown that these expert-picked phase orders can be outperformed by tailoring the phase order decisions to individual programs.
In this work, we propose further specializing the phase ordering for each function. We investigate the impact of various function and module passes and show that function-specific phase ordering provides an extra 2.3% improvement in code size reduction compared to program-specific phase ordering using a cheap search budget. Compared to using the Oz flag, the deep reinforcement learning method achieves up to 9% more code size reduction in the same dataset, and up to 6% improvement when transferring to new unseen data. Our exhaustive search experiment shows that searching on different levels of abstraction will be beneficial for solving the phase ordering problem. However, we note several limitations with our reinforcement learning approach, the observation features are not sufficient to give better generalization, the reward has high variance and the datasets are not well representative for all programs.

Autophase V2: Towards Function Level Phase Ordering Optimization

Modern state-of-the-art deep learning (DL) applications tend to scale out to a large number of parallel GPUs. Unfortunately, we observe that the collective communication overhead across GPUs is often the key limiting factor of performance for distributed DL. It under-utilizes the networking bandwidth by frequent transfers of small data chunks, which also incurs a substantial I/O overhead on GPU that interferes with computation on GPU. The root cause lies in the inefficiency of CPU-based communication event handling as well as the inability to control the GPU's internal DMA engine with GPU threads.

To address the problem, we propose a GPU-driven code execution system that leverages a GPU-controlled hardware DMA engine for I/O offloading. Our custom DMA engine pipelines multiple DMA requests to support efficient small data transfer while it eliminates the I/O overhead on GPU cores. Unlike existing GPU DMA engines initiated only by CPU, we let GPU threads to directly control DMA operations, which leads to a highly efficient system where GPUs drive their own execution flow and handle communication events autonomously without CPU intervention. Our prototype DMA engine achieves a line-rate from a message size as small as 8KB (3.87x better throughput) with only 4.32us of communication latency (9.1x faster) while it incurs little interference with computation on GPU, achieving 1.82x higher all-reduce throughput in a real training workload.

ISCA 2022 Workshop MLArchSys Submissions

Unveiling Source of Performance Variance on Search-based Compiler Optimization

DiffTune Revisited: A Simple Baseline for Evaluating Learned llvm-mca Parameters

Exploring Efficient ML-based Scheduler for Microservices in Heterogenous Clusters

Autophase V2: Towards Function Level Phase Ordering Optimization

Towards GPU-driven Code Execution for Distributed Deep Learning