As companies are more and more leveraging the power of machine learning (ML), ML models, the data they were trained on, and the training pipelines themselves, are becoming increasingly important assets. Hence, a new set of tools is being developed, which aim to help manage the ML model lifecycle in a more structured way. As the model lifecycle involves a lot of experimental back-and-forth, reproducibility is an important aspect that such tools need to provide. However, as model lifecycle management is an emerging field, only few best practices on how to support reproducibility exist. This led to a large variety of tools being developed that all have the same goal of providing reproducibility but with major differences in how this goal is achieved. As a result, users are overwhelmed with having to navigate this vast tooling landscape and having to choose a tool that best fits their needs. This is a difficult task as the reproducibility capabilities of different tools can vary and users need to determine themselves, what is supported and what functionalities would fit their specific use case.

In this paper, our goal is to add structure to the process of deciding on a specific tool in terms of its reproducibility capabilities. We identify the most significant artifacts of the ML model lifecycle and, based on these, propose a generic classification framework that allows to assess the reproducibility capabilities of a specific tool and make it comparable to other tools. To evaluate our framework, we conduct an analysis of 12 popular ML lifecycle management tools. We study each tool in detail and classify it according to our framework. We then compare the tools to each other to determine what degree of reproducibility is provided in general and where reproducibility support is still lacking. While overall we find that the majority of tools offers most features that are required for full reproducibility, there still exist gaps in terms of automating reproducibility and cross-tool reproducibility. Based on these findings, we provide a set of research challenges which need to be addressed in order to better understand reproducibility and to fill the gaps of current systems.

ML Reproducibility Systems: Status and Research Agenda

Task pipelines are common in today’s embedded systems, as data moves from source to sink in sensing-processing-actuation task chains. A real-time task pipeline is constructed by connecting a series of periodic tasks with data buffers. In a time-critical system, end-to-end timing and data-transfer properties of a task pipeline must be guaranteed. A guarantee could be mathematically expressed by assigning constraints to the tasks of a pipeline. However, deriving task scheduling parameters to meet end-to-end guarantees is an NP-hard constraint optimization problem. Hence, a traditional constraint solver is not a suitable runtime solution.

In this paper, we present a heuristic constraint solver algorithm, CoPi, to derive the execution times and periods of pipelined tasks that meet the end-to-end constraints and schedulability requirements. We consider two upper bound constraints on a task pipeline: end-to-end delay and loss-rate. After satisfying these constraints, CoPi schedules a pipeline as a set of asynchronous and data independent periodic tasks, under the rate-monotonic scheduling algorithm. Simulations show that CoPi has a comparable pipeline acceptance ratio and significantly better runtime than open-source MINLP-solvers. Furthermore, we use CoPi to map multiple task pipelines to a multiprocessor system. We demonstrate that a partitioned multiprocessor scheduling algorithm coupled with CoPi accommodates dynamically appearing pipelines, while attempting to minimize task migrations.

Solution: End-to-end Scheduling of Real-time Task Pipelines on Multiprocessors

We identify new opportunities in video streaming, involving the joint consideration of offline video chunking and online rate adaptation. Due to a video's complexity varying over time, certain parts are more likely to cause performance impairments during playback with a particular rate adaptation algorithm. To address such an issue, we propose Segue, which carefully uses variable-length video segments, and augment specific segments with additional bitrate tracks. The key novelty of our approach is in making such decisions based on the video's time-varying complexity and the expected rate adaptation behavior over time.
We propose and implement several methods for such adaptation-aware chunking. Our results show that Segue substantially reduces rebuffering and quality fluctuations, while maintaining video quality delivered; Segue improves QoE by $9\%$ on average, and by $22\%$ in low-bandwidth conditions. Finally, we view our problem framing as a first step in a new thread on algorithmic and design innovation in video streaming, and leave the reader with several interesting open questions.

JSYS 2021 Nov Papers Submissions

ML Reproducibility Systems: Status and Research Agenda

Solution: End-to-end Scheduling of Real-time Task Pipelines on Multiprocessors

Prepare your video for streaming with Segue