SysML: On System and Algorithm Co-design for Practical Machine Learning

Eric Xing

Abstract: The rise of Big Data and AI computing has led to new demands for Machine Learning systems to learn complex models with millions to billions of parameters that promise adequate capacity to digest massive datasets and offer powerful and real-time predictive analytics thereupon. In this talk, I discuss a recent trend toward building new distributed frameworks for AI at massive scale known as "system and ML algorithm co-design", or SysML -- system designs are tailored to the unique properties of ML algorithms, and algorithms are re-designed to better fit into the system architecture. I show how one can explore the underlying statistical and algorithmic characteristics unique to ML programs but not typical in traditional computer programs in designing the system architecture to achieve significant, universal, and theoretically sound power-up of ML program across the board. I also present a briefly introduction of the Petuum system based on such interdisciplinary innovations, which intends to dramatically improve adoption of AI solutions by lowering the barrier of entry to AI technologies via Automatic Machine Learning through Petuum. I show how, through automatable, product-grade, hardware-agnostic, standardized building blocks that can be assembled and customized, AI users can liberate themselves from the demanding experience of algorithm programming and system tuning, and easily experiment with different AI methods, parameters, and speed/resource trade-offs by themselves or automatically. To put this in a broader context, recent discussions about AI in both research community, and the general public have been championing a novelistic view of AI, that AI can mimic, surpass, threaten, or even destroy mankind. And such discussions are fueled by mainly recent advances in deep learning experimentations and applications, which are however often plagued by its craftiness, un-interpretability, and poor generalizability. I will discuss a different view of AI as a rigorous engineering discipline and as a commodity, where standardization, modularity, repeatability, reusability, and transparency are commonly expected, just as in civil engineering where builders apply principles and techniques from all sciences to build reliable constructions. I will discuss how such a view sets different focus, approach, metric, and expectation for AI research and engineering, which we practiced in our SysML work. Supplemental Material video https://acm-prod-streaming.literatumonline.com/3219819.3219934/980d3672-108b-4769-b585-a3027cc600e1/xing_automatic_machine.,180,300,750,964,1500,.mp4.m3u8?b92b4ad1b4f274c70877518315abb28be831d54738a81f1de54388f7ee05e6ea71ace279646dd94550fcc09d3fbca6d0a894412d8608a937cb27e892006e3cbf69eae44d61176e8396d486559edeb4035c2da4e2360ff8fab85c044a0a5610eca1a92a31b2 application/x-mpegurl flowplayer 00:00 00:00 space play / pause q unload | stop f fullscreen shift + ← → slower / faster ↑ ↓ volume m mute ← → seek . seek to previous 1 2 … 6 seek to 10%, 20% … 60% © 2015 About Flowplayer GPL based license mp4 650.5 MB xing_automatic_machine.mp4 mp4 650.5 MB Play stream Download