SSDStreamer: Specializing I/O Stack for Large-Scale Machine Learning

Jonghyun Bae, Hakbeom Jang, Jeonghun Gong, Wenjing Jin, Shine Kim, Jaeyoung Jang, Tae Jun Ham, Jinkyu Jeong, Jae W. Lee

Abstract: This article presents SSDStreamer, an SSD-based caching system for large-scale machine learning. By using DRAM as stream buffer, instead of an upper-level cache, SSDStreamer significantly outperforms state-of-the-art multilevel caching systems on Apache Spark, while requiring much less DRAM capacity.