SC2 Benchmark: Supervised Compression for Split Computing

Yoshitomo Matsubara, Ruihan Yang, Marco Levorato, Stephan Mandt

Abstract: With the increasing demand for deep learning models on mobile devices, splitting neural network computation between the device and a more powerful edge server has become an attractive solution. However, existing split computing approaches often underperform compared to a naive baseline of remote computation on compressed data. Recent studies propose learning compressed representations that contain more relevant information for supervised downstream tasks, showing improved tradeoffs between compressed data size and supervised performance. However, existing evaluation metrics only provide an incomplete picture of split computing. This study introduces supervised compression for split computing (SC2) and proposes new evaluation criteria: minimizing computation on the mobile device, minimizing transmitted data size, and maximizing model accuracy. We conduct a comprehensive benchmark study using 10 baseline methods, three computer vision tasks, and over 180 trained models, and discuss various aspects of SC2. We also release our code and sc2bench, a Python package for future research on SC2. Our proposed metrics and package will help researchers better understand the tradeoffs of supervised compression in split computing.

Submission Length: Long submission (more than 12 pages of main content)

Changes Since Last Submission: We added a connection from Information Bottleneck, clarified our claims in the conclusion section, and fixed typos.

Video: https://youtu.be/uwwV_vAOvX4

Code: https://github.com/yoshitomo-matsubara/sc2-benchmark

Assigned Action Editor: ~Yanwei_Fu2

License: Creative Commons Attribution 4.0 International (CC BY 4.0)

Submission Number: 906