Leveraging AutoML for Sustainable Deep Learning: A Multi- Objective HPO Approach on Deep Shift Neural Networks

TMLR Paper4526 Authors

Abstract:

Deep Learning (DL) has advanced various fields by extracting complex patterns from large
datasets. However, the computational demands of DL models pose environmental and
resource challenges. Deep Shift Neural Networks (DSNNs) present a solution by leveraging
shift operations to reduce computational complexity at inference. Compared to common
DNNs, DSNNs are still less well understood and less well optimized. By leveraging AutoML
techniques, we provide valuable insights into the potential of DSNNs and how to design them
in a better way. Since we consider complementary objectives such as accuracy and energy
consumption, we combine state-of-the-art multi-fidelity (MF) HPO with multi-objective
optimization to find a set of Pareto optimal trade-offs on how to design DSNNs. Our
approach led to significantly better configurations of DSNNs regarding loss and emissions
compared to default DSNNs. This includes simultaneously increasing performance by about
20% and reducing emissions, in some cases by more than 60%. Investigating the behavior
of quantized networks in terms of both emissions and accuracy, our experiments reveal
surprising model-specific trade-offs, yielding the greatest energy savings. For example, in
contrast to common expectations, selectively quantizing smaller portions of the network
with low precision is optimal while retaining or improving performance. We corroborated
these findings across multiple backbone architectures, highlighting important nuances in
quantization strategies and offering an automated approach to balancing energy efficiency
and model performance.

Submission Length: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Christopher_Mutschler1

Submission Number: 4526