LESS: LEARNING TO SELECT A STRUCTURED ARCHITECTURE OVER FILTER PRUNING AND LOW-RANK DECOMPOSITION

Moonjung Eo, Suhyun Kang, Wonjong Rhee

Keywords: Hybrid compression, Low-rank compression, Filter pruning, Low-resource budget

TL;DR: The paper introduces LeSS, a novel DNN design determining filters for pruning and ranks for low-rank decomposition, offering an integrated, non-iterative approach within resource constraints.

Abstract: Designing a deep neural network (DNN) for efficient operation in low-resource environments necessitates strategic application of compression techniques. Filter pruning and low-rank decomposition stand out as two prominent methods employed for DNN compression. While these techniques possess complementary properties, their integration has been only partially explored, resulting in limited reported gains thus far. In this study, we present a novel fully joint learning algorithm named LeSS, aiming to concurrently determine filters for filter pruning and ranks for low-rank decomposition. Unlike previous methods, LeSS simultaneously determines both filters and ranks, eliminating the need for iterative or heuristic processes. Notably, LeSS adheres strictly to the specified resource budget constraint, ensuring practical applicability in resource-constrained scenarios. LeSS outperforms state-of-the-art methods on a number of benchmarks demonstrating its effectiveness.

Submission Number: 9