From layer-wise pruning to layer-over global rank optimization for adaptive sparse and low rank based deep learning model compression

Donggyu Beom, Min-Yeong Park, Suhyung Park

Keywords: Model compression, low-rank, sparsity, layer-wise, layer-over.

TL;DR: From layer-wise to layer-over adaptive sparse and low rank based compression framework

Registration Requirement: Yes

Abstract: Deep learning has achieved substantial progress in biomedical image analysis, yet high computational and memory demands hinder deployment in resource-constrained clinical settings. While low-rank factorization via SVD is a prevalent compression strategy, conventional rank truncation introduces information loss that degrades performance under aggressive compression rates. In this work, we propose a layer-over rank optimization framework that decomposes layer-wise weight matrices into low-rank and sparse components while enabling adaptive rank allocation across layers under a unified parameter budget. Layer-wise optimization proceeds with principal component analysis (RPCA) that alternates between full-rank reconstruction and structured constraint enforcement by balancing reconstruction and task-specific losses to compensate for truncation-induced information loss. Additionally, global expansion of rank optimization controls layer-over flexible rank allocation, enabling dynamic distribution of representational capacity across layers. Extensive experiments on biomedical image analysis benchmarks confirm that the proposed method achieves superior accuracy and robustness over conventional low-rank compression methods, particularly under high compression rates.

Visa & Travel: Yes

Read CFP & Author Instructions: Yes

Originality Policy: Yes

Single-blind & Not Under Review Elsewhere: Yes

LLM Policy: Yes

Submission Number: 116