CPET: Effective Parameter Efficient Tuning for Compressed Large Models

Anonymous

Abstract: In recent times, parameter-efficient tuning (PET) has been widely explored, as it tunes significantly fewer parameters than full-parameter fine-tuning (FT) while still stimulating sufficient knowledge from large language models (LLMs) for downstream tasks. Moreover, when adopting PET to serve multiple tasks, various tiny task-specific PET modules can be built on a frozen backbone LLM, avoiding redundantly deploying LLMs. Although PET methods have significantly reduced the cost of tuning and deploying LLMs, the inference still suffers from the computation bottleneck of the LLM. To address this issue, we build an effective PET framework based on compressed backbone LLMs, named ``CPET''. In CPET, we systematically evaluate the impact of mainstream compression techniques on the performance of PET modules, and then introduce knowledge inheritance and knowledge recovery to restore the knowledge loss caused by compressing the backbone LLM. Our experimental results demonstrate that, owing to the restoring strategies of CPET, collaborating task-specific PET modules with a compressed LLM can achieve comparable performance to collaborating with its non-compressed version, and significantly outperform directly applying FT or PET to the compressed LLM.

Paper Type: long

Research Area: Efficient/Low-Resource Methods for NLP