Poster: Module Lightweighting and Path Transferring in Vision-Language Models for Efficient Edge Deployment

Jiahua Bao, Zhipeng Wang, Ziqian Li, Jiaxing Du, Jialiang Yan, Jie Liu

Abstract: We propose an efficient lightweight fine-tuning method that simplifies model design and reduces parameters, focusing on optimizing Visual-Language Models (VLMs) for edge deployment. As VLMs evolve, the parameter size becomes increasingly challenging for edge devices. To overcome this limitation, we combine lightweighting and fine-tuning into a single step. We decompose large linear layers in the vision encoder and introduce smaller matrices in parallel, creating a new path.During fine tuning, performance is improved by reducing the matrix size and increasing the depth, gradually phasing out the original path. We deploy the lightened and fine-tuned model on a Jetson TX2 and shows comparable performance compared to VLMs with larger parameters.