Post-Deployment Fine-Tunable Semantic Communication

Peiyuan Si, Renyang Liu, Liangxin Qian, Jun Zhao, Kwok-Yan Lam

Abstract: Semantic communication (SemCom) is an emerging way that aims to improve communication efficiency based on the semantics of content, which relies on the knowledge base (KB) and is usually dedicated to specific tasks or datasets. To improve the adaptability of SemCom systems on unknown datasets, we propose a post-deployment Fine-Tunable Semantic Communication (FTSC) system for image transmission. Towards an adaptive and efficient SemCom system, our research consists of the framework design of FTSC and its system optimization study. Firstly, the generalizability study is conducted based on a two-layer hierarchical vector quantized-variational autoencoder (VQ-VAE-2). Unlike traditional SemCom that can work on limited pretrained datasets, FTSC adapts to varied input data post-deployment, enhancing practicality in diverse communication scenarios. This system incorporates two novel fine-tuning methods: Decoder Fine-Tuning (DFT) and Latent Space-based Decoder Fine-Tuning (LSDFT). DFT updates the decoder for new images post-deployment without transmitting gradients, while LSDFT eliminates the need for raw image transmission during fine-tuning. Secondly, we study the system optimization of the proposed FTSC framework to improve the efficiency of communication resource allocation with the concern of recovery quality, time delay, and energy cost in downlink transmissions. Extensive experiments demonstrate the superiority of FTSC over Joint Photographic Experts Group (JPEG) and Joint Source-Channel Coding (JSCC) across various datasets and noise levels, and both DFT and LSDFT significantly enhance image recovery on unfamiliar datasets compared to pre-trained models.

External IDs:dblp:journals/twc/SiLQZL25