A Robust Aggregation of Federated Large Language Models for Multimodal Knowledge Discovery in Computational Social Systems

Jinhua Chen, Chinmay Chakraborty, Ashok Polavarapu, Yuning Qiu, Qibin Zhao, Osama Alfarraj, Keping Yu

Abstract: Amid a rapidly evolving information era, large-scale multimodal knowledge discovery in computational social systems emerges as a key research domain. Large language models (LLMs) play a crucial role in this field, providing contextual understanding and task adaptability. Yet, centralized training of LLM raises privacy concerns. Federated learning (FL) offers a distributed alternative, but it struggles with data heterogeneity and security issues related to model parameters. To this end, we propose a robust aggregation method that leverages the relative total distance of models to improve global model performance in heterogeneous settings, complemented by Cheon-Kim-Kim-Song (CKKS) encryption to secure parameters against parameter stealing without performance loss. Extensive numeric results show our approach excels in LLM testing, scoring 3.74 on MTBenchmark and 8.17 on Vicuna, outperforming state-of-the-art FL methods against data heterogeneity challenges. It also achieves consistent gains on image datasets such as SVHN, CIFAR10, MNIST, TinyImageNet200, and CIFAR100, TinyImageNet200. In summary, our method offers an effective solution for secure multimodal data analysis in computational social systems.

External IDs:dblp:journals/tcss/ChenCPQZAY25