Federated Learning with Knowledge Distillation to Mitigate Catastrophic Forgetting and Data Heterogeneity in IoV Systems

Jiayu Wang, Jiechao Gao

Abstract: In Internet of Vehicles (IoV), intelligent transportation recognition is key to smart transportation systems. However, training models using data from individual vehicles often results in unreliable performance due to limited storage, leading to new data overwriting old samples. This approach suffers from catastrophic forgetting, low accuracy, and poor generalization. Current industry solutions, such as federated learning, incremental learning (iCaRL), and elastic weight consolidation (EWC), face limitations when dealing with the frequent, dynamic changes in IoV data and the heterogeneity of system structures.In this paper, we propose the FedHotpot framework based on federated learning to tackle these challenges. On the one hand, it employs a dual knowledge distillation strategy to ensure that knowledge learned by the iteratively updated model is not forgotten. On the other hand, we have optimized the model aggregation algorithm to mitigate the adverse effects of system and statistical heterogeneity arising from the diverse clients in the IoV environment on the federated learning architecture. We conducted multiple sets of comparative experiments on the proposed FedHotpot framework. The experimental results demonstrate that FedHotpot performs exceptionally well across multiple datasets, particularly when dealing with non-independently and identically distributed (non-IID) data, where its performance significantly outperforms traditional federated learning methods. These findings not only validate the effectiveness of the dual knowledge distillation strategy, but also underscore the pivotal role of optimized aggregation algorithms in enhancing the overall performance of federated learning.