FedMMA-HAR: Federated Learning for Human Activity Recognition With Missing Modalities Using Head-Worn Wearables

Alessandro Gobbetti, Martin Gjoreski, Hristijan Gjoreski, Nicholas D. Lane, Marc Langheinrich

Abstract: Regular physical activity benefits physical and mental health, making it essential for overall population health. Wearable devices using data from multiple modalities (e.g., accelerometers and gyroscopes), combined with machine learning (ML) insights, help improve exercise regimens. However, continuous data collection and personal data processing raise privacy concerns. Federated learning (FL) mitigates these by keeping data on user devices, training ML models locally, and sharing only model updates, but its applicability to human activity recognition is limited. We propose the missing modality agnostic (MMA) method, which makes FL robust to missing data, enabling accurate output even if some modalities are missing. Experimental results on two datasets (OCOsense Smart Glasses and USI-HEAR Earbuds) show that MMA-based models are effective with feature-based ML and deep learning and compatible with centralized ML and FL. This work fills a critical gap in current FL methodologies, ensuring robustness under imperfect data conditions, common in real-world scenarios.