Federated Morozov Regularization for Shortcut Learning in Privacy Preserving Learning with Watermarked Image Data
Abstract: Federated learning is a promising privacy-preserving learning paradigm in which multiple clients can collaboratively learn a model with their image data kept local. For protecting data ownership, personalized watermarks are usually added to the image data by each client. However, the introduced watermarks can lead to a shortcut learning problem, where the learned model performs predictions over-rely on the simple watermark-related features and represents a low accuracy on real-world data. Existing works assume the central server can directly access the predefined shortcut features during the training process. However, these may fail in the federated learning setting as the shortcut features of the heterogeneous watermarked data are difficult to obtain.
In this paper, we propose a federated Morozov regularization technique, where the regularization parameter can be adaptively determined based on the watermark knowledge of all the clients in a privacy-preserving way, to eliminate the shortcut learning problem caused by the watermarked data. Specifically, federated Morozov regularization firstly performs lightweight local watermark mask estimation in each client to obtain the locations and intensities knowledge of local watermarks. Then, it aggregates the estimated local watermark masks to generate the global watermark knowledge with a weighted averaging. Finally, federated Morozov regularization determines the regularization parameter for each client by combining the local and global watermark knowledge. With the regularization parameter determined, the model is trained as normal federated learning. We implement and evaluate federated Morozov regularization based on a real-world deployment of federated learning on 40 Jetson devices with real-world datasets. The results show that federated Morozov regularization improves model accuracy by 11.22\% compared to existing baselines.
Primary Subject Area: [Content] Media Interpretation
Secondary Subject Area: [Experience] Multimedia Applications, [Engagement] Summarization, Analytics, and Storytelling, [Systems] Systems and Middleware
Relevance To Conference: With the growth of applying multimedia technology to commercial applications, concerns about user data privacy have increased, and research on privacy-preserving learning has come into being. Federated learning emerges as a promising privacy-preserving learning paradigm, where multiple clients can collaboratively learn a model without exposing their private data to the central server. Federated learning has been widely adopted in many multimedia applications such as medical image classification, anomaly detection in public safety surveillance, and sentiment analysis in social media content.
For data ownership identification and copyright protection, digital watermarking technologies are developed and applied in many multimedia applications, through adding the well-designed digital watermark into the image data by the data owner. Training models with the watermarked data may lead to the shortcut learning problem, that is the learned model makes predictions based on the simple shortcut features in the training data, rather than learning the underlying core features, and presents a good performance on the training dataset but decreased model accuracy on the unseen data. For example, in medical image classification, a model detects pneumonia in chest X-rays relying on watermarks that represent which hospital the patient was seen instead of lung pathophysiology used by a radiologist.
Submission Number: 4209
Loading