Go to DMLR homepageDecordFace: A Framework for Degraded and Corrupted Face Recognition
Abstract: Face recognition (FR) models have become an integral part of day-to-day activities involving surveillance and biometric verification. While these models perform remarkably well in constrained settings, the performance is limited in the presence of certain challenging covariates. One such covariate is the presence of unforeseen image degradations and corruptions. These degradations, which inevitably occur during image acquisition, transmission, or storage, substantially impact real-world applicability. In order to analyze the performance of FR systems in these scenarios, we provide the first-ever Degraded and Corrupted Face Recognition (DecordFace) framework to evaluate the robustness of FR models. Corrupted versions of multiple standard datasets are created, and experiments are performed using more than 3.6 million corrupted face images with over 25 recognition models with different architectures and backbones, using 16 corruptions at 5 severity levels. For quantitative estimation of the impact of corruption, we introduce two novel evaluation metrics, error-based mVCE and embedding-based mCEI. Using these metrics and a cohort of FR models, we conduct a detailed analysis of model robustness under different model and input parameters. We observe a severe drop in the performance of models for unconstrained face recognition with performance errors over 20\% across different corruptions. The performance of model variants with shallow backbones is observed to suffer even more. The code for the DecordFace framework can be accessed at {\url{https://github.com/IAB-IITJ/DecordFace}}.
Keywords: Face recognition, Image Quality, Corruption, Framework, Corruption Metrics
Previous DMLR Submission Url: https://openreview.net/forum?id=W6nHAsxl3R
Changes Since Last Submission: We would like to sincerely thank the action editor, jnhC, and reviewers W1CT, MhTE, zPwY, and 4pWo for their detailed and thoughtful feedback. Their valuable insights have significantly contributed to the improvement of our manuscript. The strengths of our work, as appreciated by the reviewers, include:
- **Comprehensive Experimental Evidence**: Our work presents a thorough evaluation of multiple face recognition models on a variety of corrupted datasets. The experiments span multiple types of corruptions, including noise, blur, light/color distortions, and occlusions, providing a robust assessment of model performance across real-world conditions.
- **Novel Metrics**: We introduced the mVCE and mCEI metrics to quantify the impact of image corruptions on model performance. These metrics provide a new and insightful way to assess robustness, offering an alternative to traditional performance measures in face recognition tasks.
- **Framework for Robustness Evaluation**: DecordFace is a unique framework designed to evaluate the robustness of face recognition systems in the presence of different types of image corruptions. This framework enables standardized testing and comparison, helping the community assess how well models can handle real-world variations.
- **Fairness and Inclusivity**: Our work includes a fairness analysis across multiple demographic factors, such as ethnicity and age, highlighting how these factors influence model performance under corruption. This aspect is critical for ensuring that face recognition systems are inclusive and fair across diverse populations.
In response to the reviewers' comments, we have made the following revisions:
1. **Data Availability and Release**
- Included a link to the public GitHub repository for the project. The code for the DecordFace framework can be accessed at https://github.com/IAB-IITJ/DecordFace.
- Added the Dataset Datasheet to the GitHub repository for transparency.
- Updated the Broader Impact Statement to include recommendations for mitigating risks in face recognition technologies, including the need for measuring robustness at deployment time and across demographic subgroups.
2. **Additional Experiments**
- Added Section 5.3 (Quantifying mVCE on Real Samples) and Table 4 to highlight how performance degradations in real images mimic those observed in synthetic images. We performed experiments on the CelebA dataset to illustrate how observations made using synthetic noise transfer to real-world conditions. The results are provided in Section 5.3 (Pages 21-22).
- Introduced fairness analysis across ethnicity and age in Section 5.5, with visual representation in Figure 11.
- Added Section 5.6 to analyze the performance of the occlusion-based algorithm FROM on DecordFace. Using the FROM algorithm proposed by Qiu et al., we evaluated the AgeDB30 and CFP-FP variants from DecordFace. The results indicate that the FROM algorithm suffers from performance degradation in the presence of corruptions, particularly for the AgeDB dataset, which contains age variations.
3. **Writing and Details**
- Reframed the paper to describe DecordFace as a valuable framework (instead of a benchmark) to assist in the development of robust models.
- Added Algorithm 1 to outline the steps for obtaining corrupted versions of datasets for DecordFace.
- Provided more details regarding the metrics mVCE and mCEI.
- Explained the diversity of the datasets utilized in the framework for evaluation. These datasets account for variations in age, pose, and overall face image quality, reflecting real-world conditions.
- Added a Discussion section (Section 6) offering potential solutions for the identified problems. We provide insights into improving model performance, based on existing work and our understanding of face recognition models.
We have updated the revised manuscript to include the new experiments (e.g., CelebA, FROM evaluations, fairness analysis), enhanced documentation (e.g., GitHub repository, pseudocode), and clarified the framework’s role as a robustness evaluation tool.
Changes Since Previous Publication: N/A
Code: https://github.com/IAB-IITJ/DecordFace
Assigned Action Editor: ~Sergio_Escalera1
Submission Number: 115
Loading