Designing Cross-Race Tests for Forensic Facial Examiners, Super-recognizers, and Face Recognition Algorithms

Géraldine Jeckeln, Selin Yavuzcan, Kate A. Marquis, Prajay Sandipkumar Mehta, Amy N. Yates, P. Jonathon Phillips, Alice J. O'Toole

Abstract: Humans and machines vary in the accuracy with which they recognize faces of different races. This can impact the fairness of face identification in security and forensic settings. We introduce a protocol for designing a cross-race face identification test for evaluating people (e.g., forensic facial examiners, super-recognizers) and machines with superior face-identification ability. We followed this protocol to create a cross-race test and report the test's benchmarks on untrained human participants and two state-of-the-art face recognition algorithms. The goal of the protocol is to select a relatively small number of challenging test items (facial image comparisons) of two races, with approximately equally challenging items of both races. Item selection consisted of pre-screening with an open-source face recognition algorithm, followed by a second round of prescreening using the performance of untrained human participants. We sampled face-images (Black and White identities) from a large biometric data set and applied the protocol to assemble face comparisons. The protocol yielded a cross-race test with 20 comparison pairs portraying Black and White identities (10 same-identity; 10 different-identity). Untrained participants (54 Black; 51 White) judged whether face-image pairs showed the same or different identities using a 7-point scale. By design, the test proved challenging for untrained participants, with performance comparable across Black and White image pairs for both Black and White participants. Two top-performing face recognition systems from the Face Recognition Vendor Test-ongoing [6] scored perfectly (no errors) on both Black and White face-image pairs from the Cross-Race Test. The human and machine benchmarks established here make this test ideal for evaluating cross-race face recognition bias in people with high levels of skill and training.