KDFAS: Multi-stage Knowledge Distillation Vision Transformer for Face Anti-spoofing

Jun Zhang, Yunfei Zhang, Feixue Shao, Xuetao Ma, Daoxiang Zhou

Abstract: With the commercial application of face recognition systems, face anti-spoofing has been studied extensively to enhance security in recent years. In this work, a lightweight network via knowledge distillation for face anti-spoofing is proposed. The main innovations of our approach are threefold: (1) In convolutional neural network based knowledge distillation, the local receptive field of teacher network may be inconsistent with that of student network, which results in misguiding. In our method, vision transformer architecture is leveraged because of its global modeling capabilities. (2) Beyond conventional decision-level knowledge transfer in the classification step via kullback-leibler loss, we present multi-stage feature-level knowledge distillation strategy to guide the feature learning of student network which can transfer richer knowledge from teacher to student network. (3) In contrast to traditional projection head learning, we construct a covariance matrix to solve the embedding dimensionality mismatching problem between teacher and student network in middle layers. Compared to teacher model of 1.28 GB, the memory of student model is only 330.8 MB, which effectively achieves a trade-off between memory and accuracy. Extensive experiments on three standard benchmarks demonstrate the superiority of our proposed method, which evidently corroborates the significance of multi-stage knowledge distillation for face anti-spoofing.