A deep learning based bank card detection and recognition method in complex scenes

Hanyang Lin, Yongzhao Zhan, Shiqin Liu, Xiao Ke, Yuzhong Chen

Abstract: With the widespread use of mobile Internet, mobile payment has become a part of daily life, and bank card recognition in natural scenes has become a hot topic. Although printed character recognition has achieved remarkable success in recent years, bank card recognition is not limited to traditional printed character recognition. There are two types of bank cards: unembossed bank cards, such as most debit cards which usually use printed characters, and embossed bank cards, such as most credit cards which mainly use raised characters. Recognition of raised characters is very challenging due to its own characteristics, and there is a lack of fast and good methods to handle it. To better recognize raised characters, we propose an effective method based on deep learning to detect and recognize bank cards in complex natural scenes. The method can accurately recognize the card number characters on embossed and unembossed bank cards. First, to break the limitation that YOLOv3 algorithm is usually used for object detection, we propose a novel approach that enables YOLOv3 to be used not only for bank card detection and classification, but also for character recognition. The CANNYLINES algorithm is used for rectification and the Scharr operator is introduced to locate the card number region. The proposed method can satisfy bank card detection, classification and character recognition in complex natural scenes, such as complex backgrounds, distorted card surfaces, uneven illumination, and characters with the same or similar color to the background. To further improve the recognition accuracy, a printed character recognition model based on ResNet-32 is proposed for the unembossed bank cards. According to the color and morphological characteristics of embossed bank cards, raised character recognition model combining traditional morphological methods and LeNet-5 convolutional neural network is proposed for the embossed bank cards. The experimental results on the collected bank card dataset and bank card number dataset show that our proposed method can effectively detect and identify different types of bank cards. The accuracy of the detection and classification of bank cards reaches 100%. The accuracy of the raised characters recognition on the embossed bank card is 99.31%, and the accuracy of the printed characters recognition on the unembossed bank card reaches 100%.