SKILLBERT: “SKILLING” THE BERT TO CLASSIFY SKILLS!

Anonymous

Keywords: SkillBERT, BERT, recruitment, skills, XGBoost, Bi-LSTM, and Electronic Recruitment Records (ERRs)

TL;DR: SkillBERT is BERT-based model trained on ERRs to augment skill-matching between candidates and job descriptions

Abstract: In the age of digital recruitment, job posts can attract a large number of applications, and screening them manually can become a very tedious task. We propose a BERT-based model, SkillBERT, the embeddings of which are used as features for classifying skills into groups referred to as “competency groups”. A competency group is a group of similar skills and it is used as matching criteria (instead of matching on skills) for finding the overlap of skills between the candidates and the jobs. This proxy match takes advantage of the BERT’s capability of deriving meaning from the structure of competency groups present in the skill dataset. The problem that we are trying to solve is a multi-label classification problem, as a single skill can belong to multiple competency groups. To solve multi-label competency group classification using binary classifier, we have paired each skill with each competency group and tried to predict the probability of that skill belonging to that particular competency group. SkillBERT, which is trained from scratch on the skills present in job requisitions, is shown to be better performing than the pre-trained BERT (Devlin et al., 2019) and the Word2Vec (Mikolov et al., 2013). We have also explored K-means clustering (Lloyd, 1982) and spectral clustering (Chung, 1997) on SkillBERT embeddings to generate cluster-based features. Both algorithms provide similar performance benefits. Last, we have experimented with different classification models like Random Forest (Breiman, 2001), XGBoost (Chen & Guestrin, 2016), and a deep learning algorithm Bi-LSTM (Schuster & Paliwal, 1997; Hochreiter & Schmidhuber, 1997) for the tagging of competency groups to skill. We did not observe a significant performance difference among the algorithms, although XGBoost and Bi-LSTM perform slightly better than Random Forest. The features created using SkillBERT are most predictive in the classification task, which demonstrates that the SkillBERT is able to capture the information about the skills’ ontology from the data. We have made the source code, the trained models and the dataset (Electronic Recruitment Records, referred to as ERRs)1 of our experiments publicly available. ERRs are stored in the form of tables in our recruitment database.