Detecting Expert-Written Comments in Stack Exchange

Himani Musku, Alea Ritchie, Nour Jedidi, Rohan Leekha, Courtland VanDam

Abstract: Online question-answering platforms, such as Stack Exchange, have grown rapidly in recent years, making it necessary to identify the credibility of users and the information they share online to maintain trust within these communities. This issue can be addressed through accurate expert detection methods to determine whether or not a comment is written by an expert. For our study, we conducted analyses on a dataset consisting of various posts and comments written by over 10,000 Stack Exchange users to identify which classification techniques can most accurately distinguish between the written contributions of experts and non-experts. After comparing 12 methods across the six different communities that we analyzed in Stack Exchange (e.g, Physics), we discovered that GEMNET, our Generative AI based embedding approach, achieved the highest average F1-scores for experts. GEMNET also shows high generalization capabilities, achieving 17% higher F1-scores for experts compared to BERT, another top-performing model. Additionally, by testing GEMNET across different class imbalance ratios, we found that it helped maintain a high detection rate even when the class imbalance became more skewed to reality. Our results demonstrate GEMNET to be an improved expert detection method compared to traditional and newer classification techniques.

External IDs:dblp:conf/hpec/MuskuRJLV25