Unsupervised detection of random responding for Likert-type inventories with varying numbers of response categories

Michael John Ilagan, Carl F. Falk

DOI: 10.64028/fnzx384888

Keywords: Likert, random responding, careless responding, permutation test, classification, unsupervised learning

TL;DR: An algorithm for detecting Likert random responding required all items to have the same number of categories, so we removed that requirement.

Abstract: Likert-type inventories administered online risk “random” responding, such as by bots. To safeguard data quality, we consider unsupervised classification of random vs. non-random responders. Previous work proposed L1P1, an algorithm based on a permutation test with bias-corrected outlier statistics. L1P1 successfully calibrates sensitivity, assuming that for random responders, exchangeability holds for the entire response vector. However, when the items do not have the same “point-scale” (i.e., number of response categories), the same assumption is inapplicable. To extend the L1P1 classifier to inventories with multiple point-scales, we propose generating the null distribution by permuting only within subvectors of the same point-scale (PWP), otherwise following the original L1P1. Such a proposal is in contrast to doing a permutation test per point-scale then combining multiple p-values into a final predicted class. In a simulation study, the main findings were twofold. First, the proposed approach generally outperformed alternatives considered in terms of sensitivity calibration and accuracy. Second, p-values from point-scales with few items failed to calibrate sensitivity to the nominal 95% rate. PWP is implemented in the R package detranli, available on Github.

Submission Number: 7