Evaluating Academic Answer Quality: A Pilot Study on ResearchGate Q&A

Lei Li, Daqing He, Chengzhi Zhang

Abstract: Evaluating the quality of academic content on social media is a critical research topic for the further development of scholarly collaboration on the social web. This pilot study used the question/answer pairs of Library and Information Science (LIS) domain on ResearchGate to examine how scholars assess the quality of academic answers on the social web. This study aims to: (1) examine the aspects used by scholars in assessing the academic answer quality and identify the objective and subjective aspects; (2) future verify the existing of subjective aspects when judging the academic answers’ quality by detecting the agreement of evaluation between different evaluators. Though concluding the evaluation criteria of the academic content quality from the related works, the authors identified nine aspects of the quality evaluation and mapped the participants’ responds of the reasons for the answer quality judgment to the identified quality judgment framework. We found that aspects that related to the content of academic text and the users’ beliefs and preferences are the two common used aspects to judge the academic answer quality, which indicated that not only the text itself, but also the evaluator’s beliefs and preferences influence the quality judgment. Another finding is the agreement level between different evaluator’s judgments is very low, compared with other non-academic text judgment agreement level.