Effects of Collaboration on the Performance of Interactive Theme Discovery Systems

ACL ARR 2026 January Submission10744 Authors

Keywords: NLP tools for social analysis, human-computer interaction, human-in-the-loop, human-AI interaction, user-centered design, human-centered evaluation

Abstract: NLP-assisted solutions have gained considerable traction to support qualitative data analysis. However, no unified evaluation framework exists which can account for the many different settings in which qualitative researchers may employ them. In this paper, we propose an evaluation framework to study the way collaboration settings may produce different outcomes across a variety of interactive systems. Specifically, we study the impact of synchronous vs. asynchronous collaboration using three different NLP-assisted qualitative research tools and present a comprehensive analysis of significant differences in the consistency, cohesiveness, and correctness of their outputs.

Paper Type: Long

Research Area: Human-AI Interaction/Cooperation and Human-Centric NLP

Research Area Keywords: NLP tools for social analysis, human-computer interaction, human-in-the-loop, human-AI interaction, user-centered design, human-centered evaluation

Contribution Types: Model analysis & interpretability, Reproduction study, Data analysis

Languages Studied: English

Submission Number: 10744