IntE: Quantitative Framework for Qualitative Data Evaluation via Distributional Mining

Wenshuo Zhang, Leixian Shen, Qiqi ZHOU, Shuchang Xu, Feiyang Hao, Huamin Qu, Zhao CHEN, Lin-Ping Yuan

Keywords: Qualitative Data Evaluation, Semi-Structured Questions, Content-Aware Dissimilarity Extraction, Multi-Agent System

TL;DR: IntE is a novel framework that quantitatively evaluates qualitative response datasets by leveraging cluster and demographic distributions, enabling holistic and consistent quality assessments for uncovering general patterns and unique insights.

Abstract:

Evaluating the quality of qualitative datasets containing responses collected for semi-structured questions is a persistent challenge. Manual analysis is slow and subjective, while existing automated methods lack a holistic, dataset-level perspective crucial for mining insights. We introduce IntE, a novel framework for the quantitative assessment of qualitative response datasets. IntE evaluates dataset quality using the cluster distributions based on collected responses and the predefined demographic distributions based on user metadata. IntE is structured into a four-quadrant assessment that quantifies the potential of a dataset for revealing general patterns and unique insights. The four quadrants rely on the distributions reconstructed via metadata and intra-data distances. Therefore, we propose a content-aware multi-agent system that accurately computes inter-response dissimilarity. This system features a two-stage adversarial framework for generating domain-specific evaluation instructions and an adaptive anchor algorithm to ensure scoring consistency.
We validate IntE through controlled experiments on synthetic data, highlighting the effectiveness of its components. Additionally, a real-world social survey case study, validated by domain experts, demonstrates IntE's capability to enhance knowledge discovery by accurately evaluating dataset quality and identifying key responses for analysis.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 5649