Position: Insights from Survey Methodology can Improve Training Data

Stephanie Eckman, Barbara Plank, Frauke Kreuter

Keywords: data collection, annotation, annotation sensitivity, survey methodology, training data

TL;DR: Findings from survey methodology can improve the quality and efficiency of training data collection for AI models, because both tasks involve humans providing responses to prompts and share challenges around accuracy and representativeness.

Submission Type: Non-Archival

Abstract:

Whether future AI models are fair, trustworthy,
and aligned with the public’s interests rests in part
on our ability to collect accurate data about what
we want the models to do. However, collecting
high-quality data is difficult, and few AI/ML re-
searchers are trained in data collection methods.
Recent research in data-centric AI has show that
higher quality training data leads to better per-
forming models, making this the right moment to
introduce AI/ML researchers to the field of survey
methodology, the science of data collection. We
summarize insights from the survey methodology
literature and discuss how they can improve the
quality of training and feedback data. We also
suggest collaborative research ideas into how bi-
ases in data collection can be mitigated, making
models more accurate and human-centric.

Submission Number: 13