Human-in-the-Loop Mixup

Katherine M. Collins, Umang Bhatt, Ilia Sucholutsky, Bradley C. Love, Adrian Weller, Vihari Piratla, Weiyang Liu

Abstract:

Aligning model representations to humans has been found to improve robustness and
generalization. However, such methods often focus on standard observational data. Synthetic
data is proliferating and powering many advances in machine learning; yet, it is not always
clear whether synthetic labels are perceptually aligned to humans – rendering it likely model
representations are not human aligned. We focus on the synthetic data used in mixup: a
powerful regularizer shown to improve model robustness, generalization, and calibration. We
design a comprehensive series of elicitation interfaces, which we release as HILL MixE Suite,
and recruit 159 participants to provide perceptual judgments along with their uncertainties,
over mixup examples. We find that human perceptions do not consistently align with the
labels traditionally used for synthetic points, and begin to demonstrate the applicability of
these findings to potentially increase the reliability of downstream models, particularly when
incorporating human uncertainty. We release all elicited judgments in a new data hub we
call H-Mix.