When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels

Anonymous

Abstract: Deployed dialogue agents have the potential to integrate human feedback to continuously improve themselves. However humans may not always provide explicit signals when the conversational agent makes mistakes during interaction. In this work, we propose JUICER, a framework to make use of both binary and free-form textual human feedback. It works by: (i) extending sparse binary feedback by training a satisfaction classifier to label the unlabeled data; and (ii) training a reply corrector to map the bad replies to good ones. We find that augmenting training with model-corrected responses improves the final dialogue model, and we can further improve performance by using both positive and negative responses through the recently proposed DIRECTOR model.

Paper Type: long

Research Area: Dialogue and Interactive Systems