Facial Composite Generation with Iterative Human FeedbackDownload PDF

Published: 20 Oct 2022, Last Modified: 05 May 2023Gaze Meets ML 2022 OralReaders: Everyone
Keywords: gaze, mental image reconstruction, human-ai collaboration, interactive system
TL;DR: We propose a system where humans and AI collaberate through gaze and visual feedback to reconstuct the user's mental image.
Abstract: We propose the first method in which human and AI collaborate to iteratively reconstruct the human's mental image of another person's face only from their eye gaze. Current tools for generating digital human faces involve a tedious and time-consuming manual design process. While gaze-based mental image reconstruction represents a promising alternative, previous methods still assumed prior knowledge about the target face, thereby severely limiting their practical usefulness. The key novelty of our method is a collaborative, iterative query engine: Based on the user's gaze behaviour in each iteration, our method predicts which images to show to the user in the next iteration. Results from two human studies (N=12 and N=22) show that our method can visually reconstruct digital faces that are more similar to the mental image, and is more usable compared to other methods. As such, our findings point at the significant potential of human-AI collaboration for reconstructing mental images, potentially also beyond faces, and of human gaze as a rich source of information and a powerful mediator in said collaboration.
Submission Type: Full Paper
Travel Award - Academic Status: Ph.D. Student
Travel Award - Institution And Country: University of Stuttgart, Germany
Travel Award - Low To Lower-middle Income Countries: No, my institution does not qualify.
Camera Ready Latexfile: zip
5 Replies