SIGHT: Single-Image Conditioned Generation of Hand Trajectories for Hand-Object Interaction

Alexey Gavryushin, Florian Redhardt, Gaia Di Lorenzo, Luc Van Gool, Marc Pollefeys, Kaichun Mo, Xi Wang

Keywords: Human Motion Generation, Human-Object Interaction, Diffusion Models

Abstract: We introduce a novel task of generating realistic and diverse 3D hand trajectories given a single image of an object, which could be involved in a hand-object interaction scene or pictured by itself. When humans reach for an object, appropriate trajectories naturally form to manipulate it for specific tasks in our minds. Such hand-object interaction trajectory priors can greatly benefit applications in robotics, embodied AI, augmented reality and related fields. To tackle this challenging problem, we propose the SIGHT-Fusion system, consisting of a carefully curated pipeline for extracting features at various levels of hand-object interaction details from the single image input, and a conditional motion generation diffusion model processing the extracted features. We train our method given video data with corresponding hand trajectory annotations, without supervision in the form of action labels. For the evaluation, we establish benchmarks utilizing the FPHAB and HOI4D datasets, testing our method against various baselines and metrics. We also introduce task simulators for executing the generated hand trajectories and reporting task success rates as an additional metric. Experiments show that our method generates more natural and diverse hand trajectories than baselines and presents promising generalization capability on unseen objects. The accuracy of the generated hand trajectories is confirmed in a physics simulation setting, showcasing the authenticity of the created sequences and their applicability in downstream uses.

Primary Area: generative models

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 8107