Generating Contextually-Relevant Navigation Instructions for Blind and Low Vision People

Zain Merchant, Abrar Anwar, Emily H. Wang, Souti Chattopadhyay, Jesse Thomason

Confirmation: I have read and agree with the workshop's policy on behalf of myself and my co-authors.

Keywords: Contextually-Relevant Navigation

Abstract:

Navigating unfamiliar environments presents significant
challenges for blind and low-vision (BLV) individuals.
In this work, we construct a dataset of images and goals across
different scenarios such as kitchens or outdoor navigation. We
then investigate how grounded instruction generation methods
can provide contextually-relevant navigational guidance to users
in these instances. Through a study involving sighted users, we
demonstrate that large pretrained language models can produce
correct and useful instructions perceived as beneficial for BLV
users. We also conduct a survey and interview with 4 BLV
users and observe useful insights on preferences for different
instructions based on the scenario.

Submission Number: 3