PopSign ASL v1.0: An Isolated American Sign Language Dataset Collected via Smartphones

Thad Starner, Sean Forbes, Matthew So, David Martin, Rohit Sridhar, Gururaj Deshpande, Sam Sepah, Sahir Shahryar, Khushi Bhardwaj, Tyler Kwok, Daksh Sehgal, Saad Hassan, Bill Neubauer, Sofia Anandi Vempala, Alec Tan, Jocelyn Heath, Unnathi Utpal Kumar, Priyanka Vijayaraghavan Mosur, Tavenner M. Hall, Rajandeep Singh et al. (4 additional authors not shown)

Keywords: American Sign Language, gesture, dataset

TL;DR: Introduces PopSign ASL v1.0: a public isolated American Sign Language dataset of ~200,000 examples of 250 signs

Abstract:

PopSign is a smartphone-based bubble-shooter game that helps hearing parents
of deaf infants learn sign language. To help parents practice their ability to sign,
PopSign is integrating sign language recognition as part of its gameplay. For
training the recognizer, we introduce the PopSign ASL v1.0 dataset that collects
examples of 250 isolated American Sign Language (ASL) signs using Pixel 4A
smartphone selfie cameras in a variety of environments. It is the largest publicly
available, isolated sign dataset by number of examples and is the first dataset to
focus on one-handed, smartphone signs. We collected over 210,000 examples
at 1944x2592 resolution made by 47 consenting Deaf adult signers for whom
American Sign Language is their primary language. We manually reviewed 217,866
of these examples, of which 175,022 (approximately 700 per sign) were the sign
intended for the educational game. 39,304 examples were recognizable as a sign
but were not the desired variant or were a different sign. We provide a training set
of 31 signers, a validation set of eight signers, and a test set of eight signers. A
baseline LSTM model for the 250-sign vocabulary achieves 82.1% accuracy (81.9%
class-weighted F1 score) on the validation set and 84.2% (83.9% class-weighted
F1 score) on the test set. Gameplay suggests that accuracy will be sufficient for
creating educational games involving sign language recognition.

Submission Number: 952