Unsupervised Discovery of Fingerspelled Letters in Sign Language Videos

Feyza Duman, Tanya Deniz Ipek, Murat Saraçlar

Abstract: Automatic discovery and indexing in sign language videos would provide many valuable services such as search and retrieval in deaf social and news media which consists of sign language videos often with no associated texts available. As some of the context words in sign language conversations are fingerspelled, the discovery of fingerspelled letters plays a significant role. In this study, k-means clustering and Gaussian Mixture Model methods are used for the unsupervised discovery of fingerspelled letters. The clusers which mostly contain hand shapes corresponding to transitions between letters are eliminated based on their distortion level and Bayes information criterion values. The performance of the clustering methods are evaluated with the help of Adjusted Mutual Information metric. According to the results obtained with this metric, using a Gaussian Mixture Model for clustering yields better results than k-means clustering.