Multi Modal 2-D Canvas Based Gallery Content Retrieval

Pragya Paramita Sahu, Vikrant Singh, Viswanath Veera

Abstract: Clicking pictures using mobile cameras is an ubiquitous part of daily human lives. However, finding the same photo, after a couple of weeks, is a herculean task. Conventional gallery search is based on a 1D query input, limited to keywords, without any smart understanding of the user’s needs. The user is needed to scroll through hundreds of images to find the one. In this work, we propose a multi-modal canvas based content retrieval system. Our system allows the users to write/draw on a 2-D canvas, instead of a single line. We use the relative position of the keywords to better understand the user’s needs and retrieve the photo he needs faster. Complex ideas of perception, nearness and object relations are more easily explained through a 2-D input, rather than complex sentences. We integrate a handwriting recognition system with a clustering based sketch recognition engine, to allow users a truly multi-modal system to interact with their devices. Our system also allows users to specify the finer features of the objects, thus providing increased ease of retrieval. We show that using the proposed system, users are able to find the required image faster with a \(33\%\) reduction in human effort. Proposed system allows users to creatively search for images rather than monotonously scrolling through hundreds of images.