Conversational image search: a sketch-based approach

Daniel D. Braghis, Haiming Liu

Abstract: Conversational image search has emerged as a progressive step beyond traditional keyword-based methodologies, which addresses challenges in human-computer interaction during the information retrieval process. This paper introduces a demonstration called DoodleShoper, a forward-thinking conversational image search assistant centered around sketching, specifically tailored for online product searches. It underscores the importance of visual diversity, often eluding verbal expression while highlighting the efficacy of a sketch-based approach in enhancing user interaction. The proposed modular architecture integrates a state-of-the-art Language Model with advanced Stable Diffusion technologies in the image generation field to offer users a more intuitive and precise conversational search experience. Unlike most conventional methods that directly align prompts or sketches with images, our approach leverages a generative model to produce an intermediate search outcome. This strategic shift streamlines the search process from a zero-shot query - where the query directly corresponds to an image - to a reverse image search task, facilitating the discovery of similar images through multimodal interaction. The implemented demonstration involves refining and expanding the application to diverse user information needs and preferences, including exploring the potential of utilising sketches as an alternative or complementary search environment, a novel concept rooted in current research.

External IDs:doi:10.1145/3652583.3657594