Perceive before Respond: Improving Sticker Response Selection by Emotion Distillation and Hard Mining

Wuyou Xia, Shengzhe Liu, Rong Qin, Guoli Jia, Eunil Park, Jufeng Yang

Abstract: In online chatting, people are increasingly favoring the use of stickers to supplement or replace text for replies, as sticker images can express more vivid and varied emotions. The Sticker Response Selection (SRS) task aims to predict the sticker image that is most relevant to the history dialogue context. Previous researches explore the semantic similarity between context and stickers, while ignoring the role of both unimodal and cross-modal emotional information. In this paper, we propose a “Perceive before Respond” training paradigm. We perceive the emotion of stickers through a knowledge distillation module, which acquires emotion knowledge from the existing large-scale sticker emotion recognition dataset and distills it into our framework to enhance the understanding of sticker emotion. To further distinguish stickers with similar subject elements within the same topic, we perform contrastive learning at both inter-topic and intra-topic levels to obtain discriminative and diverse sticker representations. In addition, we improve the hard negative sampling method for image-text matching based on cross-modal sentiment association, conducting hard sample mining from both semantic similarity and sentiment polarity similarity for sticker-to-dialogue and dialogue-to-sticker. Extensive experiments verify the effectiveness of each proposed component. Ablation experiments on different backbone networks demonstrate the generality of our approach. The code is provided in the supplement material and will be released to the public.

Primary Subject Area: [Engagement] Emotional and Social Signals

Relevance To Conference: Stickers are becoming increasingly popular in instant messaging applications because they can express vivid and diverse emotions of users. The Sticker Response Selection (SRS) task aims to predict the most contextually relevant sticker image of the multi-turn dialogue.

Submission Number: 2084