Retagging social images based on visual and semantic consistency

Dong Liu, Xian-Sheng Hua, Meng Wang, Hong-Jiang Zhang

Abstract: The tags on social media websites such as Flickr are frequently imprecise and incomplete, thus there is still a gap between these tags and the actual content of the images. This paper proposes a social image ``retagging'' scheme that aims at assigning images with better content descriptors. The refining process is formulated as an optimization framework based on the consistency between ``visual similarity'' and ``semantic similarity'' in social images. An effective iterative bound optimization algorithm is applied to learn the optimal tag assignment. In addition, as many tags are intrinsically not closely-related to the visual content of the images, we employ a knowledge-based method to differentiate visual content related from unrelated tags and then constrain the tagging vocabulary of our automatic algorithm within the content related tags. Experimental results on a Flickr image collection demonstrate the effectiveness of this approach.