PROMPT: Personalized User Tag Recommendation for Social Media Photos Leveraging Personal and Social Contexts

Rajiv Ratn Shah, Anupam Samanta, Deepak Gupta, Yi Yu, Suhua Tang, Roger Zimmermann

Abstract: Social media platforms such as Flickr allow users to annotate photos with descriptive keywords, called, tags with the goal of making multimedia content easily understandable, searchable, and discoverable. However, manual annotation is very time-consuming and cumbersome for most users, which makes it difficult to search relevant photos. Moreover, predicted tags for a photo are not necessarily relevant to users' interests. Thus, it necessitates for an automatic tag prediction system that considers users' interests and describes objective aspects of the photo such as visual content and activities. To this end, this paper presents a tag recommendation system, called, PROMPT, that recommends personalized tags for a given photo leveraging personal and social contexts. Specifically, first, we determine a group of users who have similar tagging behavior as the user of the photo, which is very useful in recommending personalized tags. Next, we find candidate tags from visual content, textual metadata, and tags of neighboring photos, and recommends five most suitable tags. We initialize scores of the candidate tags using asymmetric tag co-occurrence probabilities and normalized scores of tags after neighbor voting, and later perform random walk to promote the tags that have many close neighbors and weaken isolated tags. Finally, we recommend top five user tags to the given photo. Experimental results on a Flickr dataset (46,700 photos in the test set and 28 million photos in the train set) with 1,540 unique user tags confirm that the proposed algorithm outperforms state-of-the-arts.