Multi-modal kernel ridge regression for social image classification

Xiaoming Zhang, Wenhan Chao, Zhoujun Li, Chunyang Liu, Rui Li

Abstract: Highlights • We propose to tackle the problem of classifying social image with multi-modal content. A classifier with multi-modal kernel ridge regression is proposed to capture the correlation between different types of features. • Two kernel ridge regression classifiers are learned for the visual features and text features, and a joint learning model is used to reinforce the learning of the two classifiers. Abstract There is growing interest in social image classification because of its importance in web-based image application. Though there are many approaches on image classification, it is still a great problem to integrate multi-modal contents of social images simultaneously for classification, since the textual content and visual content are represented in two heterogeneous feature spaces. In this study, a multi-modal learning algorithm is proposed to fuse the multiple features through their correlation seamlessly. Specifically, two classification modules based on the kernel ridge regression (KRR) are learned for the two types of features, and they are integrated via a joint model. With the joint model, the classification based on visual features can be reinforced by the classification based on textual features, and vice verse. Then, an efficient optimization method is proposed to resolving the object function. The query image can be classified based on both of the textual features and visual features by combing the results of the two classifiers. Two methods are proposed to combine the classification results to obtain the final result. To evaluate the approach, extensive experiments are conducted on the real-world datasets, and the result demonstrates the superiority of our approach.