Fisher vector with weakly-supervised Gaussian dictionary for scene classification

Peng Tang, Bin Feng, Xinggang Wang, Bi Li, Sihua Yi, Wenyu Liu

Abstract: The Fisher Vector (FV) is a very successful image representing method, which has achieved the state-of-the-art performance on scene classification. It concatenates the gradient of parameters in generative model as the image representation, which takes the advantage of generative and discriminative models. Using Gaussian mixture model (GMM) as the dictionary model, it can be regarded as an extension of the Bag-of-Words (BoW). But using unsupervised GMM to learn the dictionary makes a great loss for the information of image labels, which counts a lot for discrimination. To address the problem, we propose a novel strategy named Weakly-Supervised Gaussian Dictionary for Fisher Vector (WSGD-FV) to get the image representation. Specifically, we first use the weakly-supervised method to learn the Gaussian words, and then we combine these words to a Gaussian dictionary as the probability density function, so we can use this function to generate the FV. Our method is shown to get much better performance than the conventional FV for scene classification.