Efficient Codebook and Factorization for Second Order Representation Learning

Pierre jacob, David Picard, Aymeric Histace, Edouard Klein

Abstract: Learning rich and compact representations is an open topic in many fields such as word embedding, visual question-answering, object recognition or image retrieval. Although deep neural networks (convolutional or not) have made a major breakthrough during the last few years by providing hierarchical, semantic and abstract representations for all of these tasks, these representations are not necessary as rich as needed nor as compact as expected. Models using higher order statistics, such as bilinear pooling, provide richer representations at the cost of higher dimensional features. Factorization schemes have been proposed but without being able to reach the original compactness of first order models, or at a heavy loss in performances. This paper addresses these two points by extending factorization schemes to codebook strategies, allowing compact representations with the same dimensionality as first order representations, but with second order performances. Moreover, we extend this framework with a joint codebook and factorization scheme, granting a reduction both in terms of parameters and computation cost. This formulation leads to state-of-the-art results and compact second-order models with few additional parameters and intermediate representations with a dimension similar to that of first-order statistics.

Keywords: Second order pooling

TL;DR: We propose a joint codebook and factorization scheme to improve second order pooling.

Data: [CUB-200-2011](https://paperswithcode.com/dataset/cub-200-2011), [Stanford Online Products](https://paperswithcode.com/dataset/stanford-online-products)