InvGC: Robust Cross-Modal Retrieval by Inverse Graph Convolution
Submission Type: Regular Long Paper
Submission Track: Speech and Multimodality
Submission Track 2: Language Grounding to Vision, Robotics and Beyond
Keywords: Cross-Modal Retrieval, Data Degeneration, Graph Convolution
TL;DR: In this research, we articulate the data degeneration issue within the gallery set found in cross-modal retrieval, introducing a novel post-processing method to mitigate it. Consequently, we witness a substantial enhancement in retrieval performance.
Abstract: Over recent decades, significant advancements in cross-modal retrieval is mainly driven by breakthroughs in visual and linguistic modeling. However, a recent study shows that multi-modal data representations tend to cluster within a limited convex cone (as representation degeneration problem), which hinders retrieval performance due to the inseparability of these representations. In our study, we first empirically validate the presence of the representation degeneration problem across multiple cross-modal benchmarks and methods. Next, to address it, we introduce a novel method, called InvGC, a post-processing technique inspired by graph convolution and average pooling. Specifically, InvGC defines the graph topology within the datasets and then applies graph convolution in a subtractive manner. This method effectively separates representations by increasing the distances between data points. To improve the efficiency and effectiveness of InvGC, we propose an advanced graph topology, LocalAdj, which only aims to increase the distances between each data point and its nearest neighbors. To understand why InvGC works, we present a detailed theoretical analysis, proving that the lower bound of recall will be improved after deploying InvGC. Extensive empirical results show that InvGC and InvGC w/LocalAdj significantly mitigate the representation degeneration problem, thereby enhancing retrieval performance.
Submission Number: 4066
Loading