A Theory-Driven Approach to Inner Product Matrix Estimation for Incomplete Data: An Eigenvalue Perspective

Fangchen Yu, Yicheng Zeng, Jianfeng Mao, Wenye Li

Track: Search and retrieval-augmented AI

Keywords: Inner Product Matrix Estimation, Incomplete Data, Eigenvalue Distribution, Random Matrix Theory

Abstract: Addressing the critical challenge of data incompleteness in inner product matrix estimation, we introduce a novel eigenvalue correction method designed to precisely reconstruct true inner product matrices from incomplete data. Utilizing random matrix theory, our method adjusts the eigenvalue distribution of the estimated inner product matrix to align with the ground-truth. This approach significantly reduces estimation errors for both inner product matrices and the derived Euclidean distance matrices, thereby enhancing the effectiveness of similarity searches on incomplete data. Our method surpasses traditional data imputation and similarity calibration techniques in both maximum inner product search and nearest neighbor search tasks, demonstrating marked advancements in managing incomplete data. It exhibits robust performance across various missing rates and diverse scenarios.

Submission Number: 948