An Information Geometric Analysis of Noise Contrastive Estimation

H. S. Melihcan Erol, Jiejun Jin, Lizhong Zheng

Abstract: We study noise-contrastive estimation (NCE) through the lens of local information geometry. In particular, paralleling recent work [1], [2], we demonstrate that the optimal noise distribution for NCE is not the same as those typically chosen based on practical heuristics. Furthermore, we show that an important information geometric construct, the angle between the subspaces of distributions, dictates the asymptotic performance of an extension of NCE proposed in [3]. This analysis provides an information geometric interpretation of the method, encouraging further study of NCE and its variants using information geometric methods.

External IDs:dblp:conf/allerton/ErolJZ24