LIDL: Local Intrinsic Dimension estimation using approximate Likelihood

Piotr Tempczyk, Adam Golinski, Przemysław Spurek, Jacek Tabor

Keywords: intrinsic dimension, local intrinsic dimension, neural density estimation, normalizing flows, topology, manifold

TL;DR: We propose a new algorithm for local intrinsic dimension estimation using neural density estimators, which scales to datasets with thousands of dimensions.

Abstract: We investigate the problem of local intrinsic dimension (LID) estimation. LID of the data is the minimal number of coordinates which are necessary to describe the data point and its neighborhood without the significant information loss. Existing methods for LID estimation do not scale well to high dimensional data because they rely on estimating the LID based on nearest neighbors structure, which may cause problems due to the curse of dimensionality. We propose a new method for Local Intrinsic Dimension estimation using Likelihood (LIDL), which yields more accurate LID estimates thanks to the recent progress in likelihood estimation in high dimensions, such as normalizing flows (NF). We show our method yields more accurate estimates than previous state-of-the-art algorithms for LID estimation on standard benchmarks for this problem, and that unlike other methods, it scales well to problems with thousands of dimensions. We anticipate this new approach to open a way to accurate LID estimation for real-world, high dimensional datasets and expect it to improve further with advances in the NF literature.

Community Implementations: [ 1 code implementation](https://www.catalyzex.com/paper/arxiv:2206.14882/code)