Projection Pursuit Density Ratio Estimation

Meilin Wang, Wei Huang, Mingming Gong, Zheng Zhang

TL;DR: This paper proposes a projection pursuit-based method for density ratio estimation to address high-dimensionality and model misspecification challenges, with theoretical guarantees and superior experimental performance.

Abstract: *Density ratio estimation* (DRE) is a paramount task in machine learning, for its broad applications across multiple domains, such as covariate shift adaptation, causal inference, independence tests and beyond. Parametric methods for estimating the density ratio possibly lead to biased results if models are misspecified, while conventional non-parametric methods suffer from the curse of dimensionality when the dimension of data is large. To address these challenges, in this paper, we propose a novel approach for DRE based on the projection pursuit (PP) approximation. The proposed method leverages PP to mitigate the impact of high dimensionality while retaining the model flexibility needed for the accuracy of DRE. We establish the consistency and the convergence rate for the proposed estimator. Experimental results demonstrate that our proposed method outperforms existing alternatives in various applications.

Lay Summary:

Self-driving cars often fail in rain after being trained only in sunshine, highlighting a critical AI challenge: how can machines adapt when real-world conditions shift unexpectedly? Our research tackles this using density ratio estimation (DRE), a tool that can help models adjust to data changes. Beyond this, DRE can also help verify if two factors truly relate (e.g., smoking and cancer) or uncover causal relationships (e.g., how vaccines reduce transmission). However, traditional DRE methods have limitations: parametric methods may force data into oversimplified molds, while conventional non-parametric tools fail with complex, high-dimensional data.

Our approach employs a stepwise feature prioritization mechanism inspired by projection pursuit, where models learn to sequentially identify and amplify the most statistically significant patterns in the data. By iteratively breaking down data into meaningful layers, our method balances simplicity with flexibility. Rigorous mathematical proofs confirm its reliability, and experiments across diverse fields show it outperforms existing methods. This advance brings us closer to AI systems that robustly handle real-world unpredictability.

Primary Area: General Machine Learning

Keywords: density ratio estimation, projection pursuit, covariate shift adaptation

Submission Number: 4520