NePuDA: Neighborhood-Purifying Discriminant Analysis

TMLR Paper7396 Authors

Abstract: Linear Discriminant Analysis (LDA) is a popular technique for supervised dimensionality reduction due to its clarity and interpretability. However, LDA and its variants often assume that data within each class are Gaussian-distributed or form distinct groups/subclasses, which is not always true for high-dimensional, real-world datasets, where classes can have complex and irregular shapes and exhibit significant overlap. Recognizing this limitation, we propose a novel approach, \textbf{Ne}ighborhood-\textbf{Pu}rifying \textbf{D}iscriminant \textbf{A}nalysis, which forgoes the search for an ideal, class-separated subspace in favor of one where data samples are naturally surrounded by neighbors from the same class. Specifically, NePuDA aims to identify projection directions that reinforce this neighborhood purity for all data samples, with the intuitive logic that if an object shares characteristics with a known category, it likely belongs to that category. Accordingly, we formulate the objective function of the proposed method and introduce an iterative optimization procedure to solve it in an efficient manner. Detailed theoretical analyses are provided, covering convergence, computational complexity, and connections to existing LDA variants. Extensive empirical evaluations on a range of synthetic and real-world datasets demonstrate that NePuDA consistently extracts highly discriminative features, outperforming twelve classical and state-of-the-art supervised dimensionality reduction algorithms in classification accuracy. Our code is publicly available at \url{https://anonymous.4open.science/r/NePuDA_code-C47F/}.

Submission Type: Long submission (more than 12 pages of main content)

Assigned Action Editor: ~Shinichi_Nakajima2

Submission Number: 7396