Adjust Pearson's $r$ to Measure Arbitrary Monotone Dependence

Xinbo Ai

Keywords: Pearson's r, correlation coefficient, rearrangement inequality, nonlinear monotone dependence

TL;DR: The most used Pearson's r, typically regarded as a measure only for linear dependence, has been enhanced to accurately measure nonlinear monotone dependence, with the aid of an inequality tighter than Cauchy-Schwartz Inequality.

Abstract: Pearson's $r$, the most widely-used correlation coefficient, is traditionally regarded as exclusively capturing linear dependence, leading to its discouragement in contexts involving nonlinear relationships. However, recent research challenges this notion, suggesting that Pearson's $r$ should not be ruled out a priori for measuring nonlinear monotone relationships. Pearson's $r$ is essentially a scaled covariance, rooted in the renowned Cauchy-Schwarz Inequality. Our findings reveal that different scaling bounds yield coefficients with different capture ranges, and interestingly, tighter bounds actually expand these ranges. We derive a tighter inequality than Cauchy-Schwarz Inequality, leverage it to refine Pearson's $r$, and propose a new correlation coefficient, i.e., rearrangement correlation. This coefficient is able to capture arbitrary monotone relationships, both linear and nonlinear ones. It reverts to Pearson's $r$ in linear scenarios. Simulation experiments and real-life investigations show that the rearrangement correlation is more accurate in measuring nonlinear monotone dependence than the three classical correlation coefficients, and other recently proposed dependence measures.

Primary Area: Probabilistic methods (for example: variational inference, Gaussian processes)

Submission Number: 157