Gradient Aligned Regression via Pairwise Losses

Dixian Zhu, Tianbao Yang, Livnat Jerby

TL;DR: Novel pairwise losses for regression.

Abstract: Regression is a fundamental task in machine learning that has garnered extensive attention over the past decades. The conventional approach for regression involves employing loss functions that primarily concentrate on aligning model prediction with the ground truth for each individual data sample. Recent research endeavors have introduced novel perspectives by incorporating label similarity into regression through the imposition of additional pairwise regularization or contrastive learning on the latent feature space, demonstrating their effectiveness. However, there are two drawbacks to these approaches: (i) their pairwise operations in the latent feature space are computationally more expensive than conventional regression losses; (ii) they lack theoretical insights behind these methods. In this work, we propose GAR (Gradient Aligned Regression) as a competitive alternative method in label space, which is constituted by a conventional regression loss and two pairwise label difference losses for gradient alignment including magnitude and direction. GAR enjoys: i) the same level efficiency as conventional regression loss because the quadratic complexity for the proposed pairwise losses can be reduced to linear complexity; ii) theoretical insights from learning the pairwise label difference to learning the gradient of the ground truth function. We limit our current scope as regression on the clean data setting without noises, outliers or distributional shifts, etc. We demonstrate the effectiveness of the proposed method practically on two synthetic datasets and on eight extensive real-world tasks from six benchmark datasets with other eight competitive baselines. Running time experiments demonstrate the superior efficiency of the proposed GAR compared to existing methods with pairwise regularization or contrastive learning in the latent feature space. Additionally, ablation studies confirm the effectiveness of each component of GAR. The code is open sourced at https://github.com/DixianZhu/GAR.

Lay Summary:

In many real-world problems, like predicting a patient’s disease risk or estimating housing prices, machines are trained to make predictions from data — a process called regression. Traditional regression methods look at each example one by one, trying to predict each output as accurately as possible. However, they often miss important relationships between examples — like how two similar houses should have similar prices.

This paper introduces a new way to teach machines to better understand relationships between data points. Instead of looking at each example separately, the proposed method compares pairs of examples to see how their differences in input features relate to differences in their outputs. Think of it like this: if one student studies more than another, and also scores higher, the model learns to align that pattern — it "learns from comparisons."

The method, called Gradient-Aligned Regression (GAR) via Pairwise Losses, allows the computer to capture these kinds of trends more effectively. It does this efficiently, even with large datasets, and is theoretically related to capturing the gradients of the ground truth function.

Link To Code: https://github.com/DixianZhu/GAR

Primary Area: General Machine Learning->Supervised Learning

Keywords: Regression, Pairwise Losses

Submission Number: 5411