Residual Feature Integration is Sufficient to Prevent Negative Transfer

ICLR 2026 Conference Submission22077 Authors

Keywords: deep transfer learning theory, negative transfer, provable safe transfer mechanism

TL;DR: We prove that residual feature integration is sufficient to eliminate negative transfer in deep transfer learning, providing the first general theoretical guarantee with supporting empirical validation.

Abstract: Transfer learning has become a central paradigm in modern machine learning, yet it suffers from the long-standing problem of negative transfer, where leveraging source representations can harm rather than help performance on the target task. Although empirical remedies have been proposed, there remains little theoretical understanding of how to reliably avoid negative transfer. In this article, we investigate a simple yet remarkably effective strategy: augmenting frozen, pretrained source-side features with a trainable target-side encoder that adapts target features to capture residual signals overlooked by models pretrained on the source data. We show this residual feature integration strategy is sufficient to provably prevent negative transfer, by establishing rigorous theoretical guarantees that it never performs worse than training from scratch on the target data, and that the convergence rate can transition seamlessly from nonparametric to near-parametric when source representations are informative. To our knowledge, this is the first theoretical work that ensures protection against negative transfer. We carry out extensive numerical experiments across image, text and tabular benchmarks, and empirically verify that the method consistently safeguards performance under distribution shift, label noise, semantic perturbation, and class imbalance. Our study thus advances the theory of safe transfer learning, and provides a principled approach that is simple, robust, architecture-agnostic, and broadly applicable.

Primary Area: learning theory

Submission Number: 22077