Function Encoders: A Principled Approach to Transfer Learning in Hilbert Spaces

Tyler Ingebrand, Adam Thorpe, ufuk topcu

TL;DR: This paper introduces a novel categorization of inductive transfer based on interpolation and extrapolation, and demonstrates that the function encoder achieves superior performance across four diverse transfer learning benchmarks.

Abstract: A central challenge in transfer learning is designing algorithms that can quickly adapt and generalize to new tasks without retraining. Yet, the conditions of when and how algorithms can effectively transfer to new tasks is poorly characterized. We introduce a geometric characterization of transfer in Hilbert spaces and define three types of inductive transfer: interpolation within the convex hull, extrapolation to the linear span, and extrapolation outside the span. We propose a method grounded in the theory of function encoders to achieve all three types of transfer. Specifically, we introduce a novel training scheme for function encoders using least-squares optimization, prove a universal approximation theorem for function encoders, and provide a comprehensive comparison with existing approaches such as transformers and meta-learning on four diverse benchmarks. Our experiments demonstrate that the function encoder outperforms state-of-the-art methods on four benchmark tasks and on all three types of transfer.

Lay Summary: Artificial intelligence (AI) systems are great at automating tasks—like recognizing images—but they usually rely on large, carefully collected datasets. That means they struggle with tasks that aren't well represented in the data. This paper introduces a new way to understand and model this limitation using a mathematical framework. It also presents a new method, called the function encoder, which outperforms existing techniques on several benchmarks.

Link To Code: https://github.com/tyler-ingebrand/FEtransfer

Primary Area: Theory->Domain Adaptation and Transfer Learning

Keywords: Transfer Learning, Learned Basis Functions, Function Encoder

Submission Number: 5067