One-Pass Feature Evolvable Learning with Theoretical Guarantees

Cun-Yuan Xing, Meng-Zhang Qian, Wu-Yang Chen, Wei Gao, Zhi-Hua Zhou

TL;DR: This paper proposes a one-pass feature evolvable learning method with theoretical guarantees, introducing the Kernel Ortho-Mapping (KOM) discrepancy to characterize and exploit relationships between evolving feature spaces.

Abstract: Feature evolvable learning studies the scenario where old features will vanish and new features will emerge when learning with data streams, and various methods have been developed by utilizing some useful relationships from old features to new features, rather than re-training from scratch. In this work, we focus on two fundamental problems: How to characterize the relationships between two different feature spaces, and how to exploit those relationships for feature evolvable learning. We introduce the Kernel Ortho-Mapping (KOM) discrepancy to characterize relationships between two different feature spaces via kernel functions, and correlate with the optimal classifiers learned from different feature spaces. Based on this discrepancy, we develop the one-pass algorithm for feature evolvable learning, which requires going through all instances only once without storing the entire or partial training data. Our basic idea is to take online kernel learning with the random Fourier features and incorporate some feature and label relationships via the KOM discrepancy for feature evolvable learning. We finally validate the effectiveness of our proposed method both theoretically and empirically.

Lay Summary: When the set of features used by a machine learning model changes, because some sensors degrade or new ones are introduced, our method enables a seamless transition without discarding prior knowledge. We begin by constructing a lightweight alignment between the old and new feature spaces to capture their most salient correlations. Using this alignment, we then perform a simple projection-based initialization that produces a “warm start” model in the new feature space. As new data points stream in under the revised feature regime, the algorithm updates its parameters in a single pass, eliminating the need to store or revisit past examples. Both our theoretical analysis and empirical results demonstrate that this approach converges nearly as rapidly and achieves accuracy comparable to fully retraining on the new features, yet does so with substantially reduced computational effort and memory requirements.

Link To Code: https://github.com/WeltXing/opfes

Primary Area: General Machine Learning->Online Learning, Active Learning and Bandits

Keywords: Feature Evolvable Learning, Kernel Ortho-Mapping, Online Learning

Submission Number: 5784