Cross-domain Attention for Transfer Learning between Tabular Data without Shared Features

Kazi Fuad Bin Akhter, Ibna Kowsar, Manar Samad

Keywords: tabular data, cross-attention, transfer learning, cross-domain

TL;DR: This paper presents a new cross-attention method using transformer weights instead of key and value representations to achieve cross-domain transfer learning between disparate tabular data sets.

Abstract: Unlike image and text, the transfer learning of tabular data is challenging due to the heterogeneity in feature types, structure, and semantics across disparate application domains. Existing methods assume shared features between data tables to transfer knowledge, often by fine-tuning a large pre-trained model. To facilitate learning between domains without shared features, we propose a \emph{data-agnostic} Cross-domain Attention Transfer Learning (CATTLE). CATTLE performs self-supervised learning of $key$, $value$, $query$ projection weights of a transformer using source data. Pre-trained weights of selective attention layers are used in a separate transformer to learn cross-domain attention for target data, instead of conventionally fine-tuning the same pre-trained model. Our experiments on ten pairs of source-target data sets without shared features show that CATTLE is statistically and in terms of performance rank superior to nine state-of-the-art baselines, including traditional ML, deep tabular representation learning, and transfer learning methods proposed for tabular data sets. A single tabular data set from an arbitrary domain is sufficient to achieve cross-domain attention to generalize to new downstream learning, eliminating the need for large foundation models pre-trained by many disparate tabular data sets. CATTLE source code is available publicly.

Primary Area: transfer learning, meta learning, and lifelong learning

Submission Number: 7631