PlicoTabTransformer: Folding Tabular Embeddings Into M Vectors

Jonathan Chung, Giridhar Narasapura Rajagopalaiah

Keywords: Tabular data, Transformer, Self-attention, Positional embeddings, Contrastive loss, Classification

Abstract:

Tabular data represents the most prevalent and extensively utilized form of structured data in various domains. Traditionally dominated by tree-based algorithms, researchers are actively exploring the application of deep neural networks on tabular data. Notably, the TabTransformer (Huang et al., 2020) and FT-transformer (Gorishniy et al., 2021) showed that feeding column embeddings of the tabular
features into a transformer could learn a representation of the columns and how the embeddings interact with one another. This paper introduces PlicoTabTransformer, an enhancement of the previous methods, which is designed to learn multiple representations of the column embeddings. By incorporating a transformer with multiple learnable position embeddings and a contrastive learning loss, our
method learns multiple distinct and orthogonal representations (denoted as plicovectors) of the column embeddings. We evaluated the PlicoTabTransformer with the pytorch-frame benchmark. Our experimental demonstrated that the PlicoTabTransformer is overall top ranked algorithm and achieves state of the art performance in several datasets compared to other deep learning method closing the gap
with tree based algorithms. Our method provides an added advantage to visualise redundancies and a potential dimensionality reduction technique.

Primary Area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.

Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 7899