ProtoLLM: Training and Example-free LLMs for Few-shot Tabular Learning
Keywords: Tabular data, Few-shot learning, Large language models
TL;DR: We use LLMs to generate oracle features for tabular data without relying on examples, thereby boosting few-shot performance through a training-free approach.
Abstract: Recent breakthroughs in large language models (LLMs) have opened the door to in-depth investigation of their potential in tabular data modeling. However, the paradigm for effectively utilizing advanced LLMs in few-shot and even unseen scenarios remains to be explored. We observed an unusual phenomenon: directly using LLMs for data augmentation or rule generation by feeding a few examples significantly degrades the reasoning ability in tabular data understanding. We identified two main obstacles behind this issue: overfitting to the examples and knowledge disruption. Specifically, the provided examples may introduce noisy patterns that interfere with the model's prior knowledge, leading to unexpected and less reliable results. To this end, we propose an example-free framework to leverage the inherent knowledge of LLMs. Our key idea is to prompt the LLM for oracle feature generation based solely on task and feature description. Without such example pollution, each output feature is treated as a standard guideline, and they together act as a prototype for each class. To transfer the LLM's knowledge to a given task, we further design an efficient fusion strategy to integrate the prototype with example features, showing impressive generalizability in the few-shot setting. Importantly, our pipeline requires no learnable variables, resulting in a desired training-free property. Extensive comparisons and ablations on multiple tabular datasets demonstrate the improvements of our simple framework.
Supplementary Material: zip
Primary Area: applications to computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Reciprocal Reviewing: I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 763
Loading