DELPHYNE: A Pre-Trained Model for General and Financial Time Series

Xueying Ding, Aakriti Mittal, Achintya Gopal

Keywords: Time-series Forecast, AI for finance applications, Generative Models

TL;DR: We study negative transfer and architecture choices for training transformer-based models for financial time-series data.

Abstract: Time-series data is a vital modality within data science communities, particularly in financial applications, where it helps in detecting patterns, understanding market behavior, and making informed financial decisions based on historical data. Recent advances in language modeling have led to the rise of time-series pre-trained models that are trained on vast collections of datasets and applied to diverse tasks across financial domains. However, across financial applications, existing time-series pre-trained models have not shown promising performance boost over simple finance benchmarks in both zero-shot and fine-tuning settings due to lack of financial data within the pre-training stage, and the negative transfer effect due to inherently different time-series patterns across domains. To address the above problems, we introduce a Pre-trained Model for Finance Time-series (Delphyne). Delphyne achieves superior performance on various financial tasks as compared to existing foundation and full-shot models and we also use mechanistic analysis to explore attention routing during fine-tuning.

Submission Number: 174