Do We Need Too Much Attention? A Time Series Perspective

ICLR 2025 Workshop ICBINB Submission23 Authors

Track: long paper (up to 4 pages)

Keywords: Time Series, Large Language Models, Energy Optimization, Wireless Networks, 6G

TL;DR: This work explores time series prediction using LLMs, focusing on telecom energy optimization. PatchTST leverages multiscale inputs for better forecasting, while Chronos struggles to capture temporal dependencies.

Abstract: The present work proposes a method for time series prediction with applications across domains, like in agriculture for optimal crop timing, stock market forecasting, and in e-commerce. Studies suggest that with slight modification, Large Language Models (LLMs) can be adapted for time series prediction. In the telecom sector, this approach could help in significant energy conservation during network operations. In this work, various models have been evaluated for this purpose and their performances are compared that includes traditional Machine Learning and Deep Learning methods like ARIMA, RNNs and LSTMs. More recent LLM-based models were also explored such as Chronos, and PatchTST which utilizes fewer attention layers compared to Chronos. It was surprising to observe that among these models, PatchTST achieved the best performance only after fine-tuning. While Chronos is designed for zero-shot forecasting and captures some intricate temporal dependencies, PatchTST’s multiscale input helps the model to understand the macro and the micro level trends and therefore might help it perform better than other methods. The results seem to indicate that effective forecasting could be achieved with fewer attention layers when supported by well-engineered input contextual representations.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 23