Context is Key: A Benchmark for Forecasting with Essential Textual Information

Arjun Ashok, Andrew Robert Williams, Étienne Marcotte, Valentina Zantedeschi, Jithendaraa Subramanian, Roland Riachi, James Requeima, Alexandre Lacoste, Irina Rish, Nicolas Chapados, Alexandre Drouin

Keywords: forecasting, context-aided forecasting, large language model, llm, reasoning, multimodal, pretrained

TL;DR: A forecasting benchmark with problems that require the combined use of numerical historical data and textual context

Abstract: Forecasting is a task of pinnacle importance in decision making across various fields. Numerical data alone often lacks crucial information for accurate forecasting, and in many cases, humans possess additional contextual information that is essential for forecasting, such as background knowledge or constraints on the quantity to predict. One convenient way to provide such essential information to models is through natural language. Yet, the extent to which existing forecasting approaches can effectively utilize contextual information in text is still an open question. To address this, we propose Context is Key (CiK), a time series forecasting benchmark consisting of tasks that combine numerical data with diverse kinds of textual context, requiring models to leverage a variety of skills to succeed. We evaluate a range of approaches and introduce a simple LLM prompting method that serves as a strong baseline. By presenting this challenging benchmark, we aim to foster progress in multimodal forecasting, paving the way for advancements that will lead to forecasting methods accessible to decision-makers irrespective of their technical expertise. The benchmark can be visualized at https://anon-forecast.github.io/benchmark_report/.

Submission Number: 35