Measuring Time Series Forecast Stability for Demand Planning

Steven Klee, Yuntian Xia

Keywords: Demand forecasting, time series, deep learning, ensemble models, supply chain planning

Abstract:

Time series forecasting is a critical first step in generating demand
plans for supply chains. Experiments on time series models typically
focus on demonstrating improvements in forecast accuracy
over existing/baseline solutions, quantified according to some accuracy
metric. There is no doubt that forecast accuracy is important;
however in production systems, demand planners often value consistency
and stability over incremental accuracy improvements.
Assuming that the inputs have not changed significantly, forecasts
that vary drastically from one planning cycle to the next require
high amounts of human intervention, which frustrates demand
planners and can even cause them to lose trust in ML forecasting
models. We study model-induced stochasticity, which quantifies the
variance of a set of forecasts produced by a single model when the
set of inputs is fixed. Models with lower variance are more stable.

Recently the forecasting community has seen significant advances
in forecast accuracy through the development of deep machine
learning models for time series forecasting. We perform a
case study measuring the stability and accuracy of state-of-the-art
forecasting models (Chronos, DeepAR, PatchTST, Temporal Fusion
Transformer, TiDE, and the AutoGluon best quality ensemble) on
public data sets from the M5 competition and Favorita grocery
sales. We show that ensemble models improve stability without
significantly deteriorating (or even improving) forecast accuracy.
While these results may not be surprising, the main point of this
paper is to propose the need for further study of forecast stability
for models that are being deployed in production systems

Submission Number: 7