UnitNorm: Rethinking Normalization for Transformers in Time Series

Nan Huang, Christian Kümmerle, Xiang Zhang

Keywords: time series, Transformer, normalization

TL;DR: We propose UnitNorm, a novel normalization design that scales input vectors by their norms to address token shift, attention shift, and sparse attention issues in Transformer-based time series analysis.

Abstract:

Normalization techniques are crucial for enhancing Transformer models' performance and stability in time series analysis tasks, yet we originally identify that traditional methods like batch and layer normalization often lead to issues such as token shift, attention shift, and sparse attention.
We propose UnitNorm, a novel normalization approach that scales input vectors by their norms and modulates attention patterns, effectively circumventing these challenges.
Grounded in existing normalization frameworks, UnitNorm's effectiveness is demonstrated across diverse time series analysis tasks, including forecasting, classification, and anomaly detection, via a rigorous evaluation on 6 state-of-the-art models and 10 datasets.
UnitNorm demonstrates superior performance, particularly where robust attention and contextual understanding are vital, achieving up to a 1.46 MSE decrease in forecasting and a 4.89\% accuracy increase in classification.
This work not only calls for a re-evaluation of normalization strategies in time series Transformers but also sets a new direction for enhancing model performance and stability.

Primary Area: learning on time series and dynamical systems

Submission Number: 2879