Factorization Machine-Enhanced Crossformer for Multivariate Time Series Event Detection in Epidemic

Fahim Tasneema Azad

Keywords: Spatio-Temporal data, Event Detection, Cross-former, Factorization Machine

Abstract:

We propose FM-Crossformer, a novel architecture that replaces the traditional attention mechanism in Crossformer with factorization
machine (FM) functions for epidemic event detection in multivariate time series. Traditional attention-based models face computational
limitations and struggle to capture complex feature interactions in high-dimensional epidemic data. Our approach leverages the strengths of
factorization machines to model pairwise feature interactions while maintaining the cross-dimensional and cross-temporal design of Cross-
former.
Let X ∈RN ×L represent a multivariate time series with N variables (e.g., infection rates, mobility indices, vaccination rates) and sequence
length L. The traditional Crossformer attention mechanism Attention(Q,K,V) = softmax(QKT /√dk )V is replaced with a factorization
machine function:
FM(x) = w0 +
n
wixi +
i=1
n
i=1
n
j=i+1
⟨vi,vj ⟩xixj (1)
where vi ∈Rk are latent factor vectors, and ⟨·,·⟩denotes the dot product. For the cross-dimension stage, we adapt the complete FM
formulation:
Hcross-dim
= Wdim
0 +
N
Wdim
i Xi,seg +
i=1
N
i=1
N
j=i+1
where Xseg ∈RN ×s represents dimension-wise segments and Hcross-dim ∈Rs×d. For the cross-time stage:
⟨v
dim
dim
i ,v
j ⟩Xi,seg ⊙Xj,seg (2)
Hcross-time
= Wtime
0 +
s
t=1
Wtime
t Hcross-dim
t +
s
t=1
s
t′ =t+1
⟨v
time
time
t ,v
t′ ⟩Hcross-dim
t ⊙Hcross-dim
t′ (3)
The event detection probability is computed as:
P(eventt) = σ(WoutHcross-time
t + bout) (4)
where σ is the sigmoid function, and Wout and bout are learnable parameters.
Traditional epidemic modeling relies on compartmental models (SIR, SEIR) and statistical approaches. Recent deep learning advances
include LSTM-based models for COVID-19 forecasting (Chimmula & Zhang, 2020), graph neural networks for spatial-temporal epidemic
prediction (Kapoor et al., 2020), and transformer architectures adapted for pandemic data analysis. Crossformer (Zhang & Yan, 2023)
introduced cross-dimensional attention for multivariate time series, while factorization machines (Rendle, 2010) have proven effective for
sparse feature interaction modeling in recommendation systems.
Our FM-Crossformer architecture maintains the two-stage design of Crossformer while replacing attention computation with complete
factorization machine functions. This approach reduces computational complexity from O(N2L2d) to O(N2sk+ s2k) where k is the latent
factor dimension and sis the segment length. When s≪L, this provides computational savings while enabling the FM functions to naturally
capture second-order feature interactions critical for epidemic dynamics, such as mobility and infection rate correlations as well as vaccination
and transmission dependencies.
The main contributions are: (1) Novel integration of factorization machines with transformer architecture for time series analysis;
(2) Computationally efficient approach for multivariate epidemic event detection; (3) Enhanced modeling of feature interactions in high-
dimensional epidemic data; (4) Theoretical framework demonstrating reduced computational overhead while maintaining interpretability
through explicit feature interaction modeling.
The proposed FM-Crossformer provides a computationally efficient alternative to attention-based models for detecting epidemic events
such as outbreak of diseases, peak occurrences, and intervention effectiveness, while maintaining interpretability through explicit feature
interaction modeling. Future work will validate performance across real-world epidemic datasets.
References
1. Chimmula, V. K. R., & Zhang, L. (2020). Time series forecasting of COVID-19 transmission in Canada using LSTM networks. Chaos,
Solitons & Fractals, 135, 109864.
2. Kapoor, A., Ben, X., Liu, L., Perozzi, B., Barnes, M., Blais, M., & O’Banion, S. (2020). Examining COVID-19 forecasting using
spatio-temporal graph neural networks. arXiv preprint arXiv:2007.03113.
3. Rendle, S. (2010). Factorization machines. In 2010 IEEE International Conference on Data Mining (pp. 995-1000). IEEE.
4. Zhang, Y., & Yan, J. (2023). Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting.
In International Conference on Learning Representations.

Submission Number: 329