Doubly residual transitions for deep variational state-space models

TMLR Paper5310 Authors

Abstract: Sequential data modeling often relies on capturing underlying dynamics through Variational State-Space Models (VRSSMs), yet the architecture of transition functions in these models remains underexplored. Here we investigate highway layers as latent transitions in VRSSMs, leveraging their trainable gating mechanisms that allow flexible combination of raw and transformed representations. Through extensive empirical evaluation across multiple datasets, we demonstrate that highway transitions consistently outperform standard multi-layer perceptron (MLP) baselines. Our results show that highway-based VRSSMs achieve better validation performance while demonstrating enhanced robustness to hyperparameter choices. The findings highlight how established neural network techniques can significantly impact probabilistic sequential modeling when applied in new contexts. We recommend that practitioners incorporate highway connections in their modeling toolbox for VRSSMs, as they provide a simple yet effective architectural enhancement for capturing temporal dependencies in sequential data.

Submission Length: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Laurent_Dinh1

Submission Number: 5310