HyperTime: Hyperparameter Optimization for Combating Temporal Distribution Shifts

Shaokun Zhang, Yiran Wu, Zhonghua Zheng, Qingyun Wu, Chi Wang

Abstract: In this work, we propose a hyperparameter optimization method named HyperTime to find hyperparameters robust to potential temporal distribution shifts in the unseen test data. Our work is motivated by an important observation that it is, in many cases, possible to achieve temporally robust predictive performance via hyperparameter optimization. Based on this observation, we leverage the ‘worst-case-oriented’ philosophy from the robust optimization literature to help find such robust hyperparameter configurations. HyperTime imposes a lexicographic priority order on average validation loss and worst-case validation loss over chronological validation sets. We perform a theoretical analysis on the upper bound of the expected test loss, which reveals the unique advantages of our approach. We also demonstrate the strong empirical performance of the proposed method on multiple machine learning tasks with temporal distribution shifts.

Primary Subject Area: [Content] Multimodal Fusion

Secondary Subject Area: [Experience] Multimedia Applications

Relevance To Conference: There exist severe temporal distribution shift problems in multimodal data, which may considerably degrade the performance of multimedia applications. This work aims to design a hyperparameter optimization algorithm dedicated to solving the temporal distribution shifts problem in both vision and language data to enhance multimedia applications. We verify the proposed method in both multiple data types, demonstrating this method could greatly mitigate the temporal distribution shift problem.

Submission Number: 5013