Time-sensitive Weight Averaging for Practical Temporal Domain Generalization

Aoming Liu, Kevin Miller, Venkatesh Saligrama, Kate Saenko, Ser-Nam Lim, Bryan A. Plummer

Keywords: Domain Generalization, Weight Averaging, Temporal Distribution Shift

Abstract: Temporal Domain Generalization (TDG) is a valuable yet challenging task that requires models to support temporal distribution shifts without access to future samples. Prior work utilized time-sensitive models that take timestamps as input or directly estimated optimal model parameters for each temporal domain. However, these methods were evaluated in oversimplified settings that do not scale to complex scenarios. To fundamentally enhance TDG's value for real-world applications, we propose three key principles for TDG method design: 1) Time-sensitive model, 2) Generic method, and 3) Realistic evaluation. Reflecting these guidelines, we propose Time-sensitive Weight Averaging (TWA), a simple yet effective approach to apply weight averaging (WA) of specialists for every temporal domain. For principle 1), we train a selector network to estimate the good coefficients to average weights based on timestamp input. For principle 2), TWA is inherently generic, as WA requires no modification to model architecture. For principle 3), we incorporate more realistic benchmarks into TDG, including CLEAR-10, CLEAR-100, Yearbook, and FMoW-Time, which feature complex data distributions and natural temporal shifts. Extensive experiments conducted on these benchmarks demonstrate the practical value of TWA, e.g., on CLEAR-10/100, TWA consistently improves accuracy over the baselines by up to 4%. We also demonstrate TWA boosts performance on common TDG benchmarks used in prior work. Lastly, we provide theoretical insights behind the outstanding performance of TWA.

Primary Area: transfer learning, meta learning, and lifelong learning

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 6406