Learning High-Order Motion Patterns from Event Stream for Continuous Space-Time Video Super-Resolution

TMLR Paper6570 Authors

Abstract: Current methods in the domain of continuous space-time video super-resolution achieve temporal alignment by predicting motion between frames. However, these frame-based approaches encounter challenges with inaccurate optical flow estimation. To overcome this, we incorporate event data, enhancing both temporal and spatial aspects of video super-resolution. Based on the motion details conveyed by event streams, our proposed method, EvTaylor-Net, performs a Taylor expansion approximation of the object motion function at specified timestamps to estimate more precise forward optical flow. Our method estimates the masks from the event surface to alleviate the issue of multiple source pixels mapping to the same target position during the forward warping process. Furthermore, EvTaylor-Net adopts local implicit neural representation to simultaneously enhance the resolution of videos in both temporal and spatial domain, ensuring a comprehensive improvement of video quality. Extensive experimental results demonstrate that the proposed EvTaylor-Net, bolstered by event streams, outperforms state-of-the-art methods for spatio-temporal video super-resolution tasks.

Submission Type: Regular submission (no more than 12 pages of main content)

Assigned Action Editor: ~Xiao_Luo3

Submission Number: 6570