Optical Flow Estimation through Fusion Network based on Self-supervised Deep Learning

Cong Liu, Dianxi Shi, Ruihao Li, Huachi Xu

Abstract: Compared with conventional cameras, dynamic vision sensors (namely event cameras) are especially suitable for high-speed and high-dynamic applications due to their advantages (high dynamic range, low latency, etc.). However, it still struggles when encountering low-speed and low-texture scenes. Aiming at the problem of optical flow estimation, we propose a novel unsupervised learning estimation method with both event data and gray image frames as the input. Two different fusion mechanisms are presented and discussed in our work, one is the Direct Fusion Network (DFEV-FlowNet), and the other one is Local Squeeze Extraction Network (LSENet). DFEV-FlowNet directly fuses synthesized event frames and gray image frames, and LSENet adopts a local squeeze extraction weights adaptive mechanism. In order to achieve self-supervised learning, photometric constraints between consecutive frames are used to drive the network training. We use the public event dataset MVSEC to evaluate the proposed optical flow estimation method qualitatively and quantitatively. The results show that our method demonstrate better performance in terms of the estimation accuracy.