AdaFNIO: A Physics-Informed Adaptive Fourier Neural In- terpolation Operator for Synthetic Frame Generation
Abstract: We present, \textbf{AdaFNIO} - Adaptive Fourier Neural Interpolation Operator, a neural operator-based architecture to perform synthetic frame generation. Current deep learning-based methods rely on local convolutions for feature learning and suffer from not being scale-invariant, thus requiring training data to be augmented through random flipping and re-scaling. On the other hand, \textbf{AdaFNIO} leverages the principles of physics to learn the features in the frames, independent of input resolution, through token mixing and global convolution in the Fourier spectral domain by using Fast Fourier Transform (FFT). We show that \textbf{AdaFNIO} can produce visually smooth and accurate results. To evaluate the visual quality of our interpolated frames, we calculate the structural similarity index (SSIM) and Peak Signal to Noise Ratio (PSNR) between the generated frame and the ground truth frame. We provide the quantitative performance of our model on Vimeo-90K dataset, DAVIS, UCF101 and DISFA+ dataset. Lastly, we apply the model to in-the-wild videos such as photosynthesis, brain MRI recordings and red blood cell animations
Submission Length: Long submission (more than 12 pages of main content)
Assigned Action Editor: ~Yanwei_Fu2
Submission Number: 2704
Loading