Video Generation from Text Employing Latent Path Construction for Temporal Modeling

Amir Mazaheri, Mubarak Shah

Abstract: Video generation is one of the most challenging tasks in Machine Learning and Computer Vision fields of study. In this paper, we tackle the text to video generation problem, which is a conditional form of video generation. Humans can listen/read natural language sentences, and can imagine or visualize what is being described; therefore, we believe that video generation from natural language sentences will have an important impact on Artificial Intelligence. Video generation is relatively a new popular task in Computer Vision (CV), which is far from being solved. The majority of recent works deal with synthetic datasets or real datasets with very limited types of objects, scenes and motions. To the best of our knowledge, this is the very first work on the text (free-form sentences) to video generation on more realistic video datasets like Actor and Action Dataset (A2D) or UCF101. We tackle the complicated problem of video generation by regressing the latent representations of the first and last frames and employing a context-aware interpolation method to build the latent representations of in-between frames. We propose a stacking "upPooling" block to sequentially generate RGB frames out of each latent representations and progressively increase the resolution. Moreover, our proposed Discriminator encodes videos based on single and multiple frames.