Vitexco: Exemplar-based Video Colorization using Vision Transformer

Duong Thanh Tran, Phuong-Nam Tran, Nguyen Doan Hieu Nguyen, Thuy-Duong Thi Vu, Trung Thanh Pham, Duc Ngoc Minh Dang

Abstract: In the field of image and video colorization, the existing research employs a CNN to extract information from each video frame. However, due to the local nature of a kernel, it is challenging for CNN to capture the relationships between each pixel and others in an image, leading to inaccurate colorization. To solve this issue, we introduce an end-to-end network called Vitexco for colorizing videos. Vitexco utilizes the power of the Vision Transformer (ViT) to capture the relationships among all pixels in a frame with each other, providing a more effective method for colorizing video frames. We evaluate our approach on DAVIS datasets and demonstrate that it outperforms the state-of-the-art methods regarding color accuracy and visual quality. Our findings suggest that using a ViT can significantly enhance the performance of video colorization.