A Neurally Guided Patch-Based Style Transfer for Mobile Devices

Jose Ivson S. Silva, Kevin Ian Ruiz Vargas, Antônio A. Carlos, Lucas P. de Albuquerque, Mateus Baltazar de Almeida, Allan Soares Vasconcelos, Victor Ximenes C. Oliveira, José Gabriel P. Tavares, Danilo Vaz Marcolino Alves, Diêgo J. C. Santiago, Bernardo Augusto de Oliveira, Carlos Padilha, Tsang Ing Ren

Abstract: Style transfer is an application that has increased interest, primarily because of the impressive results obtained using neural networks. However, this application demands a lot of computational resources, thus preventing its use in low-end mobiles. The patch-based approach is an interesting alternative that consumes less memory. This work uses two methods to generate high-resolution stylized images. Gated convolution in the neural network and the Halide language in the patch-based implementation are used for optimization. As a result, we were able to apply the style transfer method on a mobile device with 4 GB of memory RAM running under 6 seconds for $1920\times 1080$ image size preserving the high-frequency details.