Oboyob: A sequential-semantic Bengali image captioning engine

Tonmoay Deb, Mohammad Zariff Ahsham Ali, Sanchita Bhowmik, Adnan Firoze, Syed Shahir Ahmed, Muhammad Abeer Tahmeed, N. S. M. Rezaur Rahman, Rashedur M. Rahman

Abstract: Understanding the context with generation of textual description from an input image is an active and challenging research topic in computer vision and natural language processing. However, in the case of Bengali language, the problem is still unexplored. In this paper, we address a standard approach for Bengali image caption generation though subsampling the machine translated dataset. Later, we use several pre-processing techniques with the state-of-the-art CNN-LSTM architecture-based models. The experiment is conducted on standard Flickr-8K dataset, along with several modifications applied to adapt with the Bengali language. The training caption subsampled dataset is computed for both Bengali and English languages for further experiments with 16 distinct models developed in the entire training process. The trained models for both languages are analyzed with respect to several caption evaluation metrics. Further, we establish a baseline performance in Bengali image captioning defining the limitation of current word embedding approaches compared to internal local embedding.