Learning Document Graphs with Attention for Image Manipulation Detection

Hailey Joren, Otkrist Gupta, Dan Raviv

Abstract: Detecting manipulations in images is becoming increasingly important for combating misinformation and forgery. While recent advances in computer vision have lead to improved methods for detecting spliced images, most state-of-the-art methods fail when applied to images containing mostly text, such as images of documents. We propose a deep-learning method for detecting manipulations in images of documents which leverages the unique structured nature of these images in comparison with those of natural scenes. Specifically, we re-frame the classic image splice detection problem as a node classification problem, in which Optical Character Recognition (OCR) bounding boxes form nodes and edges are added according to a text-specific distance heuristic. We propose a Variational Autoencoder (VAE)-based embedding algorithm, which when combined with a graph neural network with attention, outperforms both a state-of-the-art image splice detection method and a document-specific method.