Biased-Predicate Annotation Identification via Unbiased Visual Predicate Representation

Li Li, Chenwei Wang, You Qin, Wei Ji, Renjie Liang

Abstract:

Panoptic Scene Graph Generation (PSG) translates visual scenes to linguistic descriptions, i.e., mapping visual instances to subjects/objects, and their relationships to predicates.
However, annotators' preferences and semantic overlaps between predicates inevitably lead to different predicate mapping of one relationship, i.e., biased predicate annotations.
As a result, with the contradictory mapping between visual and linguistics, it is hard for PSG models to construct clear decision planes among predicates, leading to poor PSG performances.
Obviously, it is essential for PSG task to tackle with this multimodal contradictory. Therefore, we propose a new method that utilizes unbiased visual predicate representations for Biased-Annotation Identification, named BAI. Our BAI includes three main steps, which are predicate representation extraction, predicate representation debiasing, and biased-annotation identification.
With various biased-annotation processing methods, our BAI can act as a basic approach of PSG dataset debiasing.
Experimental results demonstrate the effectiveness of BAI, which promotes the performance of benchmark models using simple and intuitive biased-annotation processing methods, achieving state-of-the-art performance. Furthermore, our BAI shows great generalization and validity on multiple datasets. Our code and unbiased annotation are released at https://anonymous.4open.science/r/ACMMM2023-F463.