Competing for Pixels: A Self-Play Algorithm for Weakly-Supervised Semantic Segmentation

Shaheer U. Saeed, Shiqi Huang, João Ramalhinho, Iani J. M. B. Gayo, Nina Montaña Brown, Ester Bonmati, Stephen P. Pereira, Brian R. Davidson, Dean C. Barratt, Matthew J. Clarkson, Yipeng Hu

Abstract: Weakly-supervised semantic segmentation (WSSS) methods, reliant on image-level labels indicating object presence, lack explicit correspondence between labels and regions of interest (ROIs), posing a significant challenge. Despite this, WSSS methods have attracted attention due to their much lower annotation costs compared to fully-supervised segmentation. Leveraging reinforcement learning (RL) self-play, we propose a novel WSSS method that gamifies image segmentation of a ROI. We formulate segmentation as a competition between two agents that compete to select ROI-containing patches until exhaustion of all such patches. The score at each time-step, used to compute the reward for agent training, represents likelihood of object presence within the selection, determined by an object presence detector pre-trained using only image-level binary classification labels of object presence. Additionally, we propose a game termination condition that can be called by either side upon exhaustion of all ROI-containing patches, followed by the selection of a final patch from each. Upon termination, the agent is incentivised if ROI-containing patches are exhausted or disincentivised if a ROI-containing patch is found by the competitor. This competitive setup ensures minimisation of over- or under-segmentation, a common problem with WSSS methods. Extensive experimentation across four datasets demonstrates significant performance improvements over recent state-of-the-art methods.

External IDs:dblp:journals/pami/SaeedHRGBBPDBCH25