Multi-modal constraint propagation via compatible conditional distribution reconstruction

Yaoyi Li, Hongtao Lu

Abstract: In many applications, the pairwise constraint is a kind of weaker supervisory information which can be collected more efficiently than the label information. The constraint propagation is a constrained clustering approach, which has been proved to be a success of exploiting such side-information. Recent years have witnessed many methods of multi-modal constraint propagation. However, the problem of reasonably fusing different modalities under the framework of constraint propagation remains unaddressed. In this paper, we first identify a necessary and sufficient condition for compatible conditional distributions under a specific assumption and address the problem of Compatible Conditional Distributions Reconstruction (CCDR). With the help of CCDR, we propose a multi-modal constraint propagation method dubbed Instance Level Multi-Modal Constraint Propagation (ILMCP). ILMCP fuses the affinity of different modalities at the data instance level instead of the modality level and constructs a unified affinity matrix. Extensive experiments on two publicly available multi-modal datasets show the superior performance of the proposed method.