Imagine How To Change: Explicit Procedure Modeling for Change Captioning

Jiayang Sun, Zixin Guo, Min Cao, Guibo Zhu, Jorma Laaksonen

Keywords: dynamic procedure understanding, confidence-guided sampling, change captioning

Abstract:

Change captioning generates descriptions that explicitly describe the differences between two visually similar images. Existing methods operate on static image pairs, thus ignoring the rich temporal dynamics of the change procedure, which is the key to understand not only what has changed but also how it occurs. We introduce ProCap, a novel framework that reformulates change modeling from static image comparison to dynamic procedure modeling.
ProCap features a two-stage design: The first stage trains a procedure encoder to learn the change procedure from a sparse set of keyframes.
These keyframes are obtained by automatically generating intermediate frames to make the implicit procedural dynamics explicit and then sampling them to mitigate redundancy.
Then the encoder learns to capture the latent dynamics of these keyframes via a caption-conditioned, masked reconstruction task.
The second stage integrates this trained encoder within an encoder-decoder model for captioning.
Instead of relying on explicit frames from the previous stage---a process incurring computational overhead and sensitivity to visual noise---we introduce learnable procedure queries to prompt the encoder for inferring the latent procedure representation, which the decoder then translates into text. The entire model is then trained end-to-end with a captioning loss, ensuring the encoder's output is both temporally coherent and captioning-aligned. Experiments on three datasets demonstrate the effectiveness of ProCap. Code and pre-trained models are available at https://github.com/BlueberryOreo/ProCap.

Primary Area: applications to computer vision, audio, language, and other modalities

Submission Number: 7411