CL-Gen: An Inference-Time Iterative Optimization Framework for Reference-Consistent Image Generation Based on Closed-Loop Control

Baixuan Zhao, Xinyu Zhang, Huayu Zheng, Shuaicheng Liu, Xiongkuo Min, Guangtao Zhai, Xiaohong Liu

Keywords: Image generation, Closed loop control, Iterative optimization, Reference consistency

TL;DR: This paper proposes a closed-loop framework to iteratively optimize reference consistency in generated images at inference time.

Abstract: Controllable image generation technology enables precise content synthesis based on user-provided reference conditions, garnering significant research attention. However, existing methods still face significant challenges in maintaining reference consistency, as they lack the observation and error correction for the reference consistency of generated images. Inspired by the concept of closed-loop systems in control theory, we propose a framework that enhances reference consistency through an iterative optimization scheme during inference time. It takes the image generation model as the control plant, observes and feeds back the actual generation state, and then adjusts the input of the generation model through a modified PID (Proportianl Integral Derivative) controller to enhance reference consistency. This framework supports a variety of controllable generation methods and different types of control conditions. Moreover, it is easy to implement, requiring only the addition of a few lines of code without the need for extra training. We validate the application of this framework in three key tasks: identity-preserving portrait generation, pose-controlled generation, and depth-controlled generation. For identity-preserving portrait generation, our method improves facial similarity by 12.07\%. For pose-controlled and depth-controlled generation, errors are reduced by 32.64\% and 33.49\%, respectively. This work not only provides a solution for reference-consistent image generation but also offers a new perspective: controllable image generation can be conceptualized as a control problem, wherein control theory is amenable to application for performance optimization. Our code will be released upon publication.

Primary Area: generative models

Submission Number: 2798