Image-Generation AI Model Retrieval by Contrastive Learning-Based Style Distance Calculation

Vu Thi Ngoc Anh, Yoshiyuki Shoji, Yuma Oe, Huu-Long Pham, Hiroaki Ohshima

Abstract: This paper proposes a method for retrieving trained image-generation LoRA (Low-Rank Adaptation) models. This search algorithm takes a single arbitrary image input and then ranks the models in the order in which they will likely transform the image to the same style as the input image. We adopted a contrastive learning approach using a Triplet Network (Siamese network with triplet loss). We created a sample image set and performed style transfers on the pre-collected LoRA models to be retrieved. Using these transferred images, the network was fine-tuned to calculate the distance by their style rather than by their subject; the distance becomes large for pairs of images of the same subject transformed by different LoRA models and small for pairs of images of different subjects transformed by the same LoRA model. The search algorithm was evaluated through accuracy assessment tasks that estimated whether two images were transformed by the same model and user experiments that ranked the models. The experimental results demonstrated that fine-tuning is crucial and that the diversity of the sample image set is also important.

External IDs:dblp:conf/mmm/AnhSOPO25