Fantasy: Transformer Meets Transformer in Text-to-Image Generation

Enxin Song, Wenhao Chai, Xun Guo, Gaoang Wang, Jenq-Neng Hwang, Yan Lu

Keywords: Text-to-Imgae Generation, Transformer, Masked Image Modeling

Abstract: We present Fantasy, an efficient text-to-image generation model marrying the decoder-only Large Language Models (LLMs) and transformer-based masked image modeling (MIM). While diffusion models are currently in a leading position in this task, we demonstrate that with appropriate training strategies and high-quality data, MIM can also achieve comparable performance. By incorporating pre-trained decoder-only LLMs as the text encoder, we observe a significant improvement in text fidelity compared to the widely used CLIP text encoder, enhancing the text-image alignment. Our training approach involves two stages: 1) large-scale concept alignment pre-training, and 2) fine-tuning with high-quality instruction-image data. Evaluations on FID, HPSv2 benchmarks, and human feedback demonstrate the competitive performance of Fantasy against state-of-the-art diffusion and autoregressive models.

Primary Area: Generative models

Submission Number: 4342