Large Language Models can $\textit{Share}$ Images, Too!

Young-Jun Lee, Jonghwan Hyeon, Ho-Jin Choi

Keywords: multi-modal dialogue, language model, image-sharing behavior, gpt3

TL;DR: We explore whether a large-scale language model can share images in a given dialogue without any visual foundation models.

Abstract:

This paper explores the \textit{image-sharing} capability of Large Language Models (LLMs), such as InstructGPT, ChatGPT, and GPT-4, in a zero-shot setting, without the help of visual foundation models.
Inspired by the two-stage process of image-sharing in human dialogues, we propose a two-stage framework that allows LLMs to predict potential image-sharing turns and generate related image descriptions using our effective restriction-based prompt template.
With extensive experiments, we unlock the \textit{image-sharing} capability of LLMs in zero-shot prompting, with GPT-4 achieving the best performance.
Additionally, we uncover the emergent \textit{image-sharing} ability in zero-shot prompting, demonstrating the effectiveness of restriction-based prompts in both stages of our framework.
Based on this framework, we augment the PhotoChat dataset with images generated by Stable Diffusion at predicted turns, namely PhotoChat++.
To our knowledge, this is the first study to assess the \textit{image-sharing} ability of LLMs in a zero-shot setting without visual foundation models.
The source code and the dataset will be released after publication.

Primary Area: general machine learning (i.e., none of the above)

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 1137