Instruction-ViT: Multi-modal prompts for instruction learning in vision transformer

Zhenxiang Xiao, Yuzhong Chen, Junjie Yao, Lu Zhang, Zhengliang Liu, Zihao Wu, Xiaowei Yu, Yi Pan, Lin Zhao, Chong Ma, Xinyu Liu, Wei Liu, Xiang Li, Yixuan Yuan, Dinggang Shen, Dajiang Zhu, Dezhong Yao, Tianming Liu, Xi Jiang

Abstract: Highlights•An Instruction-ViT model is to design prompts based on instruction tuning in ViT.•Multi-modal (text and image) prompts are fused to fine-tune the model.•Model performance and adaptability are improved in several image understanding tasks.•A novel strategy to fuse multi-modal prompts for visual models is offered.