PILL: Plug into LLM with adapter expert and attention gate

Yuyu Yin, Fangyuan Zhang, Zhengyuan Wu, Qibo Qiu, Tingting Liang, Xin Zhang

Abstract: Highlights•PILL bridges the gap between pre-trained LM and multimodal understanding.•MAG prevents visual information from interfering with the LLM’s text modeling.•MoMAE equips each modality with dedicated FFNs to address modal entanglement issue.•PILL exhibits superior efficiency and competitive performance.