Integrating Visual Cues via Prompting for Low-Resource Multimodal Named Entity Recognition
Primary Area: representation learning for computer vision, audio, language, and other modalities
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Keywords: Named Entity Recognition, Deep Learning, Prompting, Visual Cues, Multimodality, Few shot
Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.
TL;DR: By reformulating multimodal named entity recognition as open-ended question answering, our method obtains strong zero-shot performance in experiments.
Abstract: In the field of Natural Language Processing (NLP), the task of Named Entity Recognition (NER) is quite established. However, most existing methods predominantly rely on textual data alone, overlooking the information that can be derived from other modalities such as images. This issue is particularly pronounced in low-resource settings, where the absence of extensive labeled data can significantly impede the performance of NER systems. Existing solutions, while attempting to address this limitation, often require comprehensive fine-tuning and are not readily applicable in such low-resource conditions. This research confronts these challenges by proposing a novel approach to Multimodal Named Entity Recognition (MNER) under low-resource constraints. We recast the MNER task as an open-ended question-answering problem, particularly suitable for modern generative language models. Our findings provide novel insights into the complex interplay between model design, prompt crafting, and training data characteristics that determine the efficacy of visual integration. The strengths and limitations elucidated can inform future efforts at the intersection of multimodal representation learning, generative modeling, and prompting.
Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.
No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.
Submission Number: 5229
Loading