Mix-Persona Comment Generation and Geographically Enhanced Context Retrieval for LLM Fine-Tuning in Multimodal Crisis Post Classification

Tong Bie, Yongli Hu, Yu Fu, Linjia Hao, Tengfei Liu, Kan Guo, Huajie Jiang, Junbin Gao, Yanfeng Sun, Baocai Yin

Abstract: Social media has become a vital source for humanitarian organizations to gather information during crises. However, existing multimodal classification methods operate primarily as isolated systems, while neglecting external references crucial for accurate judgment. Furthermore, while user comments can provide valuable context, they are often scarce during the early stages of a crisis. To address these limitations, we propose a framework named Mix-Persona Comment Generation with Geographically Enhanced Context Retrieval for LLM Instruction Fine-tuning (MPCG-GECR). To mitigate comment scarcity, we employ a Synthetic Persona Generator (SPG) that prompts LLMs to adopt diverse mix-personas, generating synthetic comments that simulate multi-perspective public discourse. To incorporate external references, we introduce a Geographically Enhanced Context Retrieval (GECR) module. Unlike standard retrieval approaches, GECR utilizes a hybrid re-ranking strategy to identify samples that are both multimodally similar and geographically consistent, serving as reliable reference anchors for the LLM. By integrating these social perspectives and geographic references into a unified instruction-tuning format, we transform the classification task into a context-aware text generation problem and fine-tune the LLM using Low-Rank Adaptation (LoRA). Extensive experiments on the CrisisMMD and DMD datasets demonstrate that MPCG-GECR effectively overcomes data scarcity and context isolation, significantly outperforming existing methods.