Adaptive Trimodal Fusion for Mental-Health Symptom Classification in Memes

Arush Gumber

Keywords: multimodal classification, mental health, memes, social media, AI for social good, frozen encoders, gated fusion, supervised contrastive learning

TL;DR: A 1.7M-parameter trimodal fusion head on frozen CLIP and MentalBART encoders for fine-grained anxiety and depression symptom classification in memes, reaching state-of-the-art macro-F1 on AxiOM.

Abstract: Public social-media memes often express anxiety and depression through irony, visual metaphor, and figurative reference, carrying psychological signal that is frequently missed by purely textual analysis. Models for this task may support aggregate public-health research and moderation triage, provided their outputs are treated as uncertain signals rather than individual-level assessments. We study fine-grained anxiety and depression symptom classification on the AxiOM and RESTORE benchmarks and propose TCAF, a trimodal classifier that operates on top of three frozen pretrained encoders: a vision encoder over the meme image and a frozen MentalBART text encoder applied separately to OCR text and to figurative reasoning triples. The trainable head combines an Adaptive Gated Trimodal Fusion (AGTF) block that learns a per-dimension soft mask over the concatenated modalities, an additional three-way modality gate, a dual classifier head that blends a linear MLP with learnable class prototypes, and a supervised contrastive auxiliary loss on the fused representation. With only $1.7$M trainable parameters, TCAF reaches a test macro-F1 of $0.7067$ on AxiOM and $0.6857$ on RESTORE, exceeding the prior trimodal state of the art on AxiOM under the same split, with strong results on the official RESTORE test set. We further provide a per-class analysis of modality weighting and a geometric study of the learned class prototypes.

Email Sharing: We authorize the sharing of all author emails with Program Chairs.

Data Release: We authorize the release of our submission and author names to the public in the event of acceptance.

Submission Number: 427