EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental Health Safety

ACL ARR 2025 May Submission5994 Authors

Abstract:

The rise of LLM-driven AI characters raises safety concerns, particularly for vulnerable human users with psychological disorders. To address these risks, we propose **EmoAgent**, a multi-agent AI framework designed to evaluate and mitigate mental health hazards in human-AI interactions.
EmoAgent comprises two components: **EmoEval** simulates virtual users, including those portraying mentally vulnerable individuals, to assess mental health changes before and after interactions with AI characters. It uses clinically proven psychological and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks induced by LLM. **EmoGuard** serves as an intermediary, monitoring users’ mental status, predicting potential harm, and providing corrective feedback to mitigate risks.
Experiments conducted in popular character-based chatbots show that emotionally engaging dialogues can lead to psychological deterioration in vulnerable users, with mental state deterioration in more than 34.4% of the simulations. EmoGuard significantly reduces these deterioration rates, underscoring its role in ensuring safer AI-human interactions.

Paper Type: Long

Research Area: Human-Centered NLP

Research Area Keywords: human-AI interaction, multi-agent, mental health NLP, safety alignment

Contribution Types: NLP engineering experiment

Languages Studied: English

Submission Number: 5994