The Dark Side of Empathy: When Dialogue Systems Face Moral Challenges

Zhexin Zhang, Jiale Cheng, Jiawen Deng, Minlie Huang

Keywords: empathy, morality, dialogue

Abstract:

Empathy has long been considered a key component in building helpful and trustful dialogue systems. However, as these systems are used in a broad range of scenarios, they may also face many challenging contexts requiring them to show empathy in a more elaborate manner. In this paper, we investigate the dark side of the dialogue model's empathy in responding to immoral contexts, a typical collection of contexts that can induce empathetic but inappropriate responses due to the misuse of empathy strategies. Specifically, we summarize five types of adversarial contexts containing unethical behaviors and construct a corresponding dataset containing 10K adversarial contexts. Then, we use these contexts to assess popular traditional conversational models and demonstrate their vulnerability in dealing with such attacks. To address this problem, we design a pipeline approach to construct responses that incorporate ethics and empathy based on rules of thumb (RoTs). We show that after being fine-tuned on the constructed responses, traditional dialogue models exhibit improved ability to handle the adversarial contexts containing unethical behaviors. We also manually create a hard test containing implicitly immoral contexts. Experiments demonstrate that even SOTA AI assistants such as ChatGPT and Claude would generate significantly more immoral responses when responding to these implicitly immoral contexts with empathy. Based on experiments conducted on both traditional dialogue systems and advanced AI assistants, we systematically summarize the impact of empathy when dialogue systems face moral challenges.
We will release our code and data to facilitate further research.

Primary Area: societal considerations including fairness, safety, privacy

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 3329