Inducing Human-like Biases in Moral Reasoning Language Models

Austin Meek, Artem Karpov, Seong Hah Cho, Raymond Koopmanschap, Lucy Farnik, Bogdan-Ionut Cirstea

Track: Extended Abstract Track

Keywords: BrainScore, moral reasoning, large language model, theory of mind, fine-tuning, alignment, fMRI

TL;DR: We describe an approach to try increasing alignment between fMRI data and large language models on moral reasoning tasks; however we fail to achieve significant results beyond our control group.

Abstract:

In this work, we study the alignment (BrainScore) of large language models (LLMs)
fine-tuned for moral reasoning on behavioral data and/or brain data of humans
performing the same task. We also explore if fine-tuning several LLMs on the
fMRI data of humans performing moral reasoning can improve the BrainScore.
We fine-tune several LLMs (BERT, RoBERTa, DeBERTa) on moral reasoning
behavioral data from the ETHICS benchmark Hendrycks et al. [2020], on the
moral reasoning fMRI data from Koster-Hale et al. [2013], or on both. We study
both the accuracy on the ETHICS benchmark and the BrainScores between model
activations and fMRI data. While larger models generally performed better on both
metrics, BrainScores did not significantly improve after fine-tuning.

Submission Number: 74