Llama meets EU: Investigating the European political spectrum through the lens of LLMs

Anonymous

TL;DR: We use political debates from the European parliament to analyse Llama-Chat's political knowledge and reasoning ability.

Abstract: Instruction-finetuned Large Language Models (LLMs) inherit clear political leanings which has been shown to influence downstream task performance. We expand this line of research beyond the two-party system in the US and audit Llama Chat on political debates from the EU parliament in various settings to analyze the model's political knowledge and its ability to reason in context. We adapt, i.e., further fine-tune, Llama Chat on parliamentary debates of individual europarties to reevaluate its political leaning based on the euandi questionnaire. Llama Chat shows extensive prior knowledge of party positions and is capable of reasoning in context. The adapted, party-specific, models are substantially re-aligned towards respective positions which we see as a starting point for using chat-based LLMs as data-driven conversational engines to assist research in political science. We release our code, the new datasets and adapted models to foster future research.

Paper Type: short

Research Area: Computational Social Science and Cultural Analytics

Contribution Types: Model analysis & interpretability, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English