Logical Reasoning Evaluation and Social Bias

Sofia Martinelli, Guido Ivetta, Luciana Benotti

Track: tiny / short paper (up to 4 pages)

Keywords: Large Language Models, Logical Reasoning, Social Bias, Benchmark, Reasoning Robustness

TL;DR: Reasoning in LLMs interacts with social bias, and existing benchmarks often fail to capture this interaction, motivating the integration of social bias evaluations.

Abstract: Current benchmarks for logical reasoning in Large Language Models (LLMs) typically focus on formal deduction, induction, and abduction, implicitly assuming that reasoning operates independently from social biases encoded in model representations. However, in this work we review evidence indicating that reasoning closely interacts with socially grounded priors. Recent work shows that stereotypical associations can influence both intermediate reasoning steps and final predictions. At the same time, leveraging this interaction, interventions on intermediate reasoning steps—such as filtering biased reasoning steps or self-debiasing prompts—can reduce bias. Moreover, we provide an overview of social bias benchmarks and metrics, across diverse cultural and geographic contexts, as a starting point for more comprehensive evaluation of reasoning models.

Anonymization: This submission has been anonymized for double-blind review via the removal of identifying information such as names, affiliations, and identifying URLs.

Submission Number: 165