Investigating Human and LLMs' Decisions in Unverifiable Environments: A Case Study with GitHub Activity Overview

ACL ARR 2026 January Submission9078 Authors

Keywords: Large Language Model, Decision-making, Unverifiable Environments

Abstract: The behaviors of Large Language Models (LLMs) as artificial social actors are largely underexplored, particularly in unverifiable scenarios where conventional benchmarking has little to help improve their abilities. Thus, examining their behaviors in such scenarios can help understand and improve LLMs' capabilities of simulating real-world social actors in many tasks such as LLM-empowered social agents. We draw a typical unverifiable scenario--a simplified pull request scenario on \textsc{GitHub} focusing on decision-making based on Activity Overview signal--to investigate how human and LLMs behave. We introduce a systematic method to collect, compare, and reason about human and LLMs' decisions. Our results reveal that there are both similarities and differences between human and LLMs' decisions, and proprietary LLMs generally behave more like human than open-source LLMs do. We further find that human and LLMs may rely on different information and reasoning mechanisms in decision-making. Our study thus urges more future work on human and LLMs decision-making in unverifiable environments.

Paper Type: Long

Research Area: Human-AI Interaction/Cooperation and Human-Centric NLP

Research Area Keywords: human-centered evaluation

Contribution Types: Model analysis & interpretability

Languages Studied: English

Submission Number: 9078