Persona is a Double-Edged Sword: Rethinking the Impact of Role-play Prompts in Zero-shot Reasoning Tasks

ACL ARR 2025 July Submission338 Authors

Abstract:

Recent studies have shown that prompting large language models (LLMs) with role-playing personas can enhance their reasoning capabilities. While the benefits of role-playing personas in reasoning tasks are widely recognized, it remains uncertain whether a persona aligned with the given dataset can consistently achieve these improvements. In this work, we empirically investigate the potential drawbacks of using dataset-aligned personas (referred to as **coarsely aligned personas**) and introduce Jekyll \& Hyde, a novel framework that enhances reasoning robustness by ensembling solutions from both role-playing and neutral (non-persona) prompts.
Jekyll \& Hyde first predicts an instance-specific persona tailored to each query using an LLM, then generates answers with both persona and neutral prompts, and finally selects the superior output through an LLM-based evaluator.
Experimental results claim that across twelve widely used natural language reasoning datasets and three backbone large language models, Jekyll \& Hyde consistently outperforms single-perspective LLMs, achieving an average accuracy gain of **9.98\%** on GPT‑4.
We further demonstrate that using instance‑aligned personas yields more accurate and stable performance than using dataset-aligned personas.

Paper Type: Long

Research Area: Generation

Research Area Keywords: text-to-text generation, prompting, interactive and collaborative generation

Contribution Types: NLP engineering experiment

Languages Studied: English

Previous URL: https://openreview.net/forum?id=nwPGunqL3F

Reassignment Request Area Chair: Yes, I want a different area chair for our submission

Reassignment Request Reviewers: Yes, I want a different set of reviewers

Justification For Not Keeping Action Editor Or Reviewers: I have change the Research Area of the paper.

A1 Limitations Section: This paper has a limitations section.

A2 Potential Risks: N/A

B Use Or Create Scientific Artifacts: Yes

B1 Cite Creators Of Artifacts: Yes

B1 Elaboration: 4.1

B2 Discuss The License For Artifacts: Yes

B2 Elaboration: Appendix C

B3 Artifact Use Consistent With Intended Use: N/A

B4 Data Contains Personally Identifying Info Or Offensive Content: N/A

B5 Documentation Of Artifacts: Yes

B5 Elaboration: Appendix C

B6 Statistics For Data: Yes

B6 Elaboration: Appendix C

C Computational Experiments: Yes

C1 Model Size And Budget: Yes

C1 Elaboration: 4.1

C2 Experimental Setup And Hyperparameters: Yes

C2 Elaboration: J, K

C3 Descriptive Statistics: Yes

C3 Elaboration: Table 2, 3

C4 Parameters For Packages: N/A

D Human Subjects Including Annotators: No

D1 Instructions Given To Participants: N/A

D2 Recruitment And Payment: N/A

D3 Data Consent: N/A

D4 Ethics Review Board Approval: N/A

D5 Characteristics Of Annotators: N/A

E Ai Assistants In Research Or Writing: No

E1 Information About Use Of Ai Assistants: N/A

Author Submission Checklist: yes

Submission Number: 338