LLM-based Related Work Section Generation Framework Incorporating Perspectives Researchers Value

Anonymous

Abstract: This paper proposes a Large Language Model (LLM)-based framework to generate paper's related work section, incorporating perspectives valued by researchers. While LLMs excel at summarization, ambiguous instructions limit the clarity of a generated related work section for researchers. Through the surveys, we identified the preferred perspectives for a related work section: "categorization'', "comparison'', and "pointing out problems''. We incorporate these perspectives into a prompt with few-shot examples. Furthermore, to provide the framework with explainability and aid in the fact-checking, we have the LLM select salient sentences from cited papers to extract evidences. Experimental results with human evaluation demonstrate that the generated related work section tends to be preferred over human-written ones and has fewer hallucinations. Our codes and the dataset we collected are available at https://anonymous.4open.science/r/anony_rwg/.

Paper Type: long

Research Area: Generation

Contribution Types: NLP engineering experiment

Languages Studied: English

Preprint Status: We are considering releasing a non-anonymous preprint in the next two months (i.e., during the reviewing process).

A1: yes

A1 Elaboration For Yes Or No: Limitation & Ethical Consideration

A2: yes

A2 Elaboration For Yes Or No: Limitation & Ethical Consideration

A3: yes

A3 Elaboration For Yes Or No: Abstract, 1. Introduction

B: yes

B1: yes

B1 Elaboration For Yes Or No: 1. Introduction, 2. Preliminaries, 3.2 Methodology

B2: yes

B2 Elaboration For Yes Or No: 3.2 Methodology, Limitation & Ethical Consideration

B3: yes

B3 Elaboration For Yes Or No: 3.2 Methodology, Limitation & Ethical Consideration

B4: yes

B4 Elaboration For Yes Or No: 3.2 Methodology, Limitation & Ethical Consideration (The dataset we publish contains paper information, which adhere to the CC-BY 4.0 license.)

B5: yes

B5 Elaboration For Yes Or No: 4. Experiments

B6: yes

B6 Elaboration For Yes Or No: 4.2 Dataset

C: yes

C1: no

C1 Elaboration For Yes Or No: We used OpenAI GPT-4-turbo by their official API and not our own GPU, etc.

C2: yes

C2 Elaboration For Yes Or No: 4. Experiments

C3: yes

C3 Elaboration For Yes Or No: 4.1 Evaluation Methodology, 4.3 Results

C4: yes

C4 Elaboration For Yes Or No: 1. Introduction, 2. Preliminaries, 3.2 Methodology, 4. Experiments (We used GPT-4-turbo.)

D: yes

D1: yes

D1 Elaboration For Yes Or No: 4.1 Evaluation Methodology, Appendix A Survey and Human Evaluation

D2: yes

D2 Elaboration For Yes Or No: 4.1 Evaluation Methodology, 4.2 Dataset

D3: yes

D3 Elaboration For Yes Or No: Appendix A Survey and Human Evaluation (in Figure 4 and Figure 5)

D4: no

D4 Elaboration For Yes Or No: We, members who have received ethical education (eAPRIN), discussed this issue. We concluded that the data collection process in this paper is designed solely to gather subjective opinions. Moreover, as these opinions are anonymized for use, we determined that there are no ethical concerns for participants.

D5: yes

D5 Elaboration For Yes Or No: 4.1 Evaluation Methodology, 4.2 Dataset

E: yes

E1: yes

E1 Elaboration For Yes Or No: 5. Related Work, 6. Discussion