WebCiteS: Attributed Query-Focused Summarization on Chinese Web Search Results with Citations

Anonymous

Abstract: Enhancing the attribution in large language models (LLMs) is a crucial task. One feasible approach is to enable LLMs to cite external sources that support their generations. However, existing datasets and evaluation methods in this domain still exhibit notable limitations. In this work, we formulate the task of attributed query-focused summarization (AQFS) and present WebCiteS, a Chinese dataset featuring 7k human-annotated summaries with citations. WebCiteS derives from real-world user queries and web search results, offering a valuable resource for model training and evaluation. Prior works in attribution evaluation do not differentiate between groundedness errors and citation errors. They also fall short in automatically verifying sentences that draw partial support from multiple sources. We tackle these issues by developing detailed metrics and enabling the automatic evaluator to decompose the sentences into sub-claims for fine-grained verification. Our comprehensive evaluation of both open-source and proprietary models on WebCiteS highlights the challenge LLMs face in correctly citing sources, underscoring the necessity for further improvement. The dataset and code will be open-sourced to facilitate further research in this crucial field.

Paper Type: long

Research Area: Resources and Evaluation

Contribution Types: NLP engineering experiment, Publicly available software and/or pre-trained models, Data resources, Data analysis

Languages Studied: Chinese

Preprint Status: We plan to release a non-anonymous preprint in the next two months (i.e., during the reviewing process).

A1: yes

A1 Elaboration For Yes Or No: Section Limitation

A2: yes

A2 Elaboration For Yes Or No: Section Limitation

A3: yes

A3 Elaboration For Yes Or No: Section 1

B: yes

B1: yes

B1 Elaboration For Yes Or No: Section 4 and 5

B2: yes

B2 Elaboration For Yes Or No: Appendix C, D, E

B3: yes

B3 Elaboration For Yes Or No: Section 1

B4: yes

B4 Elaboration For Yes Or No: Section 2

B5: yes

B5 Elaboration For Yes Or No: Section 2

B6: yes

B6 Elaboration For Yes Or No: Section 2

C: yes

C1: yes

C1 Elaboration For Yes Or No: Appendix E

C2: yes

C2 Elaboration For Yes Or No: Appendix E

C3: yes

C3 Elaboration For Yes Or No: Appendix E

C4: yes

C4 Elaboration For Yes Or No: Appendix B

D: yes

D1: n/a

D2: n/a

D3: yes

D3 Elaboration For Yes Or No: Appendix A

D4: n/a

D5: n/a

E: yes

E1: yes

E1 Elaboration For Yes Or No: Section 2,4,5