SPAGHETTI: Open-Domain Question Answering from Heterogeneous Data Sources with Retrieval and Semantic Parsing

Anonymous

Abstract: We introduce SPAGHETTI: Semantic Parsing Augmented Generation for Hybrid English information from Text Tables and Infoboxes, a hybrid question-answering (QA) pipeline that utilizes information from heterogeneous knowledge sources, including knowledge base, text, tables, and infoboxes. Our LLM-augmented approach achieves state-of-the-art performance on the Compmix dataset, the most comprehensive heterogeneous open-domain QA dataset, with 56.5% exact match (EM) rate. More importantly, manual analysis on a sample of the dataset suggests that SPAGHETTI is more than 90% accurate, indicating that EM is no longer suitable for assessing the capabilities of QA systems today.

Paper Type: short

Research Area: Question Answering

Contribution Types: NLP engineering experiment, Data analysis

Languages Studied: English

Preprint Status: We plan to release a non-anonymous preprint in the next two months (i.e., during the reviewing process).

A1: yes

A1 Elaboration For Yes Or No: After section 6, we have a "Limitations" section.

A2: yes

A2 Elaboration For Yes Or No: We have an "Ethical Considerations" section in the end.

A3: yes

A3 Elaboration For Yes Or No: Abstract is on the first page and Introduction is section 1.

B: yes

B1: yes

B1 Elaboration For Yes Or No: Section 2, 3, and 4

B2: yes

B2 Elaboration For Yes Or No: Ethical considerations section

B3: yes

B3 Elaboration For Yes Or No: Ethical considerations section

B4: n/a

B4 Elaboration For Yes Or No: We work with Wikipedia, which is a public information that does not involve privacy issues.

B5: yes

B5 Elaboration For Yes Or No: Section 2

B6: yes

B6 Elaboration For Yes Or No: Section 4

C: yes

C1: yes

C1 Elaboration For Yes Or No: Ethical considerations section

C2: yes

C2 Elaboration For Yes Or No: Section 4

C3: yes

C3 Elaboration For Yes Or No: Section 5

C4: yes

C4 Elaboration For Yes Or No: Section 3

D: no

D1: n/a

D2: n/a

D3: n/a

D4: n/a

D5: n/a

E: no

E1: n/a