SciGen: a Dataset for Reasoning-Aware Text Generation from Scientific TablesDownload PDF

Published: 11 Oct 2021, Last Modified: 23 May 2023NeurIPS 2021 Datasets and Benchmarks Track (Round 2)Readers: Everyone
Keywords: Benchmark, Reasoning, Data-to-Text Generation, Scientific Articles
TL;DR: In this paper, we introduce the first reasoning-aware data-to-text generation dataset based on scientific articles and propose a pipeline for automatically extracting high-quality unsupervised training data
Abstract: We introduce SciGen, a new challenge dataset consisting of tables from scientific articles and their corresponding descriptions, for the task of reasoning-aware data-to-text generation. Describing scientific tables goes beyond the surface realization of the table content and requires reasoning over table values. The unique properties of SciGen are that (1) tables mostly contain numerical values, and (2) the corresponding descriptions require arithmetic reasoning. SciGen is the first dataset that assesses the arithmetic reasoning capabilities of generation models on complex input structures, such as tables from scientific articles, and thus it opens new avenues for future research in reasoning-aware text generation and evaluation. The core part of SciGen, including the test data, is annotated by one of the authors of the corresponding articles. Such expert annotations do not scale to large training data sizes. To tackle this, we propose a pipeline for automatically extracting high-quality table-description pairs from the LaTeX sources of scientific articles. We study the effectiveness of state-of-the-art data-to-text generation models on SciGen and evaluate the results using common metrics and human evaluation. Our results and analyses show that adding high-quality unsupervised training data improves the correctness and reduces the hallucination in generated descriptions, however, the ability of state-of-the-art models is still severely limited on this task.
URL: https://github.com/UKPLab/SciGen
Supplementary Material: zip
Contribution Process Agreement: Yes
Dataset Url: https://github.com/UKPLab/SciGen
License: Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License
Author Statement: Yes
11 Replies

Loading