Structure-Rich Text Benchmark for Knowledge Inference Evaluation

Haoning Ye, Zhouhong Gu, Zeyang Zhou, Sihang Jiang, Hongwei Feng, Yanghua Xiao

Primary Area: datasets and benchmarks

Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.

Keywords: Language Model, Structured Text, Benchmark, Evaluation, Dataset

Submission Guidelines: I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2024/AuthorGuide.

TL;DR: A benchmark evaluating LLMs' ability understanding structure-rich input texts

Abstract: We construct a benchmark for LLMs (Large Language Models) composed of structure-rich and syntactically rigorous corpus with mainly semantics-independent tasks, in purpose of evaluating the abilities of knowledge inference from small structured text and construction rules. The tasks also involve the capacity to generate strictly formatted response given the specification, i.e. to output the same structure-rich texts as the inputs. We also experimented on the popular LLMs with our benchmark to compare their competence to mine for information from syntax and condense information into structure.

Anonymous Url: I certify that there is no URL (e.g., github page) that could be used to find authors' identity.

No Acknowledgement Section: I certify that there is no acknowledgement section in this submission for double blind review.

Submission Number: 4523