Investigating Reasoning Capabilities of LLMs over Long Documents

Anonymous

Abstract:

As we embark on a new era of LLMs, it becomes increasingly crucial to understand their
capabilities, limitations, and differences. Toward making further progress in this direction, we strive to build a deeper understanding of the gaps between massive LLMs (e.g., ChatGPT) and smaller yet effective open-source LLMs and their distilled counterparts. To this end, we
specifically focus on reasoning over long input context because it has several practical and impactful applications (e.g., troubleshooting,
customer service, etc.) yet is still understudied and challenging for LLMs. We propose a question-generation method from abstractive014
summaries and show that generating follow-up questions from summaries can create a challenging setting for LLMs to reason and infer
from long contexts. Our experimental results confirm that: (1) our proposed method of generating questions from abstractive summaries
pose a challenging setup for LLMs and shows performance gaps between LLMs like ChatGPT and open-source LLMs (2) open-source
LLMs exhibit decreased reliance on context for generated questions from the original document, but their generation capabilities drop
significantly on generated questions from summaries – especially for longer contexts

Paper Type: short

Research Area: Interpretability and Analysis of Models for NLP

Contribution Types: Model analysis & interpretability

Languages Studied: English