An Empirical Study of Focus Identification in Large Language Models with Modal Contexts
Abstract: We examine whether pre-trained large language models GPT-2 and Flan-T5 use prior context to accurately identify the focus of negated sentences. We present to the models procedurally-generated negated sentences and procedurally-generated prior contexts to prime the models to generate text coherent with a particular focus in the negated sentences. We examine model accuracy with and without explicit modal operators for necessity and possibility and formalize our examples in modal logic. We find that larger models of the same class are much more likely to generate sentence continuations that cohere with the provided context, but even the larger models often struggle to outperform a random baseline with respect to accuracy. Certain necessity modals in the prior context greatly increase the likelihood of a continuation that is both relevant to the context and consistent with the primed negation focus.
Submission Length: Regular submission (no more than 12 pages of main content)
Changes Since Last Submission: - Added short intro to logic and logic notation.
- Corrected minor typos.
- Added a bit about relevance to conclusion and discussion section.
- Said more to justify and contextualize prompt usage.
- Uploaded research code and CSVs.
- Added footnote mentioning that annotator is an author of the paper.
- Minor update to abstract
- Move first example to earlier section from Experiments
- Add footnote justifying the task setup and addressing the subjectivity issues.
Assigned Action Editor: ~Greg_Durrett1
Submission Number: 1412
Loading