Leveraging Codebook Knowledge with NLI and ChatGPT for Political Zero-Shot Relation Classification

Anonymous

TL;DR: Zero-shot approaches, including NLI and ChatGPT, for political event ontology relation classification by only using knowledge from an annotation codebook.

Abstract: We explore zero-shot approaches for political event ontology relation classification, leveraging knowledge from an annotation codebook. Our study includes the ChatGPT models (GPT-3.5 and 4) and introduces a novel natural language inference (NLI) based model called ZSP. ZSP adopts a tree-query framework that breaks down the task into context, modality, and class disambiguation levels. This improves interpretability, efficiency, and adaptability to schema changes. Through experiments conducted on our newly-built datasets, we identify both the potential and instability of GPT-3.5/4 in fine-grained classification. Furthermore, our findings demonstrate the superiority of ZSP, which achieves an impressive 40% improvement in F1 score for fine-grained Rootcode classification compared to conventional methods. ZSP's performance even rivals that of supervised models, positioning it as a valuable tool for event record validation and ontology development. Our work underscores the potential of leveraging transfer learning and existing expertise to enhance the efficiency and scalability of research in the field.

Paper Type: long

Research Area: NLP Applications

Contribution Types: NLP engineering experiment, Approaches to low-resource settings, Approaches low compute settings-efficiency, Publicly available software and/or pre-trained models, Data resources

Languages Studied: English