Evaluating ChatNetZero, an LLM-Chatbot to Demystify Climate Pledges

Angel Hsu, Mason Laney, James Zhang, Diego Manya, Linda Farczadi

Keywords: large language models, climate change, net zero pledges, natural language processing, retrieval-augmented generation

TL;DR: This paper introduces and evaluates ChatNetZero, a large-language model chatbot developed through Retrieval-Augmented Generation, which uses generative AI to produce answers grounded in verified, climate-domain specific information

Abstract: This paper introduces and evaluates ChatNetZero, a large-language model (LLM) chatbot developed through Retrieval-Augmented Generation (RAG), which uses generative AI to produce answers grounded in verified, climate-domain specific information. We describe ChatNetZero's design, particularly the innovation of anti-hallucination and reference modules designed to enhance the accuracy and credibility of generated responses. To evaluate ChatNetZero's performance against other LLMs, including GPT-4, Gemini, Coral, and ChatClimate, we conduct two types of validation: comparing LLMs' generated responses to original source documents to verify their factual accuracy, and employing an expert survey to evaluate the overall quality, accuracy and relevance of each response. We find that while ChatNetZero responses show higher factual accuracy when compared to original source data, experts surveyed prefer lengthier responses that provide more context. Our results highlight the importance of prioritizing information presentation in the design of domain-specific LLMs to ensure that scientific information is effectively communicated, especially as even expert audiences find it challenging to assess the credibility of AI-generated content.

Archival Submission: arxival

Arxival Submission: arxival

Submission Number: 9