Examining LLM's Awareness of the United Nations Sustainable Development Goals (SDGs)

Mehdi Bahrami, Ramya Srinivasan

Keywords: LLM, UN SGD, ChatGPT

TL;DR: Utilized ChatGPT to exam 8 LLMs on true/false statements and compared their performance against human-written statements

Abstract:

Utilization of Large Language Models (LLMs) is rapidly growing in diverse domains and each LLM may show different performance across different topics.
Amidst this progress, biases and other ethical concerns surrounding LLMs have raised questions regarding trust and reliability, thereby necessitating human verification and audit.
In this study, we empirically investigate six important topics of the United Nations Sustainable Development Goals (UN SDG) by utilizing ChatGPT LLM as a facilitator for generating statements needed for evaluation of eight different LLMs. We also compare the performance of these LLMs on human-written statements and questions. In addition, we study the tendency of producing of true and false statement for the eight LLMs considered. Although LLMs show comparative performance on ChatGPT and human input for relatively common issues, they are not sophisticated enough in understanding nuanced and advanced issues that demand critical and wholistic introspection.