ChEmREF: Evaluating Language Model Readiness for Chemical Emergency Response Assistance

Risha Surana, Qinyuan Ye, Swabha Swayamdipta

Keywords: Chemistry, Healthcare, Evaluation, LLM, Medicine, Medical, Public Health, Emergency Response, Safety, Risk Assessment, Incident Management

TL;DR: We introduce ChEmREF, a benchmark to evaluate LLMs on HAZMAT emergency tasks, showing their strengths and limitations in chemical identification, response recommendations, and domain-specific knowledge.

Abstract:

Emergency responders managing hazardous material HAZMAT incidents face critical, time-sensitive decisions, manually navigating extensive chemical guidelines.
We investigate whether today's language models can assist responders by rapidly and reliably understanding critical information, identifying hazards, and providing recommendations.
We introduce the Chemical Emergency Response Evaluation Framework (ChEmREF), a new benchmark comprising questions on 1,035 HAZMAT chemicals from the Emergency Response Guidebook and the PubChem Database.
ChEmREF is organized into three tasks:
(1) translation of chemical representation between structured and unstructured forms (e.g, converting ''C$_2$H$_6$O'' to ''ethanol''), (2) emergency response generation (e.g., recommending appropriate evacuation distances) and (3) domain knowledge question answering from chemical safety and certification exams.
Our best evaluated models received an exact match of 68.0\% on unstructured HAZMAT chemical representation translation, a LLM Judge score of 52.7\% on incident response recommendations, and a multiple-choice accuracy of 63.9\% on HAZMAT examinations.
These findings suggest that while language models show potential to assist emergency responders in various tasks, they require careful human oversight due to their current limitations.

Submission Number: 183