A Novel Large Language Model (LLM) Based Approach for Robotic Collaboration in Search and Rescue Operations

M. I. R. Shuvo, Nafiul Alam, Awal Ahmed Fime, Hannah Lee, Xiangxu Lin, Jong-Hoon Kim

Abstract: Effective robot communication in disaster circumstances improves object identification. A novel three-step procedure improves this interaction, focusing on early object identification. Robots use Intel RealSense cameras to view RGB frames and depth data to determine 3D coordinates, shapes, and orientations. Combining Claude 3, a Large Language Model (LLM), with a UCD module is our method. A shared General Knowledge (GK) repository stores object properties retrieved and verified by the LLM module’s Object Analysis (OA) and Semantic Correlation (SC) submodules. EasyOCR and Yolov8 algorithms in the UCD module recognize handwritten initials and strange symbols. In this collaborative framework, robots assess and develop each other’s discoveries, improving item recognition and dynamic ownership determination. Our analysis shows the method works in robotic disaster response.

External IDs:dblp:conf/iecon/ShuvoAFLLK24