Demo: A Real Time Question Answering System for Multimodal Sensors using LLMs

Xiaofan Yu, Lanxiang Hu, Benjamin Z. Reichman, Rushil Chandrupatla, Dylan Chu, Xiyuan Zhang, Larry Heck, Tajana Rosing

Abstract: Question Answering (QA) establishes a natural and intuitive way for humans to interpret and understand multimodal sensor data. However, existing sensor-based QA systems are limited in the types of questions & answers, and the duration of sensor data they can handle. In this demo, we introduce an end-to-end QA system for long-term multimodal timeseries sensors powered by Large Language Models (LLMs). Our system features a novel pipeline with LLM-based question decomposition, sensor data query and LLM-based answer assembly. We further quantize the LLMs and deploy our system on two typical edge platforms, delivering higher-quality answers with low latency.