Breaking the Cloud Barrier: Local Deployment of Conversational LLMs for Real Time NPC Interaction in Video Games

Alec Ortiz, Hossain Samar Qorbani, Mohammed Elmorsy

Keywords: Representation Learning, Multimodal Learning, Speech Processing, Natural Language Processing, Offline AI Systems, Real-Time Interaction

TL;DR: This work demonstrates a locally deployed AI companion system integrating speech recognition, fine-tuned language models, and speech synthesis to enable real-time, privacy-preserving NPC interactions in high-fidelity game environments.

Abstract: This paper explores the integration of a locally hosted Large Language Model (LLM), with Speech-to-Text (STT) and Text-to-Speech (TTS) systems within a real-time, open-world 3D video game produced on Unreal Engine 5. The research evaluates two primary aspects: the computational performance of AI processing on CPU versus GPU configurations and the impact of low-latency AI interactions on player engagement and immersion. Performance benchmarks are conducted to analyze system resource utilization, latency, and frame stability, while user experience assessments gauge the AI companion’s effectiveness in enhancing gameplay dynamics. The results provide insights into the feasibility of offline-AI-driven NPCs, highlighting the trade-offs in computational efficiency and real-time interaction quality. This study provides a practical framework for real-time, offline AI driven game mechanics, enabling immersive and private interactions without cloud infrastructure.

Primary Area: foundation or frontier models, including LLMs

Submission Number: 6232