Context Tracking with Llama 2: A Task-Technology Fit Analysis for Enhancing Customer Service conversations

Open Webpage

Khadija Essaied, Sehl Mellouli, Richard Khoury

Published: 01 Jan 2025, Last Modified: 15 Oct 2025AMCIS 2025EveryoneRevisionsBibTeXCC BY-SA 4.0
Abstract: Conversational agents play a crucial role in automating customer service interactions, assisting users in handling inquiries, processing requests, and managing various tasks. While large language models like GPT-4 and Llama 2 have improved conversational capabilities, challenges remain in tracking context and maintaining coherence in complex, multi-turn dialogues, especially in specialized technical domains. Their effectiveness in non-English, domain-specific customer service contexts also remains underexplored. This study investigates Llama 2’s ability to manage multi-turn dialogues in the French-language automobile insurance sector. Using the task-technology fit framework, we assess its alignment with customer service requirements, focusing on context tracking, coherence, and task completion. The model is fine-tuned on real French-language interactions and benchmarked against human agents. Contributions include insights into Llama 2’s strengths and limitations in complex dialogues, strategies for improving context-aware conversational AI, and refinement of the task-technology fit framework for conversational systems.