Large-Scale Analysis of User Feedback on AI-Powered Mobile Apps

Download PDF
Open Webpage

Vinaik Chhetri, Krishna Upadhyay, A. B. Siddique, Umar Farooq

Published: 2025, Last Modified: 26 May 2026IEEE Big Data 2025EveryoneRevisionsBibTeXCC BY-SA 4.0
Abstract: Artificial Intelligence (AI)-powered features have rapidly proliferated across mobile apps in various domains, including productivity, education, entertainment, and creativity. However, how users perceive, evaluate, and critique these AI features remains largely unexplored, primarily due to the overwhelming volume of user feedback. In this work, we present the first large-scale, data-driven study of user feedback on AIpowered mobile apps, leveraging a curated dataset of 292 AIdriven apps across 14 categories with nearly one million AI-specific reviews from Google Play. We develop and validate a multi-stage analysis pipeline that begins with a human-labeled benchmark and systematically evaluates large language models (LLMs) and prompting strategies. Each stage, including review classification, aspect-sentiment extraction, and clustering, is validated for accuracy and consistency. Our pipeline enables scalable, highprecision analysis of user feedback, extracting over one million aspect-sentiment pairs clustered into 18 positive and 15 negative user topics. Our analysis reveals that users consistently focus on a narrow set of themes: positive comments emphasize productivity, reliability, and personalized assistance, while negative feedback highlights technical failures (e.g., scanning and recognition), pricing concerns, and limitations in language support. Our pipeline surfaces both satisfaction with one feature and frustration with another within the same review. These fine-grained, co-occurring sentiments are often missed by traditional approaches that treat positive and negative feedback in isolation or rely on coarsegrained analysis. To this end, our approach provides a more faithful reflection of the real-world user experiences with AIpowered apps. Category-aware analysis further uncovers both universal drivers of satisfaction and domain-specific frustrations. We expect our findings to advance understanding of user-centered development of AI-powered features and provide actionable guidance to software engineers and app developers who seek to align AI features of apps with user expectations.