The Impact of AI Usage and Informativeness on Skill Development in Logical Reasoning

Shang Wu, Randol H. Yao, Catarina G Belém, Shuyuan Fu, Mark Steyvers, Padhraic Smyth

Abstract:

(To appear in HHAI 2026, July 8-10, 2026)
Artificial intelligence (AI) is being increasingly integrated into human problem-solving, yet its effects on individual skill development remain unclear. We examine how both AI usage and informativeness can shape learning in the context of a controlled logical reasoning task with on-demand access to AI assistance. We find that greater AI usage is associated with weaker skill development: heavy AI users underperform relative to comparable peers, whereas light AI users perform similarly to
%or slightly better than
matched users who do not use AI. We also find in our study that these patterns are mediated by AI informativeness. Low-information AI neither improves immediate performance nor preserves performance after AI assistance is removed, and is linked to weaker learning overall. On the other hand, high-information AI was found to improve short-run performance without reducing post-AI outcomes on average in our experiments, but with heterogeneous effects.