The Unexpected Effects of Google Smart Compose on Open-Ended Writing Tasks
Abstract: In this work we seek to understand the effects of Google Smart Compose’s auto-complete suggestions on writing products and processes in open-ended writing tasks. We recruited 119 human subjects to a carefully designed user study to write timed responses to a common prompt via word processors while recording interactions with Google Smart Compose behind the scene. Our experiment demonstrates the impacts of text suggestions on writing length, structure, cohesion, and complexity, utilizing Coh-metrix values. We also introduce new metrics for measuring and understanding detailed human subject interactions with text suggestions. We find that writers utilizing Google Smart Compose write the same amount of text as those without Smart Compose, and that the structure of written work produced with Google Smart Compose enabled is not significantly distinguishable from writing produced with Google Smart Compose disabled. In addition, there was no strong evidence that writing process was significantly impacted by Google Smart Compose. Finally, this work identifies factors that can be used in future studies measuring language model interactions with human writers.
Loading