Evaluating Pervasive and Ubiquitous Systems

Kay Connelly, Katie A. Siek, Ingrid Mulder, Steve Neely, Graeme Stevenson, Christian Kray

Abstract: Recognized evaluation strategies are essential to systematically advance a research field's state of the art. Pervasive and ubiquitous computing need such strategies to mature as a discipline and to enable researchers to objectively assess and compare new techniques' contributions. Researchers have shown that evaluating ubiquitous systems can be difficult, so approaches tend to be subjective, piecemeal, or both. To ensure that the validity and usability of proposed systems won't be compromised, researchers must reach consensus on a set of standard evaluation methods for ubiquitous systems. Otherwise, methods for scientifically testing and presenting state-of-the-art advances will remain unclear. In this article, the organizers from four different workshop series, each focused on this topic, summarize and discuss the main outcomes of these events.