IoTDeM: An IoT Big Data-oriented MapReduce performance prediction extended model in multiple edge clouds

ZhiHui Lv, Nini Wang, Jie Wu, Meikang Qiu

Abstract: Highlights •An IoT big data edge cloud architecture to address the challenges of data analytics in IoT using “edge cloud” that pushes various computing and data analysis capabilities to multiple edge clouds.•We propose IoTDeM, an extended IoT big data-oriented model for predicting MapReduce performances through extending the LWLR model in multiple Edge Clouds Hadoop 2 environments. By extracting distinguishing features for job representations, IoTDeM selects a cluster scale as a crucial parameter to improve the LWLR prediction model used in Hadoop 1 to adapt to Hadoop 2.•We propose our IoT Big data Edge Cloud Architecture based on Ceph, even though many previous researches on MapReduce performance prediction are mostly based on HDFS. Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability.•We have validated the accuracy of the proposed MapReduce performance model using TestDFSIO and Sort benchmark applications in the IoT Big data Edge Cloud Architecture environment based on Hadoop 2 with Ceph as a storage system.