Allan Variance-based Granulation Technique for Large Temporal Databases

Lorina Sinanaj, Hossein Haeri, Liming Gao, Srivenkata Satya Prasad Maddipatla, Cindy Chen, Kshitij Jerath, Craig Earl Beal, Sean Brennan

Abstract: In the era of Big Data, conducting complex data analysis tasks efficiently, becomes increasingly important and challenging due to large amounts of data available. In order to decrease query response time with limited main memory and storage space, data reduction techniques that preserve data quality are needed. Existing data reduction techniques, however, are often computationally expensive and rely on heuristics for deciding how to split or reduce the original dataset. In this paper, we propose an effective granular data reduction technique for temporal databases, based on Allan Variance (AVAR). AVAR is used to systematically determine the temporal window length over which data remains relevant. The entire dataset to be reduced is then separated into granules with size equal to the AVAR-determined window length. Data reduction is achieved by generating aggregated information for each such granule. The proposed method is tested using a large database that contains temporal informatio