Heuristic-based Parsing System for Big Data Log

Teng Li, Shengkai Zhang, Yebo Feng, Jiahua Xu, Zhuo Ma, Yulong Shen, Jianfeng Ma

Abstract: Logs play a crucial role in recording valuable system runtime information, extensively utilized by service providers and users for effective service management. A typical approach in service management, based on log analysis, involves parsing the original log messages initially presented in an unstructured format. Subsequently, a data mining model is employed to extract critical system behavior information, aiding in service management. As the volume of logs rapidly increases, training models using current log resolution methods post-log collection becomes excessively time-consuming, leading to decreased accuracy. Manual analysis of extensive logs is both time-intensive and inefficient. This article introduces Aclog, an automated log parsing tool tailored for large-scale log analysis, storage, and management. Aclog operates by storing and managing logs in a structured and unified format, thereby offering a cohesive database for comprehensive log auditing of computing systems. Key components of Aclog encompass the log updater, log parser, log storage, and log querier. In this paper, we utilize a realworld, large-scale public log dataset to showcase the capabilities of Aclog. We evaluate the log files generated by ten popular systems.