LogInsights: Understanding and Extracting Information from Logs for Fault Classification at run-time

Anonymous

Abstract: Software monitoring is the most critical part in any software management life cycle. One of the ways to detect the health of the program and the software is to monitor the logs efficiently. In this paper, we describe a method to process a stream of logs for identifying any fault being mentioned in the log at runtime. At first, we extract meaningful features for detecting the erroneous ones from the stream of logs. Next, we categorize the erroneous logs into the pre-defined categories of commonly occurring faults, using the proposed two-step framework. We propose efficient, fast and intelligent rule-based systems with the domain knowledge being incorporated using the word embedding model. We have built a domain specific corpus and trained a word embedding model for this purpose. The methods described here have shown improved results in the existing product pipeline. Experiments on logs obtained from various applications also show the efficacy of our proposed method.