MapReduce based log file analysis for system threats and problem identification

Abstract

Log files are primary source of information for identifying the System threats and problems that occur in the System at any point of time. These threats and problem in the system can be identified by analyzing the log file and finding the patterns for possible suspicious behavior. The concern administrator can then be provided with appropriate alter or warning regarding these security threats and problems in the system, which are generated after the log files are analyzed. Based upon this alters or warnings the administrator can take appropriate actions. Many tools or approaches are available for this purpose, some are proprietary and some are open source. This paper presents a new approach which uses a MapReduce algorithm for the purpose of log file analysis, providing appropriate security alerts or warning. The results of this system can then be compared with the tools available.

Log Analysis Using Temporal Logic and Reconstruction Approach: Web Server Case

“…In the first group, Vernekar and Buchade (2013) propose a system and claim to provide significant improvement in response time through large log file analysis, correlation of events and generating alerts by implementing MapReduce algorithm.…”

An Attack-based Filtering Scheme for Slow Rate Denial-of-Service Attack Detection in Cloud Environment

“…Specifically, about Slow Rate DoS attacks, the proposed systems in [12], [13], and [14] aimed to detect those attacks but they were demonstrated to be not efficient in all scenarios but only under specific attack conditions. Talking about Apache Hadoop, the solutions in [16], [17], [18], [19], [20], [21], [22], and [23] take advantage of the ability of this tool to manage big datasets, which is generally a main requirement in case of DoS attacks and in an environment like the cloud. As an additional advantage of Apache Hadoop shown in [20], [21], and [23], it includes libraries with already implemented useful tools such as machine learning algorithms.…”

Τεχνολογίες Εικονικοποίησης Και Κατανεμημένη Επεξεργασία Σε Υπολογιστικά Νέφη

“…Προκειμένου να ανταποκρίνεται στα ιδιαίτερα χαρακτηριστικά των γράφων, επεκτείνει τα RDDs ώστε να μπορούν να αναπαρασταθούν κατευθυνόμενοι γράφοι με τιμές και ιδιότητες σε κάθε ακμή και κόμβο. [51], [52], [53], [54], [55], [56], [57], [58], [59], [60] και [61].…”