A framework for Internet data real-time processing: A machine-learning approach

Abstract

Nowadays, the Internet Service Providers have to keep track of and in some cases to analyze for legal issues, a great amount of Internet data. Real-time big data processing and analysis introduce new challenges that must be addressed by system engineers. This is because: 1) traditional technologies exploiting databases are not designed to process a huge amount of data in real-time 2) classic machine learning algorithms implemented by widely adopted tools as Weka or R are not designed to perform "on the fly" analysis on streamed data. In this paper the authors propose an architecture that makes the real-time big data processing and analysis possible. The proposed architecture is based on two main components: a stream processing engine called Apache Storm and a framework called Yahoo SAMOA allowing to perform data analysis through distributed streaming machine learning algorithms. Our architecture is tested for Skype traffic recognition within network traffic generated by several Personal Computers in a streamed way. Experimental results have shown the effectiveness of proposed solution.

Cross-Site Scripting Guardian: A Static XSS Detector Based on Data Stream Input-Output Association Mining

“…Besides, this kind of HTTP traffic-based detection will increase the burden of network transmission, and may not work under the influence of network conditions such as DDoS attacks. In contrast, the source code level protection has no impact on the real-time performance of the network, and will not cause mechanism failure due to bad network conditions [24]. Especially when machine learning technology is combined with real-time processing, problems such as delay, memory, batch processing.…”

A survey of open source tools for machine learning with big data in the Hadoop ecosystem

“…Analysis was carried out through an ensemble of Vertical Hoeffding Trees using Adaptive Bagging. Other projects have leveraged SAMOA as part of frameworks for efficiently finding top-k items [154] and for recognition of internet traffic patterns [155].…”

Real Time Data Processing Using Storm

“…. A framework for Internet data real-time processing: a machine-learning approach[1], Mario Di Mauro, Cesario Di Sarnoy,IEEE 2014.In this paper objective is to perform processing and analysis of real time data using Apache Storm and Samoa framework by using the vertical hoeffding tree algorithm. Advantage is structured data can be processed and analyzed .2.…”