A scalable machine learning online service for big data real-time analysis

Abstract

Abstract-This work describes a proposal for developing and testing a scalable machine learning architecture able to provide real-time predictions or analytics as a service over domain-independent big data, working on top of the Hadoop ecosystem and providing real-time analytics as a service through a RESTful API. Systems implementing this architecture could provide companies with on-demand tools facilitating the tasks of storing, analyzing, understanding and reacting to their data, either in batch or stream fashion; and could turn into a valuable asset for improving the business performance and be a key market differentiator in this fast pace environment. In order to validate the proposed architecture, two systems are developed, each one providing classical machine-learning services in different domains: the first one involves a recommender system for web advertising, while the second consists in a prediction system which learns from gamers' behavior and tries to predict future events such as purchases or churning. An evaluation is carried out on these systems, and results show how both services are able to provide fast responses even when a number of concurrent requests are made, and in the particular case of the second system, results clearly prove that computed predictions significantly outperform those obtained if random guess was used.

A survey of machine learning for big data processing

“…Representative streaming processing systems include Borealis [98], S4 [99], Kafka [100], and many other recent architectures proposed to provide real-time analytics over big data [101,102]. A scalable machine learning online service with the power of streaming processing for big data real-time analysis is introduced in [103]. In addition, the professor G. B. Giannakis have paid more attention to the real-time processing of streaming data by using machine learning techniques in recent studies; more details can be referred to in [87,104].…”

Real-Time Prediction of Gamers Behavior Using Variable Order Markov and Big Data Technology: A Case of Study

“…Finally, while it is outside the scope of this evaluation, the time required by the system to provide a prediction is shown in previous work from Baldominos et al [4]. The experiments were performed in a single-node cluster with 8 Intel Xeon processing cores and 16GB of RAM virtualized over VMWare ESXi 5.0, and Hortonworks HDP 2.1 as the Hadoop distribution, which includes Hadoop 2.4 and HBase 0.98; and JBoss AS 7 as application server.…”

“…In particular, the Hadoop ecosystem is used following the architecture for big data real-time analysis described in Baldominos et al [4]. The general framework supporting online scalable machine learning over Big Data is shown in Fig.…”

DataCare: Big Data Analytics Solution for Intelligent Healthcare Management

“…Because of the Big Data volume and fast speed, we have used a Big Data architecture based on the one proposed in Baldominos et al [1], but updating the tools to use Apache Spark for the sake of efficiency. Also, a pilot has been conducted to evaluate the performance of the proposed system.…”