Real-Time Stream Processing in Social Networks with RAM3S

Bartolini, Ilaria; Patella, Marco

doi:10.3390/fi11120249

Cited by 8 publications

(19 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Real-time processing on geolocated data from social media apps using hadoop has been performed in a case study by [77] and implement k-NN model to investigate the power of machine learning algorithms on un-structured big data. Possibility of realtime analysis of huge multimedia stream from online social networks is highlighted in studies [61], [79]. To overcome the difficulty of details consideration of distributed computing and low latency, a framework has been introduced in this study that hides platform details and provide simple interface to programmer.…”

Section: B Assessment Of Rq2: Which Challenges Have Been Faced Durinmentioning

confidence: 99%

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Mehmood

Anees

2020

IEEE Access

View full text Add to dashboard Cite

Contribution: Recently, real-time data warehousing (DWH) and big data streaming have become ubiquitous due to the fact that a number of business organizations are gearing up to gain competitive advantage. The capability of organizing big data in efficient manner to reach a business decision empowers data warehousing in terms of real-time stream processing. A systematic literature review for real-time stream processing systems is presented in this paper which rigorously look at the recent developments and challenges of real-time stream processing systems and can serve as a guide for the implementation of real-time stream processing framework for all shapes of data streams. Background: Published surveys and reviews either cover papers focusing on stream analysis in applications other than real-time DWH or focusing on extraction, transformation, loading (ETL) challenges for traditional DWH. This systematic review attempts to answer four specific research questions. Research Questions: 1)Which are the relevant publication channels for real-time stream processing research? 2) Which challenges have been faced during implementation of real-time stream processing? 3) Which approaches/tools have been reported to address challenges introduced at ETL stage while processing real-time stream for real-time DWH? 4) What evidence have been reported while addressing different challenges for processing real-time stream? Methodology: A systematic literature was conducted to compile studies related to publication channels targeting real-time stream processing/joins challenges and developments. Following a formal protocol, semi-automatic and manual searches were performed for work from 2011 to 2020 excluding research in traditional data warehousing. Of 679,547 papers selected for data extraction, 74 were retained after quality assessment. Findings: This systematic literature highlights implementation challenges along with developed approaches for real-time DWH and big data stream processing systems and provides their comparisons. This study found that there exists various algorithms for implementing real-time join processing at ETL stage for structured data whereas less work for un-structured data is found in this subject matter. INDEX TERMS Real-time stream processing, big data streaming, structured/un-structured data, ETL, systematic literature review Publication source Channel References No. %age IEEE Transactions on Knowledge and Data Engineering

show abstract

Section: B Assessment Of Rq2: Which Challenges Have Been Faced Durinmentioning

confidence: 99%

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Mehmood

Anees

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Most of these analytics solutions are focused on using batch processing techniques and technologies like the Hadoop, for the batch processing of large data sets across distributed clusters using the MapReduce programming model [44]. However, considering the advantages of big data analytics and technologies, the application rate in the smart environment and environmental monitoring domain is slow due to underlying fundamental challenges of data and systems heterogeneity [45,46]. While a comprehensive review of existing application scenarios is beyond the scope of this paper, some notable examples are discussed and investigated.…”

Section: Related Research On the Application Of Big Data Analyticsmentioning

confidence: 99%

A Distributed Stream Processing Middleware Framework for Real-Time Analysis of Heterogeneous Data on Big Data Platform: Case of Environmental Monitoring

Akanbi

Masinde

2020

Sensors

View full text Add to dashboard Cite

In recent years, the application and wide adoption of Internet of Things (IoT)-based technologies have increased the proliferation of monitoring systems, which has consequently exponentially increased the amounts of heterogeneous data generated. Processing and analysing the massive amount of data produced is cumbersome and gradually moving from classical ‘batch’ processing—extract, transform, load (ETL) technique to real-time processing. For instance, in environmental monitoring and management domain, time-series data and historical dataset are crucial for prediction models. However, the environmental monitoring domain still utilises legacy systems, which complicates the real-time analysis of the essential data, integration with big data platforms and reliance on batch processing. Herein, as a solution, a distributed stream processing middleware framework for real-time analysis of heterogeneous environmental monitoring and management data is presented and tested on a cluster using open source technologies in a big data environment. The system ingests datasets from legacy systems and sensor data from heterogeneous automated weather systems irrespective of the data types to Apache Kafka topics using Kafka Connect APIs for processing by the Kafka streaming processing engine. The stream processing engine executes the predictive numerical models and algorithms represented in event processing (EP) languages for real-time analysis of the data streams. To prove the feasibility of the proposed framework, we implemented the system using a case study scenario of drought prediction and forecasting based on the Effective Drought Index (EDI) model. Firstly, we transform the predictive model into a form that could be executed by the streaming engine for real-time computing. Secondly, the model is applied to the ingested data streams and datasets to predict drought through persistent querying of the infinite streams to detect anomalies. As a conclusion of this study, a performance evaluation of the distributed stream processing middleware infrastructure is calculated to determine the real-time effectiveness of the framework.

show abstract

“…Each bolt performs a transformation of limited complexity, in this way, several bolt nodes are grouped in a coordinated manner to perform the entire computation. A Storm application can be defined through a topology of spout and bolt nodes forming a directed acyclic graph (DAG), with arcs representing streams of tuples flowing from one node to another [15].…”

Section: B Stormmentioning

confidence: 99%

A New Big Data Architecture for Real-Time Student Attention Detection and Analysis

Hachad¹,

Sadiq²,

Ghanimi³

2020

IJACSA

View full text Add to dashboard Cite

Big Data technologies and their analytical methods can help improve the quality of education. They can be used to process and analyze classroom video streams to predict student attention, this would greatly improve the learning-teaching experience. With the increasing number of students and the expansion of educational institutions, processing and analyzing video streams in real-time become a complicated issue. In this paper, we have reviewed the existing systems of student attention detection, open-source real-time data stream processing technologies, and the two major data stream processing architectures. We also proposed a new Big Data architecture for real-time student attention detection.

show abstract

Real-Time Stream Processing in Social Networks with RAM3S

Cited by 8 publications

References 17 publications

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

Challenges and Solutions for Processing Real-Time Big Data Stream: A Systematic Literature Review

A Distributed Stream Processing Middleware Framework for Real-Time Analysis of Heterogeneous Data on Big Data Platform: Case of Environmental Monitoring

A New Big Data Architecture for Real-Time Student Attention Detection and Analysis

Contact Info

Product

Resources

About