The Real Time Big Data Processing Framework Advantages and Limitations

Gurusamy, Vairaprakash; Kannan, Subarmaniam; Nandhini, K.

doi:10.26438/ijcse/v5i12.305312

Cited by 19 publications

(25 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, almost all compare processing frameworks without categorizing them according to the data sources that are designed to operate on them (batch, stream or hybrid). The study conducted by Gurusamy et al [46] grouped the most popular big data processing frameworks into three clusters (batch-only, stream-only and hybrid) based on the state of the data they are designed to handle. However, this study does not explicitly present the aspects considered for the comparison of frameworks from the same group.…”

Section: Popular Big Data Processing Frameworkmentioning

confidence: 99%

“…However, this study does not explicitly present the aspects considered for the comparison of frameworks from the same group. Therefore, this paper uses the approach presented by Gurusamy et al [46] to group the processing frameworks. Then, a summarized description of each processing framework is presented.…”

Section: Popular Big Data Processing Frameworkmentioning

confidence: 99%

“…It was proposed by Google and comes with MapReduce as its default engine [48]. Hadoop is composed of several components that work together to process batch data, namely Hadoop Distributed File System (HDFS), Yet Another Resource Negotiator (YARN) and MapReduce [46].…”

Section: Batch Processing Frameworkmentioning

confidence: 99%

“…HDFS is designed to store very large data sets reliably [49] and to ensure that the data remains available despite inevitable host failures [46]. It is also used as a source of data and to store intermediate processing results and make it available for the final computation.…”

Section: Batch Processing Frameworkmentioning

confidence: 99%

“…There is a considerable number of stream processing frameworks such as S4 [51] and more recent systems such as MillWheel [52], and Photon [53]. However, the most popular big data stream processing frameworks are Apache Storm and Apache Samza [46].…”

Section: Stream Processing Frameworkmentioning

confidence: 99%

See 4 more Smart Citations

Review of Big Data and Processing Frameworks for Disaster Response Applications

Cumbane

Gidófalvi

2019

IJGI

View full text Add to dashboard Cite

Natural hazards result in devastating losses in human life, environmental assets and personal, and regional and national economies. The availability of different big data such as satellite imageries, Global Positioning System (GPS) traces, mobile Call Detail Records (CDRs), social media posts, etc., in conjunction with advances in data analytic techniques (e.g., data mining and big data processing, machine learning and deep learning) can facilitate the extraction of geospatial information that is critical for rapid and effective disaster response. However, disaster response systems development usually requires the integration of data from different sources (streaming data sources and data sources at rest) with different characteristics and types, which consequently have different processing needs. Deciding which processing framework to use for a specific big data to perform a given task is usually a challenge for researchers from the disaster management field. Therefore, this paper contributes in four aspects. Firstly, potential big data sources are described and characterized. Secondly, the big data processing frameworks are characterized and grouped based on the sources of data they handle. Then, a short description of each big data processing framework is provided and a comparison of processing frameworks in each group is carried out considering the main aspects such as computing cluster architecture, data flow, data processing model, fault-tolerance, scalability, latency, back-pressure mechanism, programming languages, and support for machine learning libraries, which are related to specific processing needs. Finally, a link between big data and processing frameworks is established, based on the processing provisioning for essential tasks in the response phase of disaster management.

show abstract

Section: Popular Big Data Processing Frameworkmentioning

confidence: 99%

Section: Popular Big Data Processing Frameworkmentioning

confidence: 99%