Efficiency of Stream Processing Engines for Processing BIGDATA Streams

Srikanth, B.; Reddy, V. Krishna

doi:10.17485/ijst/2016/v9i14/84797

Cited by 5 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An effective measure was studied by Talasila et al [24] for tackling the load balancing phenomenon for efficient traffic handling in the public cloud. Another method based on Ant colony swarm optimization based on performance analysis of load balancing techniques in cloud centers was studied by Reddy et al [25], to prevent the latency in real-time stream processing engines like Apache Spark streaming, with an additional technique like dolly retreat mechanism to avoid stragglers and process data efficiently, as studied in Srikanth and Reddy [26]. Radha and Rao [27] offered a comprehensive review of techniques to increase MapReduce performance in heterogeneous cloud environments by partitioning data locality through intermediate data at the reducer side.…”

Section: Related Workmentioning

confidence: 99%

Collaborative Learning Based Straggler Prevention in Large-Scale Distributed Computing Framework

Deshmukh

Rao

Shabaz

2021

Security and Communication Networks

View full text Add to dashboard Cite

Modern big data applications tend to prefer a cluster computing approach as they are linked to the distributed computing framework that serves users jobs as per demand. It performs rapid processing of tasks by subdividing them into tasks that execute in parallel. Because of the complex environment, hardware and software issues, tasks might run slowly leading to delayed job completion, and such phenomena are also known as stragglers. The performance improvement of distributed computing framework is a bottleneck by straggling nodes due to various factors like shared resources, heavy system load, or hardware issues leading to the prolonged job execution time. Many state-of-the-art approaches use independent models per node and workload. With increased nodes and workloads, the number of models would increase, and even with large numbers of nodes. Not every node would be able to capture the stragglers as there might not be sufficient training data available of straggler patterns, yielding suboptimal straggler prediction. To alleviate such problems, we propose a novel collaborative learning-based approach for straggler prediction, the alternate direction method of multipliers (ADMM), which is resource-efficient and learns how to efficiently deal with mitigating stragglers without moving data to a centralized location. The proposed framework shares information among the various models, allowing us to use larger training data and bring training time down by avoiding data transfer. We rigorously evaluate the proposed method on various datasets with high accuracy results.

show abstract

Section: Related Workmentioning

confidence: 99%

Collaborative Learning Based Straggler Prevention in Large-Scale Distributed Computing Framework

Deshmukh

Rao

Shabaz

2021

Security and Communication Networks

View full text Add to dashboard Cite

show abstract

“…Those platforms are capable of processing millions of inputs per second using a single computer [11]. The data stream in Apache Storm is defined using a topology structure to manage what must be processed at each instant.…”

Section: Related Workmentioning

confidence: 99%

Redblock: a tool for online deduplication on large datasets

Pimentel¹,

Vicente²,

Bianco³

2017

RBCA

View full text Add to dashboard Cite

Online data deduplication aims to identify records that represent the same purpose on a continuous data flow environment. It must be able to process a range of information with high effectiveness and no delays. The purpose of this paper is to introduce a developed tool entitled Redblock, for real-time data deduplication, using a distributed platform for online processing combined with an Inverted Index. During the experimental evaluation, Redblock managed to provide good preliminary results in terms of efficiency and effectiveness in a database.Keywords: Data Integration, Online Deduplication, Blocking.Resumo: A deduplicação online tem como propósito identificar registros que representam um mesmo objetivo em ambientes com fluxo contínuo de dados. A deduplicação online deve ser capaz de processar volumes variados de informações, sem atrasos e com uma alta eficácia. Este trabalho, propõe uma ferramenta intitulada Redblock para a deduplicação de dados em tempo real. A ferramenta utiliza uma plataforma distribuída de processamento online em conjunto com um mé-todo de blocagem utilizando índice invertido. Na experimentação, Redblock demonstrou resultados preliminares promissores em relação a eficácia e a eficiência em uma base de dados.

show abstract