Scalable and Low-Latency Data Processing with Stream MapReduce

Brito, Andrey; Martin, André; Knauth, Thomas; Creutz, Stephan; Becker, Diogo; Weigert, Stefan; Fetzer, Christof

doi:10.1109/cloudcom.2011.17

Cited by 43 publications

(34 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to improve the processing latency, the mappers are continuously fed with batches of tuples (instead of input files), and they push their results to reducers as soon as they are available. This approach is similar to the one adopted by the StreamMapReduce [43] project, which uses these ideas to implement a fully-fledged event stream processing (ESP) implementation.…”

Section: Online Processingmentioning

confidence: 99%

Challenges for MapReduce in Big Data

Grolinger

Hayes

Higashino

et al. 2014

2014 IEEE World Congress on Services

128

View full text Add to dashboard Cite

Abstract-In the Big Data community, MapReduce has been seen as one of the key enabling approaches for meeting continuously increasing demands on computing resources imposed by massive data sets. The reason for this is the high scalability of the MapReduce paradigm which allows for massively parallel and distributed execution over a large number of computing nodes. This paper identifies MapReduce issues and challenges in handling Big Data with the objective of providing an overview of the field, facilitating better planning and management of Big Data projects, and identifying opportunities for future research in this field. The identified challenges are grouped into four main categories corresponding to Big Data tasks types: data storage (relational databases and NoSQL stores), Big Data analytics (machine learning and interactive analytics), online processing, and security and privacy. Moreover, current efforts aimed at improving and extending MapReduce to address identified challenges are presented. Consequently, by identifying issues and challenges MapReduce faces when handling Big Data, this study encourages future Big Data research.

show abstract

Section: Online Processingmentioning

confidence: 99%

Challenges for MapReduce in Big Data

Grolinger

Hayes

Higashino

et al. 2014

2014 IEEE World Congress on Services

128

View full text Add to dashboard Cite

show abstract

“…The prevalence and success of MapReduce has motivated many researchers to work on systems that leverage some of its advantages while at the same time trying to overcome its limitations when applied to low-latency processing. StreamMapReduce [19], M3 [20], and Twitter's Storm [16] are examples of MapReduce-inspired systems aimed at stream processing.…”

Section: Cep In the Cloudmentioning

confidence: 99%

CEPSim: A Simulator for Cloud-Based Complex Event Processing

Higashino

Capretz

Bittencourt

2015

2015 IEEE International Congress on Big Data

View full text Add to dashboard Cite

Abstract-As one of the Vs defining Big Data, data velocity brings many new challenges to traditional data processing approaches. The adoption of cloud environments in complex event processing (CEP) systems is a recent architectural style that aims to overcome these challenges. Validating cloud-based CEP systems at the required Big Data scale, however, is often a laborious, error-prone, and expensive task. This article presents CEPSim, a new simulator that has been developed to facilitate this validation process. CEPSim extends CloudSim, an existing cloud simulator, with an application model based on directed acyclic graphs that is used to represent continuous CEP queries. Once defined, the queries can be simulated in different cloud environments under diverse load conditions. Moreover, CEPSim is also customizable with different operator placement and scheduling strategies. These features enable researchers and system architects to experiment with different configurations and strategies and to promote research in this field. Experimental results show that CEPSim can successfully simulate existing cloud-based CEP systems.

show abstract

“…However, it has been shown that traditional DBMSs that use a 'storethen-process' model of computation cannot provide the low latency responses needed for real-time stream processing [3]. Moreover, distributed processing frameworks like MapReduce are not well suited to working with this form of underlying data, due to their batch-orientated nature [7] -leading to a lack of responsiveness [6]. Instead, new distributed stream processing platforms have been proposed, e.g.…”

Section: B Big Data and Distributed Stream Processingmentioning

confidence: 99%

“…Indeed, the social Twitter stream generates more than 400 million tweets each day. Other examples of real-time stream processing tasks are stock market trading (approximately 10 billion messages per day in trades) and fraud detection in mobile telephony [6].…”

Section: Introductionmentioning

confidence: 99%

Scalable distributed event detection for Twitter

McCreadie

Macdonald

Ounis

et al. 2013

2013 IEEE International Conference on Big Data

View full text Add to dashboard Cite

Abstract-Social media streams, such as Twitter, have shown themselves to be useful sources of real-time information about what is happening in the world. Automatic detection and tracking of events identified in these streams have a variety of real-world applications, e.g. identifying and automatically reporting road accidents for emergency services. However, to be useful, events need to be identified within the stream with a very low latency. This is challenging due to the high volume of posts within these social streams. In this paper, we propose a novel event detection approach that can both effectively detect events within social streams like Twitter and can scale to thousands of posts every second. Through experimentation on a large Twitter dataset, we show that our approach can process the equivalent to the full Twitter Firehose stream, while maintaining event detection accuracy and outperforming an alternative distributed event detection system.

show abstract

Scalable and Low-Latency Data Processing with Stream MapReduce

Cited by 43 publications

References 15 publications

Challenges for MapReduce in Big Data

Challenges for MapReduce in Big Data

CEPSim: A Simulator for Cloud-Based Complex Event Processing

Scalable distributed event detection for Twitter

Contact Info

Product

Resources

About