The Stratosphere platform for big data analytics

Alexandrov, A.; Bergmann, Rico; Ewen, Stephan; Freytag, Johann-Christoph; Hueske, Fabian; Heise, Arvid; Kao, Odej; Leich, Marcus; Leser, Ulf; Markl, Volker; Naumann, Felix; Peters, Mathias; Rheinländer, Astrid; Sax, Matthias J.; Schelter, Sebastian; Höger, Mareike; Τζούμας, Κώστας; Warneke, Daniel

doi:10.1007/s00778-014-0357-y

Cited by 370 publications

(163 citation statements)

References 35 publications

Supporting

Mentioning

160

Contrasting

Unclassified

Order By: Relevance

“…The software platforms for smart cities should offer high performance computing capabilities, be optimized for the hardware being used, is stable and reliable for the different data-intensive applications being executed, supports stream processing, provides a high-levels of fault resilience, and is supported by a well-trained and capable team and vendor. There are different available software platforms for big data analytics such as Hadoop Mapreduce [28], HPCC [29], Stratosphere [30], and IBM Infosphere Streams [31], which provide the stream processing required by real-time big data applications such as intelligent transportations in a smart city [19]. These platforms work well on cluster systems that can provide a powerful and scalable hardware platform to meet the requirements of big data applications for smart cities.…”

Section: Big Data Managementmentioning

confidence: 99%

Applications of big data to smart cities

Nuaimi¹,

Neyadi²,

Mohamed³

et al. 2015

J Internet Serv Appl

659

407

View full text Add to dashboard Cite

Many governments are considering adopting the smart city concept in their cities and implementing big data applications that support smart city components to reach the required level of sustainability and improve the living standards. Smart cities utilize multiple technologies to improve the performance of health, transportation, energy, education, and water services leading to higher levels of comfort of their citizens. This involves reducing costs and resource consumption in addition to more effectively and actively engaging with their citizens. One of the recent technologies that has a huge potential to enhance smart city services is big data analytics. As digitization has become an integral part of everyday life, data collection has resulted in the accumulation of huge amounts of data that can be used in various beneficial application domains. Effective analysis and utilization of big data is a key factor for success in many business and service domains, including the smart city domain. This paper reviews the applications of big data to support smart cities. It discusses and compares different definitions of the smart city and big data and explores the opportunities, challenges and benefits of incorporating big data applications for smart cities. In addition it attempts to identify the requirements that support the implementation of big data applications for smart city services. The review reveals that several opportunities are available for utilizing big data in smart cities; however, there are still many issues and challenges to be addressed to achieve better utilization of this technology.

show abstract

Section: Big Data Managementmentioning

confidence: 99%

Applications of big data to smart cities

Nuaimi¹,

Neyadi²,

Mohamed³

et al. 2015

J Internet Serv Appl

659

407

View full text Add to dashboard Cite

show abstract

“…4 is an Apache project built on the Stratosphere research prototype [5] [6] to reorder input elements before processing: an operator o sends a watermark with timestamp t to another operator o ′ to guarantee that o ′ will not receive from o any more items with timestamp lower or equal than t. When an operator receives a watermark with timestamp t from all its upstream operators, it can safely start processing all the elements up to time t.…”

Section: Flinkmentioning

confidence: 99%

Defining the execution semantics of stream processing engines

et al. 2017

View full text Add to dashboard Cite

IntroductionSeveral modern data-intensive applications need to process large volumes of data on the fly as they are produced. Examples range from credit card fraud detection systems, which analyze massive streams of credit card transactions to identify suspicious patterns, to environmental monitoring applications that continuously analyze sensor data, to click stream analysis of Web sites that identify frequent patterns of interactions. More in general, stream processing is a central requirement in today's information systems.This state of facts pushed the development of several stream processing engines (SPEs) that continuously analyze streams of data to produce new results as new elements enter the streams. Unfortunately, existing SPEs adopt different processing models and standardized execution semantics have not yet emerged. This severely hampers the usability AbstractThe ability to process large volumes of data on the fly, as soon as they become available, is a fundamental requirement in today's information systems. Modern distributed stream processing engines (SPEs) address this requirement and provide low-latency and high-throughput data stream processing in cluster platforms, offering high-level programming interfaces that abstract from low-level details such as data distribution and hardware failures. The last decade saw a rapid increase in the number of available SPEs. However, each SPE defines its own processing model and standardized execution semantics have not emerged yet. This paper tackles this problem and analyzes the execution semantics of some widely adopted modern SPEs, namely Flink, Storm, Spark Streaming, Google Dataflow, and Azure Stream Analytics. We specifically target the notions of windowing and time, traditionally considered the key distinguishing factors that characterize the behavior of SPEs. We rely on the SECRET model, introduced in 2010 to analyze the windowing semantics for the SPEs available at that time. We show that SECRET models well some aspects of the behavior of modern SPEs, and we shed light on the evolution of SPEs after the introduction of SECRET by analyzing the elements that SECRET cannot fully capture. In this way, the paper contributes to the research in the area of stream processing by: (1) contrasting and comparing some widely used modern SPEs based on a formal model of their execution semantics; (2) discussing the evolution of SPEs since the introduction of the SECRET model; (3) suggesting promising research directions to direct further modeling efforts. Affetti et al. J Big Data (2017) et al. J Big Data (2017) 4:12 and interoperability of SPEs, since a user needs to understand system-specific aspects to confront various alternatives and select the ones that better suite her needs. SURVEY PAPERPage 2 of 24 AffettiThe main factors that differentiate the behaviors of SPEs are the models of windows and time they adopt [1]. Windows enable computations that would be otherwise unfeasible on unbounded datasets such as streams. For instance, counting the number o...

show abstract

“…Flink [2] (formerly known as Stratosphere [15]) is a project that has many parallels to Spark Streaming as it also originated from research and advertises the unification of batch and stream processing in the same system, providing exactlyonce guarantees for the stream programming model and a high-level API comparable to that of Trident. In contrast to Spark Streaming, though, Flink does not rely on batching for stream processing internally and thus delivers low latency in the order of Storm or Samza.…”

Section: Further Systemsmentioning

confidence: 99%

Real-time stream processing for Big Data

Wingerath

Gessert

Friedrich

et al. 2016

It - Information Technology

View full text Add to dashboard Cite

Abstract:With the rise of the web 2.0 and the Internet of things, it has become feasible to track all kinds of information over time, in particular fine-grained user activities and sensor data on their environment and even their biometrics. However, while efficiency remains mandatory for any application trying to cope with huge amounts of data, only part of the potential of today's Big Data repositories can be exploited using traditional batch-oriented approaches as the value of data often decays quickly and high latency becomes unacceptable in some applications. In the last couple of years, several distributed data processing systems have emerged that deviate from the batchoriented approach and tackle data items as they arrive, thus acknowledging the growing importance of timeliness and velocity in Big Data analytics. In this article, we give an overview over the state of the art of stream processors for low-latency Big Data analytics and conduct a qualitative comparison of the most popular contenders, namely Storm and its abstraction layer Trident, Samza and Spark Streaming. We describe their respective underlying rationales, the guarantees they provide and discuss the trade-offs that come with selecting one of them for a particular task.

show abstract

The Stratosphere platform for big data analytics

Cited by 370 publications

References 35 publications

Applications of big data to smart cities

Applications of big data to smart cities

Defining the execution semantics of stream processing engines

Real-time stream processing for Big Data

Contact Info

Product

Resources

About