IBM infosphere streams for scalable, real-time, intelligent transportation services

Biem, Alain; Bouillet, Eric; Feng, Huijie; Ranganathan, Anand; Riabov, Anton; Verscheure, Olivier; Koutsopoulos, Haris N.; Moran, Carlos

doi:10.1145/1807167.1807291

Cited by 166 publications

(123 citation statements)

References 6 publications

Supporting

Mentioning

122

Contrasting

Order By: Relevance

“…The software platforms for smart cities should offer high performance computing capabilities, be optimized for the hardware being used, is stable and reliable for the different data-intensive applications being executed, supports stream processing, provides a high-levels of fault resilience, and is supported by a well-trained and capable team and vendor. There are different available software platforms for big data analytics such as Hadoop Mapreduce [28], HPCC [29], Stratosphere [30], and IBM Infosphere Streams [31], which provide the stream processing required by real-time big data applications such as intelligent transportations in a smart city [19]. These platforms work well on cluster systems that can provide a powerful and scalable hardware platform to meet the requirements of big data applications for smart cities.…”

Section: Big Data Managementmentioning

confidence: 99%

Applications of big data to smart cities

Nuaimi¹,

Neyadi²,

Mohamed³

et al. 2015

J Internet Serv Appl

658

399

View full text Add to dashboard Cite

Many governments are considering adopting the smart city concept in their cities and implementing big data applications that support smart city components to reach the required level of sustainability and improve the living standards. Smart cities utilize multiple technologies to improve the performance of health, transportation, energy, education, and water services leading to higher levels of comfort of their citizens. This involves reducing costs and resource consumption in addition to more effectively and actively engaging with their citizens. One of the recent technologies that has a huge potential to enhance smart city services is big data analytics. As digitization has become an integral part of everyday life, data collection has resulted in the accumulation of huge amounts of data that can be used in various beneficial application domains. Effective analysis and utilization of big data is a key factor for success in many business and service domains, including the smart city domain. This paper reviews the applications of big data to support smart cities. It discusses and compares different definitions of the smart city and big data and explores the opportunities, challenges and benefits of incorporating big data applications for smart cities. In addition it attempts to identify the requirements that support the implementation of big data applications for smart city services. The review reveals that several opportunities are available for utilizing big data in smart cities; however, there are still many issues and challenges to be addressed to achieve better utilization of this technology.

show abstract

Section: Big Data Managementmentioning

confidence: 99%

Applications of big data to smart cities

Nuaimi¹,

Neyadi²,

Mohamed³

et al. 2015

J Internet Serv Appl

658

399

View full text Add to dashboard Cite

show abstract

“…The runtime-agnostic API was open-sourced in 2015 and has evolved into the Apache Beam project (short for Batch and stream, currently incubating) [7] to bundle it with the corresponding execution engines (runners): As of writing, Flink, Spark and the proprietary Google Dataflow cloud service are supported. The only other fully managed stream processing system apart from Google Dataflow that we are aware of is IBM Infosphere Streams [17]. However, in contrast to Google Dataflow which is documented to be highly scalable (quota limit for customers: 1000 compute nodes [9]), it is hard to find evidence for high scalability of IBM Infosphere Streams; performance evaluations made by IBM [21] only indicate it performs well in small deployments with up to a few nodes.…”

Section: Further Systemsmentioning

confidence: 99%

Real-time stream processing for Big Data

Wingerath

Gessert

Friedrich

et al. 2016

It - Information Technology

View full text Add to dashboard Cite

Abstract:With the rise of the web 2.0 and the Internet of things, it has become feasible to track all kinds of information over time, in particular fine-grained user activities and sensor data on their environment and even their biometrics. However, while efficiency remains mandatory for any application trying to cope with huge amounts of data, only part of the potential of today's Big Data repositories can be exploited using traditional batch-oriented approaches as the value of data often decays quickly and high latency becomes unacceptable in some applications. In the last couple of years, several distributed data processing systems have emerged that deviate from the batchoriented approach and tackle data items as they arrive, thus acknowledging the growing importance of timeliness and velocity in Big Data analytics. In this article, we give an overview over the state of the art of stream processors for low-latency Big Data analytics and conduct a qualitative comparison of the most popular contenders, namely Storm and its abstraction layer Trident, Samza and Spark Streaming. We describe their respective underlying rationales, the guarantees they provide and discuss the trade-offs that come with selecting one of them for a particular task.

show abstract

“…For that reason, stream processing frameworks such as Yahoo's S4 [18], or IBM InfoSphere Streams [19] provide streaming programming abstractions to build and deploy tasks as distributed applications at scale for commodity clusters and clouds. Nevertheless, even that these systems support high input data rates, they do not consider variable input rates, which is our focus in this paper.…”

Section: Related Workmentioning

confidence: 99%

Extending CometCloud to Process Dynamic Data Streams on Heterogeneous Infrastructures

Tolosana-Calasanz

Diaz-Montes

Rana

et al. 2014

2014 International Conference on Cloud and Autonomic Computing

View full text Add to dashboard Cite

Abstract-Coordination of multiple concurrent data stream processing, carried out through a distributed Cloud infrastructure, is described. The coordination (control) is carried out through the use of a Reference net (a particular type of Petri net) based interpreter, implemented alongside the CometCloud system. One of the benefits of this approach is that the model can also be executed directly to support the coordination action. The proposed approach supports the simultaneous processing of data streams and enables dynamic scale-up of heterogeneous computational resources on demand, while meeting the particular quality of service requirements (throughput) for each data stream. We assume that the processing to be applied to each data stream is known a priori. The workflow interpreter monitors the arrival rate and throughput of each data stream, as a consequence of carrying out the execution using CometCloud. We demonstrate the use of the control strategy using two key actions -allocating and deallocating resources dynamically based on the number of tasks waiting to be executed (using a predefined threshold). However, a variety of other control actions can also be supported and are described in this work. Evaluation is carried out using a distributed CometCloud deployment -where the allocation of new resources can be based on a number of different criteria, such as: (i) differences between sites, i.e. based on the types of resources supported (e.g. GPU vs. CPU only, FPGAs, etc), (ii) cost of execution; (iii) failure rate and likely resilience, etc.

show abstract

IBM infosphere streams for scalable, real-time, intelligent transportation services

Cited by 166 publications

References 6 publications

Applications of big data to smart cities

Applications of big data to smart cities

Real-time stream processing for Big Data

Extending CometCloud to Process Dynamic Data Streams on Heterogeneous Infrastructures

Contact Info

Product

Resources

About