Dependable IoT Data Stream Processing for Monitoring and Control of Urban Infrastructures

Geldenhuys, Morgan K.; Will, Jonathan; Pfister, Benjamin J. J.; Haug, Martin; Scharmann, Alexander; Thamsen, Lauritz

doi:10.1109/ic2e52221.2021.00041

Cited by 16 publications

(9 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An approach orchestrated over Kubernetes [33] was presented in [10], with a reference architecture for processing IoT device data in urban infrastructure monitoring scenarios. While not specific to edge processing use cases, the authors proposed it may be applied at any scale, whether at the Cloud Layer or even the fog layer.…”

Section: Related Workmentioning

confidence: 99%

“…According to [8], one way to tackle this research challenge is to define a layer of edge computing [9], where a subset of data may be better handled by Stream Processing. In spite of some proposals in the literature regarding architectures for edge processing, e.g., [10], many do not take connectivity restrictions into account, nor do they consider the space and computation constraints imposed by edge solutions in many domains. Furthermore, there is a lack of literature on real-time processing of IoT stream data with high demands on throughput and latency.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

IRONEDGE: Stream Processing Architecture for Edge Applications

et al. 2023

View full text Add to dashboard Cite

This paper presents IRONEDGE, an architectural framework that can be used in different edge Stream Processing solutions for “Smart Infrastructure” scenarios, on a case-by-case basis. The architectural framework identifies the common components that any such solution should implement and a generic processing pipeline. In particular, the framework is considered in the context of a study case regarding Internet of Things (IoT) devices to be attached to rolling stock in a railway. A lack of computation and storage resources available in edge devices and infrequent network connectivity are not often seen in the existing literature, but were considered in this paper. Two distinct implementations of IRONEDGE were considered and tested. One, identified as Apache Kafka with Kafka Connect (K0-WC), uses Kafka Connect to pass messages from MQ Telemetry Transport (MQTT) to Apache Kafka. The second scenario, identified as Apache Kafka with No Kafka Connect (K1-NC), allows Apache Storm to consume messages directly. When the data rate increased, K0-WC showed low throughput resulting from high losses, whereas K1-NC displayed an increase in throughput, but did not match the input rate for the Data Reports. The results showed that the framework can be used for defining new solutions for edge Stream Processing scenarios and identified a reference implementation for the considered study case. In future work, the authors propose to extend the evaluation of the architectural variation of K1-NC.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

IRONEDGE: Stream Processing Architecture for Edge Applications

et al. 2023

View full text Add to dashboard Cite

show abstract

“…2) Datasets: For the evaluation of our methods, we used an existing dataset 3 which was introduced by Hsu et al in Arrow [13]. The dataset contains 1031 unique Spark and Hadoop executions, which were facilitated by the benchmarking tool HiBench by Intel, and which ran on 69 different AWS cluster configurations.…”

Section: A Experimental Setupmentioning

confidence: 99%

“…Distributed dataflow systems like Apache Flink [1] and Apache Spark [2] enable users from different domains such as public infrastructure monitoring, earth observation, or bioinformatics to develop scalable data-parallel programs [3]- [5]. They reduce the need to implement parallelism and fault tolerance and, therefore, simplifying organizations' big data analysis processes.…”

Section: Introductionmentioning

confidence: 99%

Ruya: Memory-Aware Iterative Optimization of Cluster Configurations for Big Data Processing

Will¹,

Thamsen²,

Bader³

et al. 2022

2022 IEEE International Conference on Big Data (Big Data)

Self Cite

View full text Add to dashboard Cite

“…In some domains, for example IoT sensor networks, new data gets generated rapidly and in large amounts [27]. However, some machine learning techniques struggle with increasing data sizes.…”

Section: Training Data Reductionmentioning

confidence: 99%

Training Data Reduction for Performance Models of Data Analytics Jobs in the Cloud

Will

Arslan

Bader

et al. 2021

2021 IEEE International Conference on Big Data (Big Data)

Self Cite

View full text Add to dashboard Cite

Distributed dataflow systems like Apache Flink and Apache Spark simplify processing large amounts of data on clusters in a data-parallel manner. However, choosing suitable cluster resources for distributed dataflow jobs in both type and number is difficult, especially for users who do not have access to previous performance metrics. One approach to overcoming this issue is to have users share runtime metrics to train context-aware performance models that help find a suitable configuration for the job at hand. A problem when sharing runtime data instead of trained models or model parameters is that the data size can grow substantially over time.This paper examines several clustering techniques to minimize training data size while keeping the associated performance models accurate. Our results indicate that efficiency gains in data transfer, storage, and model training can be achieved through training data reduction. In the evaluation of our solution on a dataset of runtime data from 930 unique distributed dataflow jobs, we observed that, on average, a 75% data reduction only increases prediction errors by one percentage point.

show abstract

Dependable IoT Data Stream Processing for Monitoring and Control of Urban Infrastructures

Cited by 16 publications

References 24 publications

IRONEDGE: Stream Processing Architecture for Edge Applications

IRONEDGE: Stream Processing Architecture for Edge Applications

Ruya: Memory-Aware Iterative Optimization of Cluster Configurations for Big Data Processing

Training Data Reduction for Performance Models of Data Analytics Jobs in the Cloud

Contact Info

Product

Resources

About