A data streaming performance evaluation using resource constrained edge device

Lee, Hwejoo; Oh, Junghyun; Kim, Kyung-Rae; Yeon, Hun-je

doi:10.1109/ictc.2017.8191055

Cited by 12 publications

(11 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other possibilities would be (i) to run pAElla on the DiG IoT devices as we do, and then use these distributed systems only for training and deployment of the ML models; or (ii) execute these stream services directly on the IoT devices. However, as shown in [67], which benchmarks them in a Raspberry Pi 3 (ARMv8 Quad-Core 1 GB RAM vs. ARM A8 Single-Core 1 GB RAM in our BBB), the resources needed (memory and CPU load) for running these services are not negligible, even considering only data ingestion and delivery and not considering actual computation.…”

Section: Computational Load and Scalabilitymentioning

confidence: 94%

pAElla: Edge AI-Based Real-Time Malware Detection in Data Centers

Libri

Bartolini

Benini

2020

IEEE Internet Things J.

View full text Add to dashboard Cite

The increasing use of Internet-of-Things (IoT) devices for monitoring a wide spectrum of applications, along with the challenges of "big data" streaming support they often require for data analysis, is nowadays pushing for an increased attention to the emerging edge computing paradigm. In particular, smart approaches to manage and analyze data directly on the network edge, are more and more investigated, and Artificial Intelligence (AI) powered edge computing is envisaged to be a promising direction. In this paper, we focus on Data Centers (DCs) and Supercomputers (SCs), where a new generation of high-resolution monitoring systems is being deployed, opening new opportunities for analysis like anomaly detection and security, but introducing new challenges for handling the vast amount of data it produces. In detail, we report on a novel lightweight and scalable approach to increase the security of DCs / SCs, that involves AI-powered edge computing on high-resolution power consumption. The method -called pAElla -targets real-time Malware Detection (MD), it runs on an out-of-band IoT-based monitoring system for DCs / SCs, and involves Power Spectral Density of power measurements, along with AutoEncoders. Results are promising, with an F1-score close to 1, and a False Alarm and Malware Miss rate close to 0%. We compare our method with State-of-the-Art MD techniques and show that, in the context of DCs / SCs, pAElla can cover a wider range of malware, significantly outperforming SoA approaches in terms of accuracy. Moreover, we propose a methodology for online training suitable for DCs / SCs in production, and release open dataset and code.

show abstract

Section: Computational Load and Scalabilitymentioning

confidence: 94%

pAElla: Edge AI-Based Real-Time Malware Detection in Data Centers

Libri

Bartolini

Benini

2020

IEEE Internet Things J.

View full text Add to dashboard Cite

show abstract

“…We used dockers buildx feature to create custom Apache Flink docker image for ARMv7 processor and hosted the image on DockerHub public repository 6 .…”

Section: Methodsmentioning

confidence: 99%

“…Processing: Stream Processing Engines (SPEs) represent a good paradigm for Fog computing applications because of their extensive set of features, including support for event-driven, data pipeline, and data analytics applications. Constrained resources is also here an aspect to cater for; the literature includes surveys on how various SPEs perform in a fog environments [6], [7]. We decided to use Apache Flink 2 as our SPE of choice.…”

Section: B Platform Architecturementioning

confidence: 99%

“…The processing component implements the following functionality: It filters the incoming traffic by their recorded region and sends aggregated results every 10 seconds to the cloud. On the 6 https://hub.docker.com/repository/docker/digitaljazz/flink-1.8.0-armv7 7 https://flink-fog-cluster.readthedocs.io/en/latest cloud tier, we deploy the Telegraf, InfuxDB and Grafana (TIG) stack to receive and visualize aggregated results. The dashboard is shown in Figure 5.…”

Section: Demonstrationmentioning

confidence: 99%

See 1 more Smart Citation

FogGuru: a Fog Computing platform based on Apache Flink

Battulga

Miorandi

Tedeschi

2020

2020 23rd Conference on Innovation in Clouds, Internet and Networks and Workshops (ICIN)

View full text Add to dashboard Cite

Fog computing infrastructure aims to offload computing resources from cloud providers by placing edge devices closer to end-users and/or data sources. Systems and methods for developing and deploying an application in Fog nodes are still in their infancy. In this work, we present FogGuru, a fog computing platform meant to facilitate application deployment in Fog environments. FogGuru builds upon the stream processing paradigm for fog applications design; its implementation makes use of the open-source Apache Flink stream processing engine. The demo will showcase a Fog-based traffic analytics application deployed using FogGuru and running on a fog node, composed of a cluster of five Raspberry Pis.

show abstract

“…These naïve methods also ignore throughput, latency and the sustainability feature of platforms. Similarly, [20] evaluates the throughput and latency metrics of platforms using the naïve method, which again ignores the maximum throughput and sustainability feature of platforms (Apache Flink, Apache Storm and Apache Spark) for resource constrained devices. In [21], [22], the performance metrics of Apache Storm were evaluated in various environments.…”

Section: Related Workmentioning

confidence: 99%

Maximum Sustainable Throughput Evaluation Using an Adaptive Method for Stream Processing Platforms

Chu

Hamdull

2020

IEEE Access

View full text Add to dashboard Cite

The volume and type of streaming data is increasing rapidly, thus, real-time processing scenarios for streaming data have continued to increase. The inherent volatility of streaming data causes large changes in the throughput volatility of stream processing platforms, making evaluations of the maximum sustainable throughput (MST) for such platforms a challenge. To address these problems and improve the low efficiency of manual evaluations, a naïve method is typically used to evaluate the MST of platforms by periodically measuring throughput. The throughput volatility is detected using a skip sliding window that is defined as a bounded sequence in which a limited number of throughput values stored to measure volatility in a skipping fashion in each data growth cycle, which is a fixed time interval with respect to the continuous data rate increases. A skipping sliding window that is used to describe the procedure with which the MST is calculated in an approximate way is based on a sliding window, which is used usually in the context of stateful streaming operators. Then, according to the throughput volatility, the method judges whether the MST has been reached using a latency guarantee. However, because this method has low efficiency and a large error rate, we propose an adaptive MST evaluation method that adds a data-growth factor function to the naïve method cycle that dynamically and adaptively tunes the data rate for each data growth cycle. The experimental results from four open-source benchmarks running on three mainstream stream processing platforms show that the adaptive MST evaluation method has a lower error rate and executes faster compared with the naïve evaluation method. Moreover, the proposed method is insensitive to parameter variations and is suitable for mainstream stream processing platforms.

show abstract

A data streaming performance evaluation using resource constrained edge device

Cited by 12 publications

References 6 publications

pAElla: Edge AI-Based Real-Time Malware Detection in Data Centers

pAElla: Edge AI-Based Real-Time Malware Detection in Data Centers

FogGuru: a Fog Computing platform based on Apache Flink

Maximum Sustainable Throughput Evaluation Using an Adaptive Method for Stream Processing Platforms

Contact Info

Product

Resources

About