ERAIA - Enabling Intelligence Data Pipelines for IoT-based Application Systems

Hernández, Aitor; Xiao, Bin; Tudor, Valentin

doi:10.1109/percom45495.2020.9127385

Cited by 6 publications

(2 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moving on in the distributed data processing pipelines and streams, Hernandez et al (2020) models intelligent data pipelines using an actor-based model for IoT-based applications. The focus here is to support application developers in building intelligent data processing actors in a common ecosystem.…”

Section: Related Workmentioning

confidence: 99%

Data-sharing markets for integrating IoT data processing functionalities

Kasrin

Benabbas

Elmamooz

et al. 2021

CCF Trans. Pervasive Comp. Interact.

View full text Add to dashboard Cite

The recent evolution of the Internet of Things into a cyber-physical reality has spawned various challenges from a data management perspective. In addition, IoT platform designers are faced with another set of questions. How can platforms be extended to smoothly integrate new data management functionalities? Currently, data processing related tasks are typically realized by manually developed code and functions which creates difficulties in maintenance and growth. Hence we need to explore other approaches to integration for IoT platforms. In this paper we cover both these aspects: (1) we explore several emerging data management challenges, and (2) we propose an IoT platform integration model that can combine disparate functionalities under one roof. For the first, we focus on the following challenges: sensor data quality, privacy in data streams, machine learning model management, and resource-aware data management. For the second, we propose an information-integration model for IoT platforms. The model revolves around the concept of a Data-Sharing Market where data management functionalities can share and exchange information about their data with other functionalities. In addition, data-sharing markets themselves can be combined into networks of markets where information flows from one market to another, which creates a web of information exchange about data resources. To motivate this work we present a use-case application in smart cities.

show abstract

Section: Related Workmentioning

confidence: 99%

Data-sharing markets for integrating IoT data processing functionalities

Kasrin

Benabbas

Elmamooz

et al. 2021

CCF Trans. Pervasive Comp. Interact.

View full text Add to dashboard Cite

show abstract

“…To simplify designing and deploying such data processing services, pipelines are commonly used to compose a set of individual data processing processes into a single service, where the output of a component is the input of the next component. This allows to reuse and compose of common data handling processes into more complex data pipeline services [3].…”

Section: Introductionmentioning

confidence: 99%

Serverless data pipeline approaches for IoT data in fog and cloud computing

Poojara¹,

Dehury²,

Jakovits³

et al. 2021

Preprint

View full text Add to dashboard Cite

With the increasing number of Internet of Things (IoT) devices, massive amounts of raw data is being generated. The latency, cost, and other challenges in cloud-based IoT data processing have driven the adoption of Edge and Fog computing models, where some data processing tasks are moved closer to data sources. Properly dealing with the flow of such data requires building data pipelines, to control the complete life cycle of data streams from data acquisition at the data source, edge and fog processing, to Cloud side storage and analytics. Data analytics tasks need to be executed dynamically at different distances from the data sources and often on very heterogeneous hardware devices. This can be streamlined by the use of a Serverless (or FaaS) cloud computing model, where tasks are defined as virtual functions, which can be migrated from edge to cloud (and vice versa) and executed in an eventdriven manner on data streams. In this work, we investigate the benefits of building Serverless data pipelines (SDP) for IoT data analytics and evaluate three different approaches for designing SDPs: 1) Off-the-shelf data flow tool (DFT) based, 2) Object storage service (OSS) based and 3) MQTT based. Further, we applied these strategies on three fog applications (Aeneas, PocketSphinx, and custom Video processing application) and evaluated the performance by comparing their processing time (computation time, network communication and disk access time), and resource utilization. Results show that DFT is unsuitable for compute-intensive applications such as video or image processing, whereas OSS is best suitable for this task. However, DFT is nicely fit for bandwidthintensive applications due to the minimum use of network resources. On the other hand, MQTT-based SDP is observed with increase in CPU and Memory usage as the number of users rose, and experienced a drop in data units in the pipeline for PocketSphinx and custom video processing applications, however it performed well for Aeneas which had low size data units.

show abstract