Data integration in scalable data analytics platform for process industries

Sarnovský, Martin; Bednár, Peter; Smatana, Miroslav

doi:10.1109/ines.2017.8118553

Cited by 9 publications

(6 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apache NiFi is an efficient tool to construct dataflow and data integration with an interactive GUI. One paper uses NiFi to collect streaming data in process industry [123]. NiFi is initially developed and used by the National Security Agency of the United States (NSA), which has been verified in the real application environment.…”

Section: Workflow and Dataflowmentioning

confidence: 99%

Manufacturing big data ecosystem: A systematic literature review

Cui

Kara

Chan

2020

Robotics and Computer-Integrated Manufacturing

237

View full text Add to dashboard Cite

Advanced manufacturing is one of the core national strategies in the US (AMP), Germany (Industry 4.0) and China (Made-in China 2025). The emergence of the concept of Cyber Physical System (CPS) and big data imperatively enable manufacturing to become smarter and more competitive among nations. Many researchers have proposed new solutions with big data enabling tools for manufacturing applications in three directions: product, production and business.Big data has been a fast-changing research area with many new opportunities for applications in manufacturing. This paper presents a systematic literature review of the state-of-the-art of big data in manufacturing. Six key drivers of big data applications in manufacturing have been identified. The key drivers are system integration, data, prediction, sustainability, resource sharing and hardware.Based on the requirements of manufacturing, nine essential components of big data ecosystem are captured. They are data ingestion, storage, computing, analytics, visualization, management, workflow, infrastructure and security.Several research domains are identified that are driven by available capabilities of big data ecosystem. Five future directions of big data applications in manufacturing are presented from modelling and simulation to realtime big data analytics and cybersecurity.

show abstract

Section: Workflow and Dataflowmentioning

confidence: 99%

Manufacturing big data ecosystem: A systematic literature review

Cui

Kara

Chan

2020

Robotics and Computer-Integrated Manufacturing

237

View full text Add to dashboard Cite

show abstract

“…In 2012, Elsaleh et al [23] is a multi-layer framework that combines sources data in linked data form, provides access to IoT applications, and uses SPARQL endpoints. In 2017, Sarnovsky et al [24] proposed integration by collecting heterogeneous data from diverse sources. They used Apace Nifi and designed workflows for processing real-time data from different locations.…”

Section: Figure 1 Mapping Ontology Using a Shared Ontologymentioning

confidence: 99%

A unified ontology-based data integration approach for the internet of things

Swar

Khoriba

Belal

2022

IJECE

View full text Add to dashboard Cite

<span lang="EN-US">Data integration enables combining data from various data sources in a standard format. Internet of things (IoT) applications use ontology approaches to provide a machine-understandable conceptualization of a domain. We propose a unified ontology schema approach to solve all IoT integration problems at once. The data unification layer maps data from different formats to data patterns based on the unified ontology model. This paper proposes a middleware consisting of an ontology-based approach that collects data from different devices. IoT middleware requires an additional semantic layer for cloud-based IoT platforms to build a schema for data generated from diverse sources. We tested the proposed model on real data consisting of approximately 160,000 readings from various sources in different formats like CSV, JSON, raw data, and XML. The data were collected through the file transfer protocol (FTP) and generated 960,000 resource description framework (RDF) triples. We evaluated the proposed approach by running different queries on different machines on SPARQL protocol and RDF query language (SPARQL) endpoints to check query processing time, validation of integration, and performance of the unified ontology model. The average response time for query execution on generated RDF triples on the three servers were approximately 0.144 seconds, 0.070 seconds, 0.062 seconds, respectively.</span>

show abstract

“…In [10], an architecture for ingesting data obtained from different production environments into a big data platform is presented. Apache NiFi is utilized for data ingestion to process both interval and real-time data from the production sites.…”

Section: Related Workmentioning

confidence: 99%

“…As already mentioned above, NiFi as data ingestion workflow application using a data flow like task execution and programming model already provides a good set of base technologies for setting up a computational scientific workflow environment. It is an enterprise integration and dataflow automation tool that allows a user to send, receive, route, transform, and sort data [10,41]. It provides real-time control that makes it easy to manage the movement of data between any source and any destination [41].…”

Section: Data Ingestion/transformation-apache Nifimentioning

confidence: 99%

“…Additionally, it is designed for extension that allows for building and adding own Apache NiFi processors and even allowing customizations of the key framework [5]. By using Apache NiFi as the base framework for the foreseen computational scientific workflow framework and extending it with advanced functionality for extended coordination and co-simulation interactions, the framework will allow users to easily build, execute and manage data processing or simulation workflows via an enhanced Apache NiFi web user interface [5,10]. Custom Apache NiFi processors can be designed for providing a library of custom workflow modules e.g., for integration of base models of a certain type, e.g., which are common elements for modelling smart grids [5].…”

Section: Data Ingestion/transformation-apache Nifimentioning

confidence: 99%

See 1 more Smart Citation

Architectural Concept and Evaluation of a Framework for the Efficient Automation of Computational Scientific Workflows: An Energy Systems Analysis Example

et al. 2019

View full text Add to dashboard Cite

Scientists and engineers involved in the design of complex system solutions use computational workflows for their evaluations. Along with growing system complexity, the complexity of these workflows also increases. Without integration tools, scientists and engineers are often highly concerned with how to integrate software tools and model sets, which hinders their original research or engineering aims. Therefore, a new framework for streamlining the creation and usage of automated computational workflows is introduced in the present article. It uses state-of-the-art technologies for automation (e.g., container-automation) and coordination (e.g., distributed message oriented middleware), and a microservice-based architecture for novel distributed process execution and coordination. It also supports co-simulations as part of larger workflows including additional auxiliary computational tasks, e.g., forecasting or data transformation. Using Apache NiFi, an easy-to-use web interface is provided to create, run and control workflows without the need to be concerned with the underlying computing infrastructure. Initial framework testing via the implementation of a real-world workflow underpins promising performance in the realms of parallelizability, low overheads and reliable coordination.

show abstract

Data integration in scalable data analytics platform for process industries

Cited by 9 publications

References 7 publications

Manufacturing big data ecosystem: A systematic literature review

Manufacturing big data ecosystem: A systematic literature review

A unified ontology-based data integration approach for the internet of things

Architectural Concept and Evaluation of a Framework for the Efficient Automation of Computational Scientific Workflows: An Energy Systems Analysis Example

Contact Info

Product

Resources

About