MusQ: A Multi-Store Query System for IoT Data Using a Datalog-Like Language

Abstract

The growing number of connected Internet of Things (IoT) devices has increased the necessity for processing IoT data from multiple heterogeneous data stores. IoT data integration is a challenging problem owing to the heterogeneity of data stores in terms of their query language, data models, and schemas. In this paper, we propose a multi-store query system for IoT data called MusQ, where users can formulate join operation queries for heterogeneous data sources. To reconcile the heterogeneity between source schemas of IoT data stores, we extract a global schema from local source schemas semi-automatically by applying schema-matching and schema-mapping steps. In order to minimize the burden on the user to understand the finer details of various query languages, we define a unified query language called the multi-store query language (MQL), which follows a subset of the Datalog grammar. Thus, users can easily retrieve IoT data from multiple heterogeneous sources with MQL queries. As the three MQL query-processing join algorithms are based on a mediator-wrapper approach, MusQ performs efficient data integration over significant volumes of IoT data from multiple stores. We conduct extensive experiments to evaluate the performance of the MusQ system using a synthetic and large real IoT data set for three different types of data stores (RDBMS, NoSQL, and HDFS). The experimental results demonstrate that MusQ is suitable, scalable, and efficient query processing for multiple heterogeneous IoT data stores. Those advantages of MusQ are important in several areas that involve complex IoT systems, such as smart city, healthcare, and energy management. INDEX TERMS Data management and analytics, Internet of Things, multi-store system, query processing, schema integration. FIGURE 2. Grammar of MQL queries.

Collaborative decision-making of multi-source heterogeneous sensing data for smart power IoT

“…Pearson's coefficient and distance coefficient are used to describe the data relationship between two non-homologous data, mainly to obtain the similarity coefficient of non-homologous data on the data [6]. Firstly, according to Pearson coefficient theory, given two variables x, and y, the Pearson coefficients of x and y are usually defined as: , ( , )…”

A Schema Integration Approach for Big Data Analysis

“…Ramadhan et al [8] propose a semi-automatic schema integration approach composed of two phases which are schema matching and schema mapping. In the schema matching phase, a similarity score is defined for the different attributes of local schemas by comparing schema instances as well as attribute names and datatypes using a string-based similarity measure and a WordNet based semantic similarity measure.…”

An adaptive spark-based framework for querying large-scale NoSQL and relational databases

“…However, these frameworks don’t take into account the rest of NoSQL data-stores. The articles [ 13 , 22 ]. presented a mediation platform based on rewriting queries.…”

“…The following models and algorithms were proposed by Haery: Key-Cube, an improved Z-order linearization algorithm and address tree, Accumulation, a key-cube development method, Query methods to implement key-cube and physical storage queries [ 12 ] and system architecture, components and execution of Haery. MusQ, a method of multi-store queries suggested by Ramadan, Hani and others [ 13 ], uses a structured approach. Three key features are accomplished by MusQ by solving three crucial challenges: (1) constructing a global schema by leveraging relationships from local source systems, as in the federal approach; (2) conducting complicated queries on various data stores without learning all the different database languages; and (3) effectively executing user queries (joining procedures in particular) to obtain relevant data.…”