A general framework for big data knowledge discovery and integration

Abstract

Data structure description, conceptual modeling, and logic reasoning for knowledge discovery are three critical factors for the integration of information with heterogeneity. In particular, technologies of NoSQL databases and Internet of Things raise an urgent requirement for a uniform expression of heterogeneous data, and little attention has been paid to researches on the integration of NoSQL databases with traditional data models, as well as the semantic description of big data. To tackle these problems, in this paper, a concept-and-relation-oriented grid data model called GODM model is first proposed based on the definitions of Monad, Compounder, Relation, etc. Then, the GODM model is utilized to uniformly describe traditional data models and NoSQL data models, which eliminates structure differences of heterogeneous data. Next, based on the GODM relation mechanism, an extendable semantic system is built up by choosing SHOIQ(D) description logic as the example to establish the correspondence with GODM grammar subset, providing a fundamental support for semantic integration and knowledge discovery of heterogeneous data. After that, comprehensive comparisons with GODM and other models are made, especially the distinctions between GODM and OWL on the aspects of relation mechanism, hybrid schema, description logic, grammatical constructors, etc. Besides, experimental evaluations and analyses on time and space efficiencies of some primary common data models are conducted after the proposal of a general evaluation model, with the results showing that the GODM model has great advantage on properties of expressiveness, flexibility, etc, particularly time and space efficiency. In summary, the GODM model describes heterogeneous data from both aspects of data structure and semantic relationship and realizes a hybrid schema reconciling the schemaful and schemaless data models, making it especially suitable for dynamic data integration and knowledge discovery from big data models. KEYWORDSdata integration, data model, GODM, hybrid schema, knowledge representation, NoSQL, time and space efficiency INTRODUCTIONThe development of cloud computing and big data technologies prompts data to expand dramatically and uncontrollably in size, structure, and format. To eliminate data heterogeneity, a unified mechanism is necessary to provide consistent data manipulation approaches and tap the value of mass data in maximum. According to the implementation technique and application requirement, data integration approaches are generally classified into a materialized approach and a virtual approach, 1 where the materialized approaches mainly refer to data warehousing, 2 and the virtual approaches have two sub-branches: structural approaches and semantic approaches. 1 The warehousing methods preprocess data before integration, and all the integrated data are loaded in the warehouse to guarantee the efficiency of a data query. However, this kind of methods always lead Concurrency Computat Pract Exper. 2018;30:e4422. wileyonlinelibrary.com/j...

A multilevel graph approach for rainfall forecasting: A preliminary study case on London area

“…The “things” or devices are recognizable and acquire “intelligence” thanks to the fact of being able to communicate information about themselves and access to aggregated information from other “things” and people . The continuous exchange of information between man‐man, machine‐machine, and machine‐man inevitably produces a huge amount of data, which demands analyses of data using unconventional methods within a Big Data context . This leads us to pose some questions, ie, How can we properly process these data?…”

“…3 The continuous exchange of information between man-man, machine-machine, and machine-man inevitably produces a huge amount of data, which demands analyses of data using unconventional methods within a Big Data context. [4][5][6] This leads us to pose some questions, ie, How can we properly process these data? How properly use these data in order to increase the competitiveness and efficiency of services, and how could they contribute to social development?…”