Data Integration of Heterogeneous Data Sources Using QR Decomposition

Harikumar, Sandhya; Roy, Mekha Meriam

doi:10.1007/978-3-319-23258-4_29

Cited by 6 publications

(5 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The mediated schemas were built either manually [43], semi-automatically [40,44,45], or automatically [41,42,[46][47][48][49][50][51]. The manual method is a time-consuming and inefficient solution in the case of big data, especially in the case of big data having many data sources with a massive number of attributes and relations.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Some research used upper-layer ontology to handle the semantics in the mediated schema building. One of the studies [40] used WordNet ontology as a base in finding the concepts synonyms, while other research [44,46,50] used domain ontology for the same purpose.…”

Section: Literature Reviewmentioning

confidence: 99%

“…Instead of using the domain-related semantic rules, some research used general similarity measures for calculating the similarity between concepts, such as Wu-Palmer, as in [41], cosine similarity, as in [40,44], and semantic proximity, as in [49,52]. These similarity measures previously used are suitable for calculating the similarity between objects in the surveyed works.…”

Section: Literature Reviewmentioning

confidence: 99%

See 2 more Smart Citations

A Framework for Enhancing Big Data Integration in Biological Domain Using Distributed Processing

et al. 2020

View full text Add to dashboard Cite

Massive heterogeneous big data residing at different sites with various types and formats need to be integrated into a single unified view before starting data mining processes. Furthermore, in most of applications and research, a single big data source is not enough to complete the analysis and achieve goals. Unfortunately, there is no general or standardized integration process; the nature of an integration process depends on the data type, domain, and integration purpose. Based on these parameters, we proposed, implemented, and tested a big data integration framework that integrates big data in the biology domain, based on the domain ontology and using distributed processing. The integration resulted in the same result as that obtained from the local integration. The results are equivalent in terms of the ontology size before the integration; in the number of added items, skipped items, and overlapped items; in the ontology size after the integration; and in the number of edges, vertices, and roots. The results also do not violate any logical consistency rules, passing all the logical consistency tests, such as Jena Ontology API, HermiT, and Pellet reasoners. The integration result is a new big data source that combines big data from several critical sources in the biology domain and transforms it into one unified format to help researchers and specialists use it for further research and analysis.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

A Framework for Enhancing Big Data Integration in Biological Domain Using Distributed Processing

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Specific application based data integration techniques include [13,14,22,23]. These techniques are focused on integrating data for knowledge processing in specific application scenarios.…”

Section: Related Workmentioning

confidence: 99%

Fast and Efficient Conflict Identification and Resolution in Huge Streaming Data

.¹,

Victor²

2016

IJCA

View full text Add to dashboard Cite

Increased data generation has led to an increase in the availability of rich information online. However, complications occur in the form of heterogeneity in the data storage. In order to have complete information, all the data sources must be utilized. Hence a data integration mechanism is required. However, integrating heterogeneous data leads to conflicting data in the system. This paper presents a fast and efficient mechanism to identify and resolve conflicts on huge streaming data using Spark. A wrapper based query formulation module constructs queries depending on the underlying data sources. The retrieved data is converted to a structured format and similarity between the data is identified, followed by distributed conflict identification and resolution. Experiments were conducted on streaming data. Effective conflict detections and a speed up from ~589 seconds to 10 seconds was achieved.

show abstract

“…A semantic similarity measure is a function that takes two GO terms or two sets of terms representing the annotations of two entities and returns a numerical value representing the closeness in meaning between them [6]. Standard SSMs such as Palmer's [7], cosine similarities [7, 8], and semantic proximity [9, 10] are suitable for some fields of study but are inaccurate for calculating semantic similarity between objects in other fields. In the field of biology, for example, comparing GO annotation terms is not enough; therefore, semantic similarity is measured by comparing features that describe the objects and the hierarchal relationships between these features [11–13].…”

Section: Introductionmentioning

confidence: 99%

Handling Big Data Scalability in Biological Domain Using Parallel and Distributed Processing: A Case of Three Biological Semantic Similarity Measures

Almasoud

Al‐Khalifa

Al-Salman

2019

BioMed Research International

View full text Add to dashboard Cite

In the field of biology, researchers need to compare genes or gene products using semantic similarity measures (SSM). Continuous data growth and diversity in data characteristics comprise what is called big data; current biological SSMs cannot handle big data. Therefore, these measures need the ability to control the size of big data. We used parallel and distributed processing by splitting data into multiple partitions and applied SSM measures to each partition; this approach helped manage big data scalability and computational problems. Our solution involves three steps: split gene ontology (GO), data clustering, and semantic similarity calculation. To test this method, split GO and data clustering algorithms were defined and assessed for performance in the first two steps. Three of the best SSMs in biology [Resnik, Shortest Semantic Differentiation Distance (SSDD), and SORA] are enhanced by introducing threaded parallel processing, which is used in the third step. Our results demonstrate that introducing threads in SSMs reduced the time of calculating semantic similarity between gene pairs and improved performance of the three SSMs. Average time was reduced by 24.51% for Resnik, 22.93%, for SSDD, and 33.68% for SORA. Total time was reduced by 8.88% for Resnik, 23.14% for SSDD, and 39.27% for SORA. Using these threaded measures in the distributed system, combined with using split GO and data clustering algorithms to split input data based on their similarity, reduced the average time more than did the approach of equally dividing input data. Time reduction increased with increasing number of splits. Time reduction percentage was 24.1%, 39.2%, and 66.6% for Threaded SSDD; 33.0%, 78.2%, and 93.1% for Threaded SORA in the case of 2, 3, and 4 slaves, respectively; and 92.04% for Threaded Resnik in the case of four slaves.

show abstract

Data Integration of Heterogeneous Data Sources Using QR Decomposition

Cited by 6 publications

References 8 publications

A Framework for Enhancing Big Data Integration in Biological Domain Using Distributed Processing

A Framework for Enhancing Big Data Integration in Biological Domain Using Distributed Processing

Fast and Efficient Conflict Identification and Resolution in Huge Streaming Data

Handling Big Data Scalability in Biological Domain Using Parallel and Distributed Processing: A Case of Three Biological Semantic Similarity Measures

Contact Info

Product

Resources

About