Visual integration tool for heterogeneous data type by unified vectorization

Abstract

Abstract-Data integration is the problem of combining data residing at different sources, and providing the user with a unified view of these data. One of the critical issues of data integration is the detection of similar entities based on the content. This complexity is due to three factors: the data type of the databases are heterogenous, the schema of databases are unfamiliar and heterogenous as well, and the amount of records is voluminous and time consuming to analyze. As solution to these problems we extend our work in another of our papers by introducing a new measure to handle heterogenous textual and numerical data type for coincident meaning extraction. Firstly, to in order accommodate the heterogeneous data types we propose a new weight called Bin Frequency -Inverse Document Bin Frequency (BF-IDBF) for effective heterogeneous data pre-processing and classification by unified vectorization. Secondly in order to handle the unfamiliar data structure, we use the unsupervised algorithm Self-Organizing Map. Finally to help the user to explore and browse the semantically similar entities among the copious amount of data, we use a SOM based visualization tool to map the database tables based on their semantical content.

Visual knowledge representation of conceptual semantic networks

“…The following list is not an exhaustive list of applications, but it provides an overview about the most recent areas of research where visualization became essential: a) Topic summarization: e.g., understanding newspaper articles, stories, reporting events, investigating crime reports, finding patterns in blogs, following the development of political campaigns, or observing topic trends in the bibliography of research approaches [7,3,25]; b) Visual Analysis of Social Networks: e.g., analyzing dynamic groups memberships in temporal social networks by using graphical representations [10,12,17,6,29]; c) Visual Clustering Analysis: e.g., using data mining techniques to find patterns in data to generate group of data based on (dis)similarity. Several visualization tools have been developed in this domain and gained great popularity, to mention some [21,2,28,5,30]; d) Semantic Visual Analysis: e.g., visual analysis of webpage/documents based on the semantic representation of text in a "semantic graph" [23,8,9,22,31], or exploring data in folksonomy systems based on a hierarchical semantic representation, "semantic cloud or tags" [11,14,24,23,4,15,16,22,26] …”

Clustering Relational Database Entities Using K-means

“…It appears that when two data types, such as numerical and textual, are simultaneously processed by HDM, the use of unified vectorization (UV) leads to better convergent semantic clustering results [1]. In spite of good results, the development and the use of similar data weighting measures to represent these heterogeneous data types in a unified VSM matrix improves the clustering results [2]. Let us examine these pre-processing steps.…”

“…For example, the 3-grams of the term "hello" are: "hel", "ell", and "llo". More details regarding the tokenization and the vectorization processes can be found in [2].…”

“…This is likely due to the availability and abundance of such data. For example, data integration sectors are interested to use heterogeneous data mining in order to detect similar data entities for building data warehouses [2]. Business intelligence sectors and financial institutions are extremely eager to extract coincident clustering from textual and numerical (financial specially) data based on the content.…”

“…Furthermore, when the heterogeneous data types are converted into algebraic models using similar weighting measures, the clustering results are enhanced when SOM is used [2].…”