The paper proposes a method for Big data analyzing in the presence of different data sources and different methods of processing these data. The Big data definition is given, the main problems of data mining process are described. The concept of association rules is introduced and the method of association rules searching for working with Big Data is modified. The method of finding dependencies is developed, efficiency and possibility of its parallelization are determined. The developed algorithm makes it possible to assert that the task of detecting association dependencies in distributed databases belongs to the class of P-tasks. The algorithm for finding association dependencies is well-solved with MapReduce. The low asymptotic complexity of the developed association rules mining algorithm and a wide set of data types supported for analysis allow to apply the proposed algorithm in practically all subject areas working with association dependencies in the data domain.
Background: Increasing the amount of information generated as a result of smart city activity leads to the problem of its accumulation and preprocessing. One type of data preprocessing is clustering. The cluster analysis is an objective method of classification. It provides an appropriate choice of further processing methods as well as the visualization and interpretation of the collected data, which are multidimensional objects. The most valuable feature of cluster analysis is the representation of the result by an image of a dendrogram that reflects a particular hierarchy of relationships between the selected clusters and their objects. The aim of the paper is to develop method of 3D visualization of hierarchical clustering for streaming and multidimensional data collected from IoT devices and open databases. Methods: It is suggested that a more detailed interpretation of the dendrogram is made by implementing the hypothesis given above. Testing this hypothesis means a procedure of visualizing and interpreting the result of a cluster analysis. The disclosed dendrogram allows fully usage of association metrics. Since this metric is derived from the calculation of the values of the proximity matrix in accordance with the chosen object pooling strategy, the use of the disclosed dendrogram is quite legitimate. In addition, the procedure for opening the dendrogram is specific and unambiguous. This methods is built on hierarchical clustering algorithm as the simplest and fasters one. The developed algorithm should make it impossible to cross clusters on a plane. It is also necessary to look for the distance not only between objects, but also between clusters, represented as complex geometric figures. It will allow explaining the nature of the clusters Results: The result of the research and verification of the proposed hypothesis is the diclosure of the dendrogram algorithm as the extension of classical methods of cluster analysis. This extension is made by studying and disclosing the resulting image of the dendrogram. The dendrogram visualization thus obtained differs significantly from the classical results. The opening of the dendrogram according to the developed algorithm allows us 3D visualization of the analysis results, as well as calculating the area and perimeter of the obtained clusters. Therefore, using analytical geometry methods, it is quite easy to isolate and calculate the parameters of minimum cluster coverage surfaces and the immediate distances between any objects of one or different clusters, as well as between the objects of a given cluster. This, in turn, is a significant complement to cluster analysis. Conclusion: The disclosed dendrogram retains proportions in distances between objects. On the basis of these characteristics, it is possible to determine the close relationship between the clusters themselves by correlating the values of their quantitative averaged values of the traits. Thus, the opening of the dendrogram allows us to clearly identify the set of clusters, each of which has its own distribution of the range of features values. The quantitative characteristics of clusters on both dendrograms are quite simple. In addition, the mean values of the features of objects in a given cluster can be interpreted as generalized characteristics of this cluster, and the cluster itself can be represented as a single integral object.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.