Research and Analysis for Real-Time Streaming Big Data Based on Controllable Clustering and Edge Computing Algorithm

Abstract

Aiming at the low efficiency, poor performance and weak stability of traditional clustering algorithms and the poor response to the processing of massive data in real time, a real-time streaming controllable clustering edge computing algorithm (SCCEC) is proposed. First, the data tuples that arrive in real time are pre-processed by coarse clustering, the number of clusters, and the position of the center point are determined, and a set formed by macro clusters having differences is formed. Secondly, the macro cluster set obtained by the coarse clustering is sampled, and then K-means parallel clustering is performed with the largest and smallest distances, thereby realizing fine clustering of data. Finally, the completely clustering algorithm and the edge-computing algorithm are combined to realize the clustering analysis under the edgecomputing framework. The experimental results show that the proposed algorithm has the advantages of high efficiency, good quality, and strong stability. It can quickly obtain the global optimal solution, and deal with massive data with high real-time performance. It can be used for real-time streaming data aggregation under big data background. INDEX TERMSReal-time streaming data, clustering, edge computing, algorithm.

Causal Identification Based on Compressive Sensing of Air Pollutants Using Urban Big Data

“…1) Data-driven air quality analysis. Several researchers [1][2][3][4][5][6] proposed approaches to forecast and estimate air quality by analyzing and processing the correlations and patterns found in heterogeneous big data. Shang et al [7] estimated gas consumption and pollutant emissions based on GPS data.…”

“…For the sparsity in the time-domain, data compression can further reduce the amount of data for causal analysis. The process can be described in (5)…”