Implementation of Distributed Crawler System Based on Spark for Massive Data Mining

Liu, Feng; Wang, Xin

doi:10.1109/icccs49078.2020.9118442

Cited by 4 publications

(3 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Chaturvedi et al [95] 2020 evolution mining and change mining are used to changing distributed systems in this study.…”

Section: Discussion and Comparisonmentioning

confidence: 99%

“…Web crawlers were maturely contributed with major search engines including search fields during a period of the rapid growth of Internet technologies and rising social desires of people [21]. F. Liu and W. Xin [95] demonstrates the architecture incorporation of the Spark-based distributed crawler system, provides a corresponding framework diagram, which presents the distributed framework platform in depth utilizing Spark's RDD elastic computational model and task assignment algorithm. However, we can fix the issue of insufficient resource consumption and poor collection performance using this Spark-based distributed crawler method, and then resolve the contradiction between the present exponential growth of data size as well as the speed of collecting information [96,97].…”

Section: Literature Reviewmentioning

confidence: 99%

See 1 more Smart Citation

A Survey of Data Mining Activities in Distributed Systems

Mohammad

Yasin

Salih

et al. 2021

AJRCoS

View full text Add to dashboard Cite

Distributed systems, which may be utilized to do computations, are being developed as a result of the fast growth of sharing resources. Data mining, which has a huge range of real applications, provides significant techniques for extracting meaningful and usable information from massive amounts of data. Traditional data mining methods, on the other hand, suppose that the data is gathered centrally, stored in memory, and is static. Managing massive amounts of data and processing them with limited resources is difficult. Large volumes of data, for instance, are swiftly generated and stored in many locations. This becomes increasingly costly to centralize them at a single location. Furthermore, traditional data mining methods typically have several issues and limitations, such as memory restrictions, limited processing ability, and insufficient hard drive space, among others. To overcome the following issues, distributed data mining's have emerged as a beneficial option in several applications According to several authors, this research provides a study of state-of-the-art distributed data mining methods, such as distributed common item-set mining, distributed frequent sequence mining, technical difficulties with distributed systems, distributed clustering, as well as privacy-protection distributed data mining. Furthermore, each work is evaluated and compared to the others.

show abstract

“…Chaturvedi et al [95] 2020 evolution mining and change mining are used to changing distributed systems in this study.…”

Section: Discussion and Comparisonmentioning

confidence: 99%

Section: Literature Reviewmentioning

confidence: 99%

A Survey of Data Mining Activities in Distributed Systems

Mohammad

Yasin

Salih

et al. 2021

AJRCoS

View full text Add to dashboard Cite

show abstract

“…The mechanism makes DPoS faster than normal proof of stake (PoS), and PBFT runs efficiently with the use of fewer preselected generals. The whole calculation process is similar to the spark mechanism, [55][56][57] but we use the blockchain consensus mechanism and HP-B to replace the functions of the central server (driver) and resident distributed data set in the spark mechanism to improve the security of the model training process, strengthen the verification of the node submission model, and maintain the consistency of the storage model parameters.…”

Section: Dpos + Pbft Consensus Mechanismmentioning

confidence: 99%

A blockchain‐enabled learning model based on distributed deep learning architecture

Zhang

Liang

Jia

et al. 2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

Aiming to address the unsatisfactory performance of existing distributed deep learning architectures, such as poor accuracy, slow network communication, low arithmetic speed, and insufficient security, we propose and design a learning model based on a distributed deep learning and blockchain architecture. We use a hybrid parallel algorithm based on blockchain (HP‐B) to build a distributed deep consensus learning model. The HP‐B algorithm is grouped according to the performance of computing nodes participating in training, network links and training samples, and the grouped computing equipment performs optimal distributed computing. The purpose of this approach is to solve the security and scalability concerns and improve the convergence speed and accuracy of deep learning. The proposed method achieves good results on the CIFAR‐100, CIFAR‐10, and IMAGENET data sets. Finally, the distributed deep learning model based on blockchain is combined with the generative adversarial network to solve the segmentation problem of medical imaging data, and the experimental results are superior to those of other networks.

show abstract