Design and realization of bank history data management system based on Hadoop 2.0

Guo, Meiwen

doi:10.1007/s10586-018-1867-y

Cited by 5 publications

(3 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…is also growing so fast that traditional data storage methods and data processing methods will not be able to cope with it [2]. For this reason, some new technologies such as distributed databases or multithreaded parallel computing have emerged, but issues such as data dependency and integrity, cluster load balancing, and task scheduling, as well as machine failures when dealing with parallel and distributed computing, can all cause more trouble for the average user [3]. In recent years, thanks to the development of methodologies in related scientific fields, especially some new tools applied in network analysis, scientists have started to consider the impact of network structure on recommender systems and how to use known network structure features to improve recommendation accuracy [4].…”

Section: Introductionmentioning

confidence: 99%

“…(1) Discrepancies in scope (2) Discrepancies in the description of the research reported (3) Discrepancies between the availability of data and the research described (4) Inappropriate citations (5) Incoherent, meaningless and/or irrelevant content included in the article (6) Manipulated or compromised peer review Te presence of these indicators undermines our confdence in the integrity of the article's content and we cannot, therefore, vouch for its reliability. Please note that this notice is intended solely to alert readers that the content of this article is unreliable.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Hadoop-Based Painting Resource Storage and Retrieval Platform Construction and Testing

2021

Complexity

View full text Add to dashboard Cite

This paper adopts Hadoop to build and test the storage and retrieval platform for painting resources. This paper adopts Hadoop as the platform and MapReduce as the computing framework and uses Hadoop Distributed Filesystem (HDFS) distributed file system to store massive log data, which solves the storage problem of massive data. According to the business requirements of the system, this paper designs the system according to the process of web text mining, mainly divided into log data preprocessing module, log data storage module, log data analysis module, and log data visualization module. The core part of the system is the log data analysis module. The analysis of search keywords ranking, Uniform Resource Locator (URL), and user click relationship, URL ranking, and other dimensions are realized through data statistical analysis, and Canopy coarse clustering is performed first according to search keywords, and then K-means clustering is used for the results after Canopy clustering, and the calculation of cosine similarity is adopted to realize the grouping of users and build user portrait. The Hadoop development environment is installed and deployed, and functional and performance tests are conducted on the contents implemented in this system. The constructed private cloud platform for remote sensing image data can realize online retrieval of remote sensing image metadata and fast download of remote sensing image data and solve the problems in storage, data sharing, and management of remote sensing image data to a certain extent.

show abstract

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

Hadoop-Based Painting Resource Storage and Retrieval Platform Construction and Testing

2021

Complexity

View full text Add to dashboard Cite

show abstract

“…Several studies focused on improving Hadoop performance [8][9][10]. Aydin et al [11] developed a Hadoop program for distributed data log analysis with a small cloud.…”

Section: Literature Reviewmentioning

confidence: 99%

A Multi-Optimization Technique for Improvement of Hadoop Performance with a Dynamic Job Execution Method Based on Artificial Neural Network

et al. 2020

View full text Add to dashboard Cite

The improvement of Hadoop performance has received considerable attention from researchers in cloud computing fields. Most studies have focused on improving the performance of a Hadoop cluster. Notably, various parameters are required to configure Hadoop and must be adjusted to improve performance. This paper proposes a mechanism to improve Hadoop, schedule jobs, and allocate and utilize resources. Specifically, we present an improved ant colony optimization method to schedule jobs according to the job size and the time expected for execution. Priority is given to the job with the minimum data size and minimum response time. The resource usage and running jobs by data node are predicted using an artificial neural network, and job activity and resource usage are monitored using the resource manager. Moreover, we enhance the Hadoop Name node performance by adding an aggregator node to the default HDFS framework architecture. The changes involve four entities: the name node, secondary name node, aggregator nodes, and data nodes, where the aggregator node is responsible for assigning the jobs among the data node, and the Name node keeps tracking only the aggregator nodes. We test the overall scheme among Amazon EC2 and S3, and show the results of throughput and CPU response time for different data sizes. Finally, we show that the proposed approach shows significant improvement compare to native Hadoop and other approaches.

show abstract

Innovative Secure Authentication Interface for Hadoop Cluster Using DNA Cryptography: A Practical Study

Balaraju

Rao

2020

Advances in Intelligent Systems and Computing

View full text Add to dashboard Cite

Design and realization of bank history data management system based on Hadoop 2.0

Cited by 5 publications

References 2 publications

Hadoop-Based Painting Resource Storage and Retrieval Platform Construction and Testing

Hadoop-Based Painting Resource Storage and Retrieval Platform Construction and Testing

A Multi-Optimization Technique for Improvement of Hadoop Performance with a Dynamic Job Execution Method Based on Artificial Neural Network

Innovative Secure Authentication Interface for Hadoop Cluster Using DNA Cryptography: A Practical Study

Contact Info

Product

Resources

About