Design and Implementation of Log Data Analysis Management System Based on Hadoop

Abstract

With the rapid development of the Internet, many enterprises have launched their network platforms. When users browse, search, and click the products of these platforms, most platforms will keep records of these network behaviors, these records are often heterogeneous, and it is called log data. To effectively to analyze and manage these heterogeneous log data, so that enterprises can grasp the behavior characteristics of their platform users in time, to realize targeted recommendation of users, increase the sales volume of enterprises' products, and accelerate the development of enterprises. Firstly, we follow the process of big data collection, storage, analysis, and visualization to design the system, then, we adopt HDFS storage technology, Yarn resource management technology, and gink load balancing technology to build a Hadoop cluster to process the log data, and adopt MapReduce processing technology and data warehouse hive technology analyze the log data to obtain the results. Finally, the obtained results are displayed visually, and a log data analysis system is successfully constructed. It has been proved by practice that the system effectively realizes the collection, analysis and visualization of log data, and can accurately realize the recommendation of products by enterprises. The system is stable and effective.

Single-Choice Aided Marking System Research Based on Back Propagation Neural Network

“…The realization of the auxiliary judging system is the cross integration of multiple fields, including deep learning, pattern recognition, image processing and other aspects [27][28][29][30][31]. The realization of this system is conducive to the implementation of automatic marking methods in small-scale examinations in primary and secondary schools and even colleges.…”