Big-data-based network engineering recruitment data collection, warehousing, and analysis program

Abstract

In China,Network Engineering was officially established as an undergraduate major in 2012. Hundreds of thousands of Network Engineering graduates have entered the job market since 2016. What are their salary levels and employment quality? These are the major concerns of in-college Network Engineering students. Only by obtaining comprehensive, timely and reliable market frontier recruitment data, can they know what directions will bring higher salary and better growth space, thereby allocate their time more reasonably to learn the most beneficial courses. How to obtain timely, comprehensive and reliable cutting-edge recruitment data? This research introduces an automatic recruitment data collection, warehousing and analysis program based on big data. The program takes 51job.com as the target website, scrapes all the network related recruitment data in Guangzhou from June 2021 to June 2022. Then stores and visually analyzes the data, and finally obtains the statistical data including total number of recruitment, average salary and the change curve of salary over working years for various posts in the network specialty. The statistical results show that Linux maintenance engineers and information security engineers have higher wages and better development prospects. These help guide Network Engineering students to determine their own direction of efforts. In addition, the data collection, storage and analysis process of this study has wide range of applicability, and can be extended to other situations that need data to assist decision-making.