A Web Scraping Framework for Descriptive Analysis of Meteorological Big Data for Decision-Making Purposes

Abstract

With the large increase in the amount of heterogeneous, complex, and unstructured data issued from Web sources, there is an emerging need to develop Big Data technologies and tools to extract and manage this data. In this context, Web scraping for Big Data is a technique that has gained importance because of its rapidity and efficiency in gathering data for Big Data technology. This study was conducted to propose a Web Scraping framework for descriptive analysis of meteorological Big Data. The introduced framework makes it possible to extract a set of data to process it, present it in a form that is easier to analyze and understand, and use it for decision-making purposes about weather forecasts. The study employed descriptive analysis of available big data as it allows one to easily make quality and effective decisions. The web scraping process takes place in several stages, including data extraction, data archiving in a data warehouse, and finally data filtering and analysis. To test its applicability, the proposed web scraping framework was implemented and tested in the meteorological context to extract and present meteorological data issued from a specialized web source. The proposed system makes it possible to restore the data in the form of statistical models published in a dashboard. The results of the study revealed that the predictive models provided by the system are capable of predicting certain weather-related variables, such as humidity, precipitation, and temperature. The opportunities and implications to leverage the results of this study are many, including weather forecasting and decision support.

Understanding Climate Change and Air Quality Over the Last Decade: Evidence From News and Weather Data Processing

“…With updated products, packages and even dedicated software, data is more easily Extracted, Transformed, and Loaded (ETL) into convenient, easy-to-use files or databases. To perform such extraction, newer articles use Web Scraping (WS) as a tool to obtain the needed data, in real-time [30], [31] or one-time-only [32]- [34]. WS tools used in academics mainly focus on two directions [34]: 1) data analysis and visualization for predictions and decision-making processes and 2) developing models for obtaining feasible, clean data, unattainable in the past.…”

“…To perform such extraction, newer articles use Web Scraping (WS) as a tool to obtain the needed data, in real-time [30], [31] or one-time-only [32]- [34]. WS tools used in academics mainly focus on two directions [34]: 1) data analysis and visualization for predictions and decision-making processes and 2) developing models for obtaining feasible, clean data, unattainable in the past. Regarding the first focus, data analysis and visualization, WS is sporadically utilized to extract meteorological data [34].…”

“…WS tools used in academics mainly focus on two directions [34]: 1) data analysis and visualization for predictions and decision-making processes and 2) developing models for obtaining feasible, clean data, unattainable in the past. Regarding the first focus, data analysis and visualization, WS is sporadically utilized to extract meteorological data [34]. The data is used to create a dashboard containing weather forecasts (predictions for meteorological variables such as humidity, precipitations, and temperature) as a support for decisions related to future weather.…”