Designing a Framework for Data Quality Validation of Meteorological Data System

Abstract

In the current era of data science, data quality has a significant and critical impact on business operations. This is no different for the meteorological data encountered in the field of meteorology. However, the conventional methods of meteorological data quality control mainly focus on error detection and null-value detection; that is, they only consider the results of the data output but ignore the quality problems that may also arise in the workflow. To rectify this issue, this paper proposes the Total Meteorological Data Quality (TMDQ) framework based on the Total Quality Management (TQM) perspective, especially considering the systematic nature of data warehousing and process focus needs. In practical applications, this paper uses the proposed framework as the basis for the development of a system to help meteorological observers improve and maintain the quality of meteorological data in a timely and efficient manner. To verify the feasibility of the proposed framework and demonstrate its capabilities and usage, it was implemented in the Tamsui Meteorological Observatory (TMO) in Taiwan. The four quality dimension indicators established through the proposed framework will help meteorological observers grasp the various characteristics of meteorological data from different aspects. The application and research limitations of the proposed framework are discussed and possible directions for future research are presented.

A systematic literature review on data quality assessment

“…This methodology suggests a series of stages to examine the consequences of faults, injecting faults methodically throughout the process to identify various abnormal circumstances [104]. Furthermore, the total meteorological ❒ ISSN: 2302-9285 data quality (TMDQ) framework, based on the total quality management (TQM) approach developed by [115], aims to offer observers with diverse meteorological data qualities from numerous perspectives according to four quality dimensions, accuracy, consistency, completeness, and timeliness. At the basis of this framework, a validation system is developed to assist meteorological observers in more efficiently improving and maintaining the quality of meteorological data.…”

“…Total Nb of Value It remains almost the same dimensions used for the meteorological data, however, the specific metrics and criteria used to evaluate the quality may vary based on the unique characteristics and requirements of meteorological data. Tsai and Chan [115] proposed the metrics shown in Table 14. Finally, Liu et al [110] provided a summary on the current situation of research on data quality, they analyzed most pertinent characteristics of quality and defined evaluation criteria based on user-defined requirements.…”

Cloud–edge cooperation for meteorological radar big data: a review of data quality control

“…Sophisticated weather radar networks are nowadays playing a significant role in catastrophic weather monitoring, forecasting and early warning [6][7][8][9]. Generally, weather radar data are transmitted to a centralized cloud data center through the network for data quality control and then operational applications, so it is critical to improve weather radar data quality.…”