Data Quality in Linear Regression Models: Effect of Errors in Test Data and Errors in Training Data on Predictive Accuracy

Klein, Barbara; Rossin, Donald F.

doi:10.28945/599

Cited by 8 publications

(6 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, the GPM and SMAP input datasets exhibit their own sources of spatiotemporal uncertainties. However, errors in the training data are demonstrated to favor the generalization of regression models and improve their corrective performance against an output target [61]. This is particularly expected from the more robust ANN architecture which is capable of resolving nonlinear uncertainties compared to GWR [105].…”

Section: Discussionmentioning

confidence: 99%

“…Data pre-processing (Section 3) involves further steps to reduce the impact of remaining data quality issues on the training and model performance. Uncertainties from the aforementioned standard quality control steps remain, but may favor the generalization of model correction performance during the training stage [61]. On the other hand, pronounced errors may exist over the northeastern highlands due to terrain blockage and merging uncertainties.…”

Section: Radar-based Rainfall Estimatesmentioning

confidence: 99%

See 1 more Smart Citation

Enhancing Precipitation Estimates Through the Fusion of Weather Radar, Satellite Retrievals, and Surface Parameters

2020

View full text Add to dashboard Cite

Accurate and timely monitoring of precipitation remains a challenge, particularly in hyper-arid regions such as the United Arab Emirates (UAE). The aim of this study is to improve the accuracy of the Integrated Multi-satellitE Retrievals for the Global Precipitation Measurement (GPM) mission’s latest product release (IMERG V06B) locally over the UAE. Two distinct approaches, namely, geographically weighted regression (GWR), and artificial neural networks (ANNs) are tested. Daily soil moisture retrievals from the Soil Moisture Active Passive (SMAP) mission (9 km), terrain elevations from the Advanced Spaceborne Thermal Emission and Reflection digital elevation model (ASTER DEM, 30 m) and precipitation estimates (0.5 km) from a weather radar network are incorporated as explanatory variables in the proposed GWR and ANN model frameworks. First, the performances of the daily GPM and weather radar estimates are assessed using a network of 65 rain gauges from 1 January 2015 to 31 December 2018. Next, the GWR and ANN models are developed with 52 gauges used for training and 13 gauges reserved for model testing and seasonal inter-comparisons. GPM estimates record higher Pearson correlation coefficients (PCC) at rain gauges with increasing elevation (z) and higher rainfall amounts (PCC = 0.29 z0.12), while weather radar estimates perform better for lower elevations and light rain conditions (PCC = 0.81 z−0.18). Taylor diagrams indicate that both the GWR- and the ANN-adjusted precipitation products outperform the original GPM and radar estimates, with the poorest correction obtained by GWR during the summer period. The incorporation of soil moisture resulted in improved corrections by the ANN model compared to the GWR, with relative increases in Nash–Sutcliffe efficiency (NSE) coefficients of 56% (and 25%) for GPM estimates, and 34% (and 53%) for radar estimates during summer (and winter) periods. The ANN-derived precipitation estimates can be used to force hydrological models over ungauged areas across the UAE. The methodology is expandable to other arid and hyper-arid regions requiring improved precipitation monitoring.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Radar-based Rainfall Estimatesmentioning

confidence: 99%

Enhancing Precipitation Estimates Through the Fusion of Weather Radar, Satellite Retrievals, and Surface Parameters

2020

View full text Add to dashboard Cite

show abstract

“…The authors in [17] define data quality as "the degree of fulfilment of all those requirements defined for data, which is needed for a specific purpose". On the other hand, according to the authors of [18], the data errors may affect the predictive accuracy of linear regression models in two ways. First, the training data used to build the model may contain errors.…”

Section: Data Qualitymentioning

confidence: 99%

How to Address the Data Quality Issues in Regression Models: A Guided Process for Data Cleaning

Corrales

Ledezma

2018

Symmetry

View full text Add to dashboard Cite

Today, data availability has gone from scarce to superabundant. Technologies like IoT, trends in social media and the capabilities of smart-phones are producing and digitizing lots of data that was previously unavailable. This massive increase of data creates opportunities to gain new business models, but also demands new techniques and methods of data quality in knowledge discovery, especially when the data comes from different sources (e.g., sensors, social networks, cameras, etc.). The data quality process of the data set proposes conclusions about the information they contain. This is increasingly done with the aid of data cleaning approaches. Therefore, guaranteeing a high data quality is considered as the primary goal of the data scientist. In this paper, we propose a process for data cleaning in regression models (DC-RM). The proposed data cleaning process is evaluated through a real datasets coming from the UCI Repository of Machine Learning Databases. With the aim of assessing the data cleaning process, the dataset that is cleaned by DC-RM was used to train the same regression models proposed by the authors of UCI datasets. The results achieved by the trained models with the dataset produced by DC-RM are better than or equal to that presented by the datasets' authors.

show abstract

“…Recently, DQ has become more visible as corporations learned from their costly experience. It is well accepted nowadays that for an Enterprise Resource Planning (ERP) or Data Warehouse project to be successful, firms must attend to DQ [7][8][9][10][11][12]. By the same token, in achieving TIA to fight terrorism successfully, DQ must be considered to avoid garbage-ingarbage-out.…”

Section: Research Challengementioning

confidence: 99%

“…The state department of health (CB 4 ) and the HMO (CB 5 ) are the consumers of the two information products in that order. For each of the three products, the set of data items used to generate each is different and is shown by the component data items CD 11 , CD 12 , and CD 13 .…”

Section: Quality Criteriamentioning

confidence: 99%

An Information Product Approach for Total Information Awareness

et al. 2003

View full text Add to dashboard Cite

Abstract--To fight terrorism successfully, the quality of data must be considered to avoid garbage-in-garbage-out. Research has shown that data quality (DQ) goes beyond accuracy to include dimensions such as believability, timeliness, and accessibility. In collecting, processing, and analyzing a much broader array of data than we do currently, therefore, a comprehensive approach must be developed to ensure that DQ is incorporated in determining the most probable current or future scenario for preemption, national security warning and decision making. Additional data such as who was the data source, when was the data made available, how, where, and why also need to be included to judge the quality of the information assembled from these data.We propose such an approach for Total Information Awareness with Quality (TIAQ), which includes concepts, models, and tools. Central to our approach is to manage information as a product with four principles. We have applied the information product approach to research sites where opportunities arise. For example, the Air Force Material Command uses requirements definition and forecasting processes to perform a number of functions. However, the Air Force experienced several complex problems due to DQ problems; as a result, fuel pumps were unavailable. Each engine needs a fuel pump; when a pump is not available, a military aircraft is grounded. We traced the fuel-pump throughout the process of remanufacture, and identified root causes such as delays by pump contractors and ordering problems. To a certain extent, detecting foreign terrorists and decipher their plots are analogous to tracing fuel pumps. Our research provides an interdisciplinary approach to facilitating Total Information Awareness.

show abstract

Data Quality in Linear Regression Models: Effect of Errors in Test Data and Errors in Training Data on Predictive Accuracy

Cited by 8 publications

References 41 publications

Enhancing Precipitation Estimates Through the Fusion of Weather Radar, Satellite Retrievals, and Surface Parameters

Enhancing Precipitation Estimates Through the Fusion of Weather Radar, Satellite Retrievals, and Surface Parameters

How to Address the Data Quality Issues in Regression Models: A Guided Process for Data Cleaning

An Information Product Approach for Total Information Awareness

Contact Info

Product

Resources

About