How to Address the Data Quality Issues in Regression Models: A Guided Process for Data Cleaning

Corrales, David Camilo; Corrales, Juan Carlos; Ledezma, Agapito

doi:10.3390/sym10040099

Cited by 32 publications

(27 citation statements)

References 71 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data acquired in Section 2.1 was prepared and cleaned in order to obtain an appropriate dataset for understanding the fluctuation of the avocado market in the United States, based on different weather conditions. For this purpose, we followed the Cross Industry Standard Process for Data Mining (CRISP-DM) methodology [14] and the data cleaning process proposed in [15]. Figure 1 exposes the data preparation tasks that were considered.…”

Section: Data Selection and Cleaningmentioning

confidence: 99%

Estimating Avocado Sales Using Machine Learning Algorithms and Weather Data

2018

Self Cite

View full text Add to dashboard Cite

Persea americana, commonly known as avocado, is becoming increasingly important in global agriculture. There are dozens of avocado varieties, but more than 85% of the avocados harvested and sold in the world are of the Hass one. Furthermore, information on the market of agricultural products is valuable for decision-making; this has made researchers try to determine the behavior of the avocado market, based on data that might affect it one way or another. In this paper, a machine learning approach for estimating the number of units sold monthly and the total sales of Hass avocados in several cities in the United States, using weather data and historical sales records, is presented. For that purpose, four algorithms were evaluated: Linear Regression, Multilayer Perceptron, Support Vector Machine for Regression and Multivariate Regression Prediction Model. The last two showed the best accuracy, with a correlation coefficient of 0.995 and 0.996, and a Relative Absolute Error of 7.971 and 7.812, respectively. Using the Multivariate Regression Prediction Model, an application that allows avocado producers and sellers to plan sales through the estimation of the profits in dollars and the number of avocados that could be sold in the United States was created.

show abstract

Section: Data Selection and Cleaningmentioning

confidence: 99%

Estimating Avocado Sales Using Machine Learning Algorithms and Weather Data

2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…• Build an integrated data quality framework for several knowledge discovery tasks as regression [37], clustering and association rules. The integrated data quality framework would consider the Big Data paradigm [29] and hence huge datasets.…”

Section: Discussionmentioning

confidence: 99%

From Theory to Practice: A Data Quality Framework for Classification Tasks

2018

Self Cite

View full text Add to dashboard Cite

Abstract:The data preprocessing is an essential step in knowledge discovery projects. The experts affirm that preprocessing tasks take between 50% to 70% of the total time of the knowledge discovery process. In this sense, several authors consider the data cleaning as one of the most cumbersome and critical tasks. Failure to provide high data quality in the preprocessing stage will significantly reduce the accuracy of any data analytic project. In this paper, we propose a framework to address the data quality issues in classification tasks DQF4CT. Our approach is composed of: (i) a conceptual framework to provide the user guidance on how to deal with data problems in classification tasks; and (ii) an ontology that represents the knowledge in data cleaning and suggests the proper data cleaning approaches. We presented two case studies through real datasets: physical activity monitoring (PAM) and occupancy detection of an office room (OD). With the aim of evaluating our proposal, the cleaned datasets by DQF4CT were used to train the same algorithms used in classification tasks by the authors of PAM and OD. Additionally, we evaluated DQF4CT through datasets of the Repository of Machine Learning Databases of the University of California, Irvine (UCI). In addition, 84% of the results achieved by the models of the datasets cleaned by DQF4CT are better than the models of the datasets authors.

show abstract

“…Data quality issues generally appear when the quality requirements are not met on the data values [41]. These issues are due to several factors or processes having occurred at different levels: In [21,35,42], many causes of poor data quality were enumerated, and a list of elements, which affect the quality and DQD's was produced. This list is illustrated in Table 4.…”

Section: Data Quality Issuesmentioning

confidence: 99%

Big data quality framework: a holistic approach to continuous quality management

et al. 2021

View full text Add to dashboard Cite

Big Data is an essential research area for governments, institutions, and private agencies to support their analytics decisions. Big Data refers to all about data, how it is collected, processed, and analyzed to generate value-added data-driven insights and decisions. Degradation in Data Quality may result in unpredictable consequences. In this case, confidence and worthiness in the data and its source are lost. In the Big Data context, data characteristics, such as volume, multi-heterogeneous data sources, and fast data generation, increase the risk of quality degradation and require efficient mechanisms to check data worthiness. However, ensuring Big Data Quality (BDQ) is a very costly and time-consuming process, since excessive computing resources are required. Maintaining Quality through the Big Data lifecycle requires quality profiling and verification before its processing decision. A BDQ Management Framework for enhancing the pre-processing activities while strengthening data control is proposed. The proposed framework uses a new concept called Big Data Quality Profile. This concept captures quality outline, requirements, attributes, dimensions, scores, and rules. Using Big Data profiling and sampling components of the framework, a faster and efficient data quality estimation is initiated before and after an intermediate pre-processing phase. The exploratory profiling component of the framework plays an initial role in quality profiling; it uses a set of predefined quality metrics to evaluate important data quality dimensions. It generates quality rules by applying various pre-processing activities and their related functions. These rules mainly aim at the Data Quality Profile and result in quality scores for the selected quality attributes. The framework implementation and dataflow management across various quality management processes have been discussed, further some ongoing work on framework evaluation and deployment to support quality evaluation decisions conclude the paper.

show abstract

How to Address the Data Quality Issues in Regression Models: A Guided Process for Data Cleaning

Cited by 32 publications

References 71 publications

Estimating Avocado Sales Using Machine Learning Algorithms and Weather Data

Estimating Avocado Sales Using Machine Learning Algorithms and Weather Data

From Theory to Practice: A Data Quality Framework for Classification Tasks

Big data quality framework: a holistic approach to continuous quality management

Contact Info

Product

Resources

About