Garbage in, Garbage out? An Empirical Look at Oracle Mistakes by End-User Programmers

Phalgune, A.; Kissinger, Cory; Burnett, Margaret; Cook, Curtis R.; Beckwith, Laura; Ruthruff, Joseph R.

doi:10.1109/vlhcc.2005.40

Cited by 20 publications

(26 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The error rate is consistent with earlier findings regarding end-user programmers' accuracy in serving as oracles when debugging, which have reported error rates of 5-20% (e.g., [21]). This error rate seems fairly robust across studies, and suggests a similar level of "noise" that users' judgments would introduce into the learning algorithm's data.…”

Section: Accuracysupporting

confidence: 90%

Toward harnessing user feedback for machine learning

Stumpf

Rajaram

et al. 2007

Proceedings of the 12th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

There has been little research into how end users might be able to communicate advice to machine learning systems. If this resource-the users themselves-could somehow work hand-in-hand with machine learning systems, the accuracy of learning systems could be improved and the users' understanding and trust of the system could improve as well. We conducted a think-aloud study to see how willing users were to provide feedback and to understand what kinds of feedback users could give. Users were shown explanations of machine learning predictions and asked to provide feedback to improve the predictions. We found that users had no difficulty providing generous amounts of feedback. The kinds of feedback ranged from suggestions for reweighting of features to proposals for new features, feature combinations, relational features, and wholesale changes to the learning algorithm. The results show that user feedback has the potential to significantly improve machine learning systems, but that learning algorithms need to be extended in several ways to be able to assimilate this feedback.

show abstract

Section: Accuracysupporting

confidence: 90%

Toward harnessing user feedback for machine learning

Stumpf

Rajaram

et al. 2007

Proceedings of the 12th International Conference on Intelligent User Interfaces

View full text Add to dashboard Cite

show abstract

“…In the early decades of computing, a common saying was "garbage in, garbage out." That is, mistakes in recollection of information were aberrations, and if knowledge discovery tasks have bad data (garbage in), then they should expect incorrect answers (garbage out) [31]. For this reason we proposed a conceptual framework for data quality in knowledge discovery task based on CRISP-DM, SEMMA and Data Science Area, which tackle the issues in data quality clearly through ESE taxonomy.…”

Section: Conclusion and Future Researchmentioning

confidence: 99%

A Conceptual Framework for Data Quality in Knowledge Discovery Tasks (FDQ-KDT): A Proposal

Corrales¹,

Ledezma²,

Corrales³

2015

JCP

View full text Add to dashboard Cite

Large Volume of Data is growing because the organizations are continuously capturing the collective amount of data for better decision-making process. The most fundamental challenge is to explore the large volumes of data and extract useful knowledge for future actions through data mining and data science methodologies. Nevertheless these not tackle the issues in data quality clearly, leaving out relevant activities. We proposed a conceptual framework for data quality in knowledge discovery tasks based on CRISP-DM, SEMMA and Data Science, considering the issues of ESE Taxonomy.

show abstract

“…We compute the visual effectiveness (VE) of the feedback in terms of the set of formula cells with correct and incorrect formulas, and the degree of shading [86]:…”

Section: A1 Fault Localizationmentioning

confidence: 99%

“…In our evaluation of the combined system, we modeled user behavior based on data from prior empirical work [85], [86]. It has been observed that users mark 85 percent of the cells while testing and debugging spreadsheets, placing p marks more frequently than x marks.…”

Section: A2 Experiments Setupmentioning

confidence: 99%

Mutation Operators for Spreadsheets

Abraham

Erwig

2009

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Abstract-Based on 1) research into mutation testing for general-purpose programming languages and 2) spreadsheet errors that have been reported in the literature, we have developed a suite of mutation operators for spreadsheets. We present an evaluation of the mutation adequacy of definition-use adequate test suites generated by a constraint-based automatic test-case generation system we have developed in previous work. The results of the evaluation suggest additional constraints that can be incorporated into the system to target mutation adequacy. In addition to being useful in mutation testing of spreadsheets, the operators can be used in the evaluation of error-detection tools and also for seeding spreadsheets with errors for empirical studies. We describe two case studies where the suite of mutation operators helped us carry out such empirical evaluations. The main contribution of this paper is a suite of mutation operators for spreadsheets that can be used for performing empirical evaluations of spreadsheet tools to indicate ways in which the tools can be improved.

show abstract

Garbage in, Garbage out? An Empirical Look at Oracle Mistakes by End-User Programmers

Cited by 20 publications

References 18 publications

Toward harnessing user feedback for machine learning

Toward harnessing user feedback for machine learning

A Conceptual Framework for Data Quality in Knowledge Discovery Tasks (FDQ-KDT): A Proposal

Mutation Operators for Spreadsheets

Contact Info

Product

Resources

About