Getting more out of programming-by-demonstration

International Journal of Human-Computer Studies

Rajaram

et al. 2009

155

124

Although machine learning is becoming commonly used in today's software, there has been little research into how end users might interact with machine learning systems, beyond communicating simple "right/wrong" judgments. If the users themselves could work hand-in-hand with machine learning systems, the users' understanding and trust of the system could improve and the accuracy of learning systems could be improved as well. We conducted three experiments to understand the potential for rich interactions between users and machine learning systems. The first experiment was a think-aloud study that investigated users' willingness to interact with machine learning reasoning, and what kinds of feedback users might give to machine learning systems. We then investigated the viability of introducing such feedback into machine learning systems, specifically, how to incorporate some of these types of user feedback into machine learning systems, and what their impact was on the accuracy of the system. Taken together, the results of our experiments show that supporting rich interactions between users and machine learning systems is feasible for both user and machine. This shows the potential of rich humancomputer collaboration via on-the-spot interactions as a promising direction for machine learning systems and users to collaboratively share intelligence.

Section: Incorporating User Feedbackmentioning

confidence: 99%

Interacting meaningfully with machine learning systems: Three experiments

International Journal of Human-Computer Studies

Rajaram

et al. 2009

155

124

“…A large body of work employing user feedback falls under the topic of programming by demonstration, which enables a system to learn a procedural program interactively from user demonstrations [12,20,23]. For example, the process might be ordering an item from an on-line web-site.…”

Section: Email Programmentioning

confidence: 99%

Integrating rich user feedback into intelligent user interfaces

Proceedings of the 13th International Conference on Intelligent User Interfaces

Sullivan

Fitzhenry

et al. 2008

The potential for machine learning systems to improve via a mutually beneficial exchange of information with users has yet to be explored in much detail. Previously, we found that users were willing to provide a generous amount of rich feedback to machine learning systems, and that the types of some of this rich feedback seem promising for assimilation by machine learning algorithms. Following up on those findings, we ran an experiment to assess the viability of incorporating real-time keyword-based feedback in initial training phases when data is limited. We found that rich feedback improved accuracy but an initial unstable period often caused large fluctuations in classifier behavior. Participants were able to give feedback by relying heavily on system communication in order to respond to changes. The results show that in order to benefit from the user's knowledge, machine learning systems must be able to absorb keyword-based rich feedback in a graceful manner and provide clear explanations of their predictions.

“…When debugging these kinds of programs, end-user corrections are limited to the addition or removal of training dataunless the user reverts to a traditional programming language such as Lisp (e.g., [Vander Zanden and Myers 1995]). For example, Gamut allows users to "nudge" the system when it makes a mistake, leading to the addition or deletion of training examples [McDaniel and Myers 1999]. Recent work with PBD systems allows some debugging of programs [Chen and Weld 2008], but again, their technique only allows the user to retract actions in a demonstration, which results in adding missing values to the training data rather than directly modifying the classifier's logic.…”

Section: Communicating With Machine Learning Systems **D275mentioning

confidence: 99%

Why-oriented end-user debugging of naive Bayes text classification

Kulesza

ACM Trans. Interact. Intell. Syst.

Wong

et al. 2011

This is the accepted version of the paper.This version of the publication may differ from the final published version. **Vocab: "data" is plural. So, need "data are" instead of "data is". **Vocab: It's a Why-oriented approach (capitalized) **Vocab: It's naïve Bayes (lower-case 'n') **Vocab/numbering: change the following globally: Permanent repository linkPrinciple (iv) -> Principle ML-2 **Length: TiiS suggests manuscripts be between 10,000 and 15,000 words. Machine learning techniques are increasingly used in intelligent assistants, software targeted at and continuously adapting to assisting end users with email, shopping, and other tasks. Examples include desktop SPAM filters, recommender systems, and handwriting recognition. Fixing such intelligent assistants when they learn incorrect behavior, however, has received only limited attention. To directly support end-user "debugging" of assistant behaviors learned via statistical machine learning, we present a Why-oriented approach that allows users to ask questions about how the assistant made its predictions, provides answers to these "why" questions, and allows users to interactively change these answers to "debug" these assistants' current and future predictions. To understand the strengths and weaknesses of the approach, we conducted an exploratory study to investigate barriers that participants would encounter when debugging an intelligent assistant using our approach, and the information those participants requested to overcome these barriers. To help ensure the inclusiveness of our investigation, we also explored how gender differences played a role in barriers and information needs. We then use these results to consider opportunities for Why-oriented approaches to address the users' barriers and information needs.