Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models

Yang, Qian; Suh, Jina; Chen, Nan-Chen; Ramos, Gonzalo

doi:10.1145/3196709.3196729

Cited by 112 publications

(83 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…During the workshop, we observed that the knowledge gap due to the disability and the lack of technological understandings were gradually bridged. In terms of technological understandings, it can be seen with the fact that there were no participants describing non-ML tasks in worksheet E. Our findings that non-expert people perceive ML technology from the perspective of the necessity of diversity of data supports Yang et al [39] that mentioned the tendency of non-expert people to add more data to improve ML model performance. In terms of disability and sound perception, some participants commented that they came to know how to use the sounds to train their recognition systems.…”

Section: Bridging the Knowledge Gapsupporting

confidence: 76%

“…Several prior studies have already pointed out that the technological knowledge gap is one of the key challenges to democratize ML technologies in general. Non-expert people often fail to have appropriate mental models for ML-based tools [23,36], or to cover sufficient factors when building or evaluating ML models [39]. For DHH people, the knowledge gap becomes even more critical because of their disabilities.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Use of Machine Learning by Non-Expert DHH People: Technological Understanding and Sound Perception

Nakao

Sugano

2020

Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society

View full text Add to dashboard Cite

Recent advances in machine learning demonstrated its potential in accessibility applications. However, recognition models and their application scenarios are often defined by machine learning (ML) experts and cannot fully capture users' diverse demands with disabilities. In order to open up the full potential of ML for accessibility applications, we have to bridge the gap for non-expert people doubly caused by the technological understanding and their disabilities. In this work, we investigate how non-expert deaf and hard-of-hearing (DHH) people understand ML technologies and design ML-based sound recognition systems. We conduct a workshop study consisting of an ML lecture and an interactive learning session using a sound recognition system. Through observations during the workshop and semi-structured interviews, we clarify that non-expert DHH people start to overcome the knowledge gap. They could obtain a more detailed understanding of ML technology and how to use sounds to train ML models. CCS CONCEPTS • Social and professional topics → People with disabilities; • Computing methodologies → Machine learning.

show abstract

Section: Bridging the Knowledge Gapsupporting

confidence: 76%

mentioning

confidence: 99%

Use of Machine Learning by Non-Expert DHH People: Technological Understanding and Sound Perception

Nakao

Sugano

2020

Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society

View full text Add to dashboard Cite

show abstract

“…Non-experts have long been able to participate in data collection to train predictive models and interactive ML allows non-experts and ML to work together to better address pre-identified problems (Fails & Olsen, 2003;Cheng & Bernstein, 2015;Crandall et al, 2018). However, as remarked by Yang et al (2018), despite extensive research in these areas, little work has investigated how non-experts can take creative or editorial control to design their own applications of ML.…”

Section: Introductionmentioning

confidence: 99%

“…Two sets of studies are perhaps most closely related to the research here. The first focuses on non-experts who built ML tools that enable non-experts to build ML models, and investigated their goals, methods, and the challenges they encountered Yang et al (2018). Yang et al (2018) performed an empirical study, conducting interviews and surveys of non-experts, and their study serves as an important complement to our work.…”

Section: Introductionmentioning

confidence: 99%

“…The first focuses on non-experts who built ML tools that enable non-experts to build ML models, and investigated their goals, methods, and the challenges they encountered Yang et al (2018). Yang et al (2018) performed an empirical study, conducting interviews and surveys of non-experts, and their study serves as an important complement to our work. To the best of our knowledge, our study here is the first experimental consideration of non-experts and their ability to design prediction tasks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Democratizing AI: non-expert design of prediction tasks

Bagrow

2020

PeerJ Computer Science

View full text Add to dashboard Cite

Non-experts have long made important contributions to machine learning (ML) by contributing training data, and recent work has shown that non-experts can also help with feature engineering by suggesting novel predictive features. However, non-experts have only contributed features to prediction tasks already posed by experienced ML practitioners. Here we study how non-experts can design prediction tasks themselves, what types of tasks non-experts will design, and whether predictive models can be automatically trained on data sourced for their tasks. We use a crowdsourcing platform where non-experts design predictive tasks that are then categorized and ranked by the crowd. Crowdsourced data are collected for top-ranked tasks and predictive models are then trained and evaluated automatically using those data. We show that individuals without ML experience can collectively construct useful datasets and that predictive models can be learned on these datasets, but challenges remain. The prediction tasks designed by non-experts covered a broad range of domains, from politics and current events to health behavior, demographics, and more. Proper instructions are crucial for non-experts, so we also conducted a randomized trial to understand how different instructions may influence the types of prediction tasks being proposed. In general, understanding better how non-experts can contribute to ML can further leverage advances in Automatic machine learning and has important implications as ML continues to drive workplace automation.

show abstract

Towards an AI‐driven business development framework: A multi‐case study

John

Olsson

Bosch

2022

J Software Evolu Process

View full text Add to dashboard Cite

Artificial intelligence (AI) and the use of machine learning (ML) and deep learning (DL) technologies are becoming increasingly popular in companies. These technologies enable companies to leverage big quantities of data to improve system performance and accelerate business development. However, despite the appeal of ML/DL, there is a lack of systematic and structured methods and processes to help data scientists and other company roles and functions to develop, deploy and evolve models. In this paper, based on multi-case study research in six companies, we explore practices and challenges practitioners experience in developing ML/DL models as part of large software-intensive embedded systems. Based on our empirical findings, we derive a conceptual framework in which we identify three high-level activities that companies perform in parallel with the development, deployment and evolution of models. Within this framework, we outline activities, iterations and triggers that optimize model design as well as roles and company functions. In this way, we provide practitioners with a blueprint for effectively integrating ML/DL model development into the business to achieve better results than other (algorithmic) approaches. In addition, we show how this framework helps companies solve the challenges we have identified and discuss checkpoints for terminating the business case.

show abstract

Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models

Cited by 112 publications

References 20 publications

Use of Machine Learning by Non-Expert DHH People: Technological Understanding and Sound Perception

Use of Machine Learning by Non-Expert DHH People: Technological Understanding and Sound Perception

Democratizing AI: non-expert design of prediction tasks

Towards an AI‐driven business development framework: A multi‐case study

Contact Info

Product

Resources

About