Human-AI Collaboration in Data Science

Abstract

The rapid advancement of artificial intelligence (AI) is changing our lives in many ways. One application domain is data science. New techniques in automating the creation of AI, known as AutoAI or AutoML, aim to automate the work practices of data scientists. AutoAI systems are capable of autonomously ingesting and pre-processing data, engineering new features, and creating and scoring models based on a target objectives (e.g. accuracy or run-time efficiency). Though not yet widely adopted, we are interested in understanding how AutoAI will impact the practice of data science. We conducted interviews with 20 data scientists who work at a large, multinational technology company and practice data science in various business settings. Our goal is to understand their current work practices and how these practices might change with AutoAI. Reactions were mixed: while informants expressed concerns about the trend of automating their jobs, they also strongly felt it was inevitable. Despite these concerns, they remained optimistic about their future job security due to a view that the future of data science work will be a collaboration between humans and AI systems, in which both automation and human expertise are indispensable.

Visualising data science workflows to support third-party notebook comprehension: an empirical study

“…Examples include writing and structuring code blocks differently or declaring variables that do not follow the standard naming conventions. However, not all data scientists are always familiar with standard coding conventions (Kim et al 2017 ; Wang et al 2019 ).…”

Out of Context: Investigating the Bias and Fairness Concerns of "Artificial Intelligence as a Service"

“…The AutoML paradigm of ML development will often (almost by definition) be unable to bring into account all the requisite factors, perspectives and constraints (which often require some domain expertise) that would otherwise need to be reconciled in addressing issues of fairness. Today's AutoML systems tend to limit customers' interaction with the system to a few specific decision points and present only limited to no information on how these systems work in operation and the complex processes behind the ML models generation and selection [86,[146][147][148]153]. This 'blackbox' nature of AutoML operation results in a situation where users will often be unable to understand how and why AutoML systems make the choices they make [148], thus hindering their ability to effectively reason about and mitigate potential biases embedded in their outputs.…”

Are Two Heads Better Than One?: The Effect of Student-AI Collaboration on Students' Learning Task Performance

“…But it was greedy that it wanted to control over all my drawing." Their view indicates that although AI's automation contributes to the task performance enhancement, it should reflect levels of students' domain-specific skills and students' agency (Wang et al, 2019).…”