How to Conduct Rigorous Supervised Machine Learning in Information Systems Research: The Supervised Machine Learning Report Card

Abstract

This is a PDF file of an unedited manuscript that has been accepted for publication in the Communications of the Association for Information Systems. We are providing this early version of the manuscript to allow for expedited dissemination to interested readers. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered, which could affect the content. All legal disclaimers that apply to the Communications of the Association for Information Systems pertain.

Artificial intelligence and machine learning

“…Interestingly, as of today, many AI-based information systems remain static, i.e. employ once-trained ML models (Kühl et al, 2021). With increasing focus on deployment and life cycle management, we will see more adaptive AI-based systems that sense changes in the environment and use ML to learn continuously (Baier et al, 2019).…”

Machine Learning in Lithium‐Ion Battery Cell Production: A Comprehensive Mapping Study

“…[95] Some major software companies, such as Google [96] or Microsoft, [97] have synthesized best practices in the large-scale development of data-driven applications. From the information system research community, Kühl et al [98] confirmed the same challenge. Even with full access to data, the researchers cannot evaluate or replicate the results due to the inconsistency in the documentation and reporting of ML studies.…”

“…Some major software companies, such as Google [96] or Microsoft, [97] have synthesized best practices in the large‐scale development of data‐driven applications. From the information system research community, Kühl et al [98] . confirmed the same challenge.…”

“…Such studies highlight the necessity of close cooperation between ML and domain, i. e., production experts. [98] The above-mentioned inconsistencies in reporting have also been observed during the conducted mapping study, leading to a certain degree of uncertainty. For example, a key aspect that needs to be reported is whether a data point is a replication or a unique run.…”

“…Even with full access to data, the researchers cannot evaluate or replicate the results due to the inconsistency in the documentation and reporting of ML studies. [98] They proposed a "Machine Learning Report Card" as a solution to capture critical decisions and problem characteristics in the development of the ML model and to guide researchers toward rigorous, comprehensive analysis and extension. While the article focuses on supervised ML, the findings can also be adapted for unsupervised ML.…”