Open Science Perspectives on Machine Learning for the Identification of Careless Responding: A New Hope or Phantom Menace?

Abstract

While machine learning has gained substantial attention and popularity in many scientific fields, it is largely unexplored for the detection of careless responding in survey data. On the one hand, machine learning algorithms can be highly powerful tools due to their flexibility. On the other hand, science based on machine learning has been criticized in the literature for a lack of reproducibility. We assess the potential and the pitfalls of machine learning approaches for identifying careless respondents from an open science perspective. In particular, we discuss possible sources of reproducibility issues when applying machine learning in the context of careless responding, and we give practical guidelines on how to avoid them. Furthermore, we illustrate the high potential of machine learning for the identification of careless respondents in a proof-of-concept simulation experiment. In a longer-term perspective, we elaborate on how open science practices such as sharing code and data may elevate this potential to an even higher level in the future by establishing benchmark data sets in which respondents are labeled with respect to carelessness.