A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithms

Abstract

Recent research increasingly brings to question the appropriateness of using predictive tools in complex, real-world tasks. While a growing body of work has explored ways to improve value alignment in these tools, comparatively less work has centered concerns around the fundamental justifiability of using these tools. This work seeks to center validity considerations in deliberations around whether and how to build data-driven algorithms in high-stakes domains. Toward this end, we translate key concepts from validity theory to predictive algorithms. We apply the lens of validity to re-examine common challenges in problem formulation and data issues that jeopardize the justifiability of using predictive algorithms and connect these challenges to the social science discourse around validity. Our interdisciplinary exposition clarifies how these concepts apply to algorithmic decision making contexts. We demonstrate how these validity considerations could distill into a series of high level questions intended to promote and document reflections on the legitimacy of the predictive task and the suitability of the data.

"This is not a data problem": Algorithms and Power in Public Higher Education in Canada

“…Within this work, researchers have raised significant concerns about the potential for these algorithmic systems to exacerbate existing social issues [35]. Recent research in HCI has explored, for example, the impact of outcome measurement error as raised by Guerdan et al [28], issues of data quality in machine learning by Jarrahi et al [31], how definitions of risk are formalized within algorithmic decision-making systems by Saxena et al [54], and issues of validity surrounding the use of predictive tools in complex, real-world decision-making [17].…”

“…There is well-documented risk from over-reliance on algorithms in prior Education research [48,62]; Thomas et al describe it aptly: "when a measure becomes the target, it ceases to be an effective measure. " Meanwhile, Coston et al [17] point to the issue of achieving validity in designing algorithms for high-stakes decision-making. While the EAS may appear accurate in respect to identifying those students with a higher probability of failing, it may be inaccurate in identifying those students who would most benefit from the college's limited resources (RQ1).…”

“…The academic and student experience divisions have pushed back in large part by decentralizing their data processes from the data and research office and into their own teams (P23, 7)(RQ2). This creates additional tensions and disparate narratives as siloed teams develop their own metrics and data-practices for measuring the same phenomena, further increasing the challenge of ensuring validity in data-driven and algorithmic systems [17,31].…”