When Knowledge Work and Analytical Technologies Collide: The Practices and Consequences of Black Boxing Algorithmic Technologies

Abstract

Analytical technologies that structure and process data hold great promise for organizations but also may pose fundamental challenges for how knowledge workers accomplish tasks. Knowledge workers are generally considered experts who develop deep understanding of their tools, but recent observations suggest that in some situations, they may black box their analytical technologies, meaning they trust their tools without understanding how they work. I conducted a two-year inductive ethnographic study of the use of analytical technologies across four groups in an investment bank and found two distinct paths that these groups used to validate financial analyses through what I call “validating practices”: actions that confirm whether a produced analysis is trustworthy. Surprisingly, engaging in these practices does not necessarily equate to understanding the calculations performed by the technologies. In one path, validating practices are partitioned across junior and senior roles: junior bankers engage in assembling tasks and use the analytical tools to perform analysis, while only senior bankers interpret the analysis. In the other path, junior and senior members engage in co-construction: junior bankers do both assembling and interpreting tasks, and senior bankers engage in interpreting and provide feedback on junior bankers’ reasoning and choices. Both junior and senior bankers in the partitioning groups routinely black boxed the algorithms embedded in their technologies, taking them for granted without understanding them. By contrast, bankers in the co-construction groups were conscious of the algorithms and understood their potential impact. I found that black boxing influenced the knowledge outputs of these bankers and constrained the development of junior members’ expertise, with consequences for their career trajectories.

Understanding and shaping the future of work with self-determination theory

“…This encourages workers to either overwork to the point of exhaustion, find ways to game the system 184 , or misbehave 186 . Moreover, the technical complexity and opacity of algorithmic systems 187 – 189 deprives workers of the ability to understand and master the system that governs their work, which limits their voice and enpowerment 172 , 185 , 190 . Workers’ typical response to the lack of transparency is to organize themselves on social media to share any insights they have on what the algorithm ‘wants’ as a way to gain back some control over their work 183 , 191 .…”

Explainable machine learning practices: opening another black box for reliable medical AI

“…Selection of the dataset for unsupervised ML can be particularly important, as that may capture implicit bias [9]. 2 Please note that we are not formulating a dilemma. Addressing reliability for regulatory purposes does not exclude addressing issues of trustworthiness caused by algorithmic opacity.…”

“…We base the identification of values on our professional and research experience, but we acknowledge that introspection is not a substitute for a real methodology, at least not long term. Finally, our characterization of the interplay between values and technical choices is an idealization, especially because we have represented the data-aware machine learning pipeline as a process where one person takes all decisions; however, the practice of ML is a social practice, involving different actors and stakeholders [2,26], and this can mean that there will be negotiations of values among different individuals. But these limitations can be overcome in future works.…”

“…We formulate this proposal by developing US Food and Drugs Administration's (FDA) reflections [14] on the regulation of AI tools in medicine into a framework based on works in philosophy of technology and philosophy of science. 2 The structure of the paper is as follows. In Sect.…”

Organisational learning in complex epistemic environments: reflections from studies of professional work in Norway

“…One source of change that generates increased epistemic complexity in expert cultures relates to the ongoing digitalisation processes that permeate social and professional life. A range of scholars have focused on the implications of technology use for professional work and learning, showing how the performance of expertise is formed in relation to the ongoing use of technologies (Anthony, 2021; Mäkitalo and Reit, 2014; Pachidi et al , 2020). When technologies change, so do the ways of knowing and doing work.…”