Logics and practices of transparency and opacity in real-world applications of public sector machine learning

Veale, Michael

doi:10.31235/osf.io/6cdhe

Cited by 9 publications

(5 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The increased management of data science systems is a priority [65,96,105], but "the complex decision-making structure" needed to manage them often exceeds "the human and organizational resources available for oversight" [68]. One limiting factor is the opacity of data science systems [97,100] arising from a variety of reasons: (a) algorithms are often trade-secrets, (b) data science requires specialized knowledge, and (c) novel analytic methods remain conceptually challenging [14]. Neural networks are often critiqued for their black-boxed nature, 3 but researchers argue that even simpler models are not necessarily more interpretable than their complex counterparts [64].…”

Section: Trust Objectivity and Justificationmentioning

confidence: 99%

Trust in Data Science

Passi

Jackson

2018

Proc. ACM Hum.-Comput. Interact.

130

View full text Add to dashboard Cite

The trustworthiness of data science systems in applied and real-world settings emerges from the resolution of specific tensions through situated, pragmatic, and ongoing forms of work. Drawing on research in CSCW, critical data studies, and history and sociology of science, and six months of immersive ethnographic fieldwork with a corporate data science team, we describe four common tensions in applied data science work: (un)equivocal numbers, (counter)intuitive knowledge, (in)credible data, and (in)scrutable models. We show how organizational actors establish and re-negotiate trust under messy and uncertain analytic conditions through practices of skepticism, assessment, and credibility. Highlighting the collaborative and heterogeneous nature of real-world data science, we show how the management of trust in applied corporate data science settings depends not only on pre-processing and quantification, but also on negotiation and translation. We conclude by discussing the implications of our findings for data science research and practice, both within and beyond CSCW.

show abstract

Section: Trust Objectivity and Justificationmentioning

confidence: 99%

Trust in Data Science

Passi

Jackson

2018

Proc. ACM Hum.-Comput. Interact.

130

View full text Add to dashboard Cite

show abstract

“…Updates to accounting or audit standards, in both the public and private sectors, would make the assessment of traceability substantially more straightforward. Further, investigating the effectiveness of such standards in furthering the goals of traceability (as measured by, say, perspectives on the performance of a system [30,238]) would provide useful benchmarks for those charged with the test and evaluation of practical systems. Understanding the operationalization of this principle, seemingly amenable to testing and assessment more than other, contested principles dealing with fairness, bias, and equity, would demonstrate a path to operationalizing ethical principles in the development of practical systems, and is thus of paramount importance.…”

Section: Adoptionmentioning

confidence: 99%

Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems

Kroll

2021

Preprint

View full text Add to dashboard Cite

Accountability is widely understood as a goal for well governed computer systems, and is a sought-after value in many governance contexts. But how can it be achieved? Recent work on standards for governable artificial intelligence systems offers a related principle: traceability. Traceability requires establishing not only how a system worked but how it was created and for what purpose, in a way that explains why a system has particular dynamics or behaviors. It connects records of how the system was constructed and what the system did mechanically to the broader goals of governance, in a way that highlights human understanding of that mechanical operation and the decision processes underlying it. We examine the various ways in which the principle of traceability has been articulated in AI principles and other policy documents from around the world, distill from these a set of requirements on software systems driven by the principle, and systematize the technologies available to meet those requirements. From our map of requirements to supporting tools, techniques, and procedures, we identify gaps and needs separating what traceability requires from the toolbox available for practitioners. This map reframes existing discussions around accountability and transparency, using the principle of traceability to show how, when, and why transparency can be deployed to serve accountability goals and thereby improve the normative fidelity of systems and their development processes. CCS CONCEPTS• Software and its engineering → Traceability; Software version control; • Computer systems organization → Maintainability and maintenance.

show abstract

“…In computer science, transparency has been historically a significant challenge, and it has been addressed in different ways by different disciplines. We can mention here algorithmic transparency, algorithmic accountabily [12,15,27,28,50] and, more recently, interpretable/explainable systems [23]. The common goal of all the above disciplines is that algorithms' actions must be easily understood by users (expert and non-experts) when we execute them in a particular context.…”

Section: Introduction: Ai and Explainabilitymentioning

confidence: 99%

Explainable Hopfield Neural Networks Using an Automatic Video-Generation System

et al. 2021

View full text Add to dashboard Cite

Hopfield Neural Networks (HNNs) are recurrent neural networks used to implement associative memory. They can be applied to pattern recognition, optimization, or image segmentation. However, sometimes it is not easy to provide the users with good explanations about the results obtained with them due to mainly the large number of changes in the state of neurons (and their weights) produced during a problem of machine learning. There are currently limited techniques to visualize, verbalize, or abstract HNNs. This paper outlines how we can construct automatic video-generation systems to explain its execution. This work constitutes a novel approach to obtain explainable artificial intelligence systems in general and HNNs in particular building on the theory of data-to-text systems and software visualization approaches. We present a complete methodology to build these kinds of systems. Software architecture is also designed, implemented, and tested. Technical details about the implementation are also detailed and explained. We apply our approach to creating a complete explainer video about the execution of HNNs on a small recognition problem. Finally, several aspects of the videos generated are evaluated (quality, content, motivation and design/presentation).

show abstract

Logics and practices of transparency and opacity in real-world applications of public sector machine learning

Cited by 9 publications

References 7 publications

Trust in Data Science

Trust in Data Science

Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems

Explainable Hopfield Neural Networks Using an Automatic Video-Generation System

Contact Info

Product

Resources

About