Decision Provenance: Harnessing Data Flow for Accountable Systems

Abstract

Demand is growing for more accountability regarding the technological systems that increasingly occupy our world. However, the complexity of many of these systems -often systems-of-systems -poses accountability challenges. A key reason for this is because the details and nature of the information flows that interconnect and drive systems, which often occur across technical and organisational boundaries, tend to be invisible or opaque. This paper argues that data provenance methods show much promise as a technical means for increasing the transparency of these interconnected systems. Specifically, given the concerns regarding ever-increasing levels of automated and algorithmic decision-making, and so-called 'algorithmic systems' in general, we propose decision provenance as a concept showing much promise. Decision provenance entails using provenance methods to provide information exposing decision pipelines: chains of inputs to, the nature of, and the flow-on effects from the decisions and actions taken (at design and run-time) throughout systems. This paper introduces the concept of decision provenance, and takes an interdisciplinary (tech-legal) exploration into its potential for assisting accountability in algorithmic systems. We argue that decision provenance can help facilitate oversight, audit, compliance, risk mitigation, and user empowerment, and we also indicate the implementation considerations and areas for research necessary for realising its vision. More generally, we make the case that considerations of data flow, and systems more broadly, are important to discussions of accountability, and complement the considerable attention already given to algorithmic specifics.

What lies beneath: transparency in online service supply chains

“…As discussed above, applications and infrastructure services are dependent on others in a complex, interconnected, and increasingly consolidated system-of-systems. In such a context, failures by one infrastructural component-whether as a result of emergent behaviours or otherwise-can propagate through components located downstream from the point of failure (Singh, Cobbe, and Norval 2019). This can result in problems affecting the range of other components that rely on that underlying infrastructure.…”

“…From a technical perspective, more transparency would assist in gaining a better understanding of the implications of infrastructure consolidation. Provenance-based approaches to tracking data flows throughout systems may indicate one potential way forward (Singh, Cobbe, and Norval 2019). However, any such solutions need more than just technology; important are the motivations and incentives for application and service providers to make their supply chains more transparent in the first place.…”

“…metadata about data and system interactions)-which show promise in providing greater visibility within complex systems and over systems supply chains. However, despite the potential, much work is required and many research opportunities in the space remain (Singh, Cobbe, and Norval 2019).…”

The Best of Both Worlds: Challenges in Linking Provenance and Explainability in Distributed Machine Learning

“…In fact, there are publications dating back as long as 15 years [14], and again very recently in the context of data science [15], stressing the importance of tracing provenance throughout the data preparation pipeline so that it may be leveraged in succeeding stages. Keeping track of data flows and decisions or actions taken based on data has been identified as key for accountable data-driven interconnected decision systems [16]. This concept was further called decision provenance, as a term for system accountability, by the authors of [16].…”

Monitoring Misuse for Accountable 'Artificial Intelligence as a Service'

“…At the same time, AI is coming under increasing scrutiny [5,30]. As the discussions of 'algorithmic accountability', 'AI regulation' and 'ethical AI' make clear, there are many situations in which the use of AI will be inappropriate, controversial and unlawful [16].…”