The Keys to Unlocking Public Payments Data

Abstract

Summary We mechanize some of the richest yet significantly under‐utilized data resources within developed, ‘Open Data' economies. We show how it is possible to scrape, parse, clean and merge tens of thousands of disaggregated public payments datasets in an attempt to bridge the methodological gap between newly available data from the administrative sphere and applications in empirical social science research. We outline techniques to unambiguously link records to various freely available institutional registers. In particular, we offer guidance on overcoming the substantial challenges of heterogeneous provision and administrative recording errors in the absence of Uniform Resource Identifiers, namely in the form of an approximate, domain‐specific ‘record‐linkage' type matching algorithm. As an illuminating example, we construct a cleaned database of 24,581,192 local government payments subject to the Local Transparency Codes which total £169.87bn in value. We overcome various challenges in a detailed examination of the procurement of services by local government from the voluntary sector: an important contemporary issue due to the rise of the ‘Big Society’ political ideology of the early 21st century. Finally, we motivate future work in this area and discuss potential international applications and practical advancements.

Tools for Transparency in Central Government Spending

“…The limitations of the data are alluded to above and are discussed elsewhere [3,9]. However, we surmise and order in importance what we view as the most pressing issues, and provide corresponding recommendations below:…”

“…Various challenges are responsible for a lack of social science literature which utilizes granular public payments despite pioneering efforts by social enterprises, third sector entities (such as the National Council of Voluntary Organisations) and Non-Government Organizations (such as OpenCorporates, Spend Network and the Open Contracting Partnership). Rahal [9] outlines the methodological tools required to map payments from over 300 local authorities to multiple registers, as mandated by the Local Authority Transparency Code, and the Institute for Government [3] uses data from the Spend Network and others to provide a comprehensive description of what is procured, and who from. The most methodologically similar paper to ours [10] develops the Company, Organization Firm name Unifier (CORFU) approach, using it to approximately string match a procurement dataset from Australia from between 2004-2012, with the main difference being that our external reconciliation service normalizes, cleans 'stopwords' and expands acronyms on our behalf (steps 1-3 of CORFU), as discussed below.…”

Tools for Transparency in Central Government Spending

“…The limitations of the data are alluded to above and are discussed elsewhere [2,8]. However, we surmise and order in importance what we view as the most pressing issues, and provide corresponding recommendations below:…”

“…Various challenges are responsible for a lack of social science literature which utilizes granular public payments despite pioneering efforts by social enterprises, third sector entities (such as the National Council of Voluntary Organisations) and Non-Government Organizations (such as OpenCorporates, Spend Network and the Open Contracting Partnership). Rahal [8] outlines the methodological tools required to map payments from over 300 local authorities to multiple registers, as mandated by the Local Authority Transparency Code, and the Institute for Government [2] uses data from the Spend Network and others to provide a comprehensive description of what is procured, and who from. The most methodologically similar paper to ours [9] develops Company, Organization Firm name Unifier (CORFU) approach, using it to approximately string match a procurement dataset from Australia from between 2004-2012, with the main difference being that our external reconciliation service normalizes, cleans 'stopwords' and expands acronyms on our behalf (steps 1-3 of CORFU), as discussed below.…”

Nowcashing: Using Daily Fiscal Data for Real-Time Macroeconomic Analysis

“…Furthermore, industry-level analysis suggests that the increase in investment at the firm level translates into an industry-wide effect without crowding-out capital investment of other firms in the same industry. Rahal (2016), in turn, analyzes disaggregated daily public payments data from the UK, constructing a database of almost 25 million local government payments. With these data, the author examines several types of public spending, such as: (1) which third-sector organizations in the UK receive local government funding;…”