Improving metadata infrastructure for complex surveys: 
Insights from the Fragile Families Challenge

Abstract

Researchers rely on metadata systems to prepare data for analysis. As the complexity of datasets increases and the breadth of data analysis practices grow, existing metadata systems can limit the efficiency and quality of data preparation. This article describes the redesign of a metadata system supporting the Fragile Families and Child Wellbeing Study based on the experiences of participants in the Fragile Families Challenge. We demonstrate how treating metadata as data-that is, releasing comprehensive information about variables in a format amenable to both automated and manual processing-can make the task of data preparation less arduous and less error-prone for all types of data analysis. We hope that our work will facilitate new applications of machine learning methods to longitudinal surveys and inspire research on data preparation in the social sciences. We have open-sourced the tools we created so that others can use and improve them.Keywords metadata, survey research, data sharing, quantitative methodology, computational social science 1 We thank all the participants in the Fragile Families Challenge who shared their data processing scripts and stories with us, particularly Greg Gundersen for creating the first version of machine-actionable Fragile Families metadata. We also thank Ian Lundberg who hosted several of the getting started workshops, supported participants throughout the Challenge, and shared his own experiences with survey metadata. Participants in the Princeton Sociology Proseminar provided valuable feedback on a draft of the article, and Brandon Stewart provided catalytic conversation. We thank Ian Fellows for his help with the R package, Greg Gundersen for his help with the Python package, and Cambria Naslund for her assistance with the question text data. Finally, we gratefully acknowledge

Explanation, prediction, and causality: Three sides of the same coin?

“…Social scientists could adapt the "Common Task Framework" that is popular in AI and machine learning communities (Donoho 2017) . An example of the CTF applied to a social science problem is the ("Fragile Families Challenge" n.d.) (FFC), a Netflix-style "prize contest" was based on a large, longitudinal data set (the Fragile Families and Child Wellbeing Study ) that has followed a cohort of nearly 5,000 children from birth through adolescence (Kindel et al 2018). The "challenge" part comprised an open solicitation to researchers of any discipline and seniority to submit predictive models of individual-level outcomes (e.g.…”

“…Over 400 participants, ranging from undergraduate machine learning students to full professors of sociology, economics, and political science submitted entries, either as individuals or in teams, over the course of several months. Although in principle this model of mass collaboration could be applied to any longitudinal social dataset, the nature of social data encounters complications such as protection of data-subject privacy and autonomy (Lundberg et al 2018) and absence of machine readable metadata that are less problematic for more traditional AI applications (Kindel et al 2018). Another challenge is how to incentive large numbers of researchers to collaborate on a single problem, although recent large-scale collaborations in psychology ( https://osf.io/wx7ck/ , https://psysciacc.org/ ) are encouraging.…”

Accelerating progress in the social sciences: the potential of benchmarks

“…One of the main strengths of the challenge is its diverse and multidisciplinary nature: the organizers attracted participants from various disciplines and they managed to combine social science and data science approaches. The challenge was also exceptionally powerful in identifying infrastructural constraints to the implementation of social science benchmarks, such as machine-actionable metadata (Kindel et al, 2019) and the collection and storage of reproducible code using existing social science workflows (Liu and Salganik, 2019). This contribution alone has led to a plethora of new studies that looked to extend, build upon, and improve the application of benchmarks in the social sciences.…”