A modeler's manifesto: Synthesizing modeling best practices with social science frameworks to support critical approaches to data science

Abstract

In the face of the "crisis of reproducibility" and the rise of "big data" with its associated issues, modeling needs to be practiced more critically and less automatically. Many modelers are discussing better modeling practices, but to address questions about the transparency, equity, and relevance of modeling, we also need the theoretical grounding of social science and the tools of critical theory. I have therefore synthesized recent work by modelers on better practices for modeling with social science literature (especially feminist science and technology studies) to offer a "modeler’s manifesto": a set of applied practices and framings for critical modeling approaches. Broadly, these practices involve 1) giving greater context to scientific modeling through extended methods sections, appendices, and companion articles, clarifying quantitative and qualitative reasoning and process; 2) greater collaboration in scientific modeling via triangulation with different data sources, gaining feedback from interdisciplinary teams, and viewing uncertainty as openness and invitation for dialogue; and 3) directly engaging with justice and ethics by watching for and mitigating unequal power dynamics in projects, facing the impacts and implications of the work throughout the process rather than only afterwards, and seeking opportunities to collaborate directly with people impacted by the modeling.

Ten simple rules for building and maintaining a responsible data science workflow

“…Logistics constraints can make responsible collaboration challenging as teams understandably are pressured to do more with less (time and/or money). For example, Eitzel [ 105 ] documents how a member of an interdisciplinary working group decided to stop attending group meetings because of a mix of logistical burdens (heavy teaching load and a conference deadline) and personal burdens (always being called upon to teach the group about the basics of their discipline while simultaneously having the discipline being undervalued by the group members from another discipline). Eitzel [ 105 ] perceived this group member’s missing contributions to the discussions after leaving to be a huge loss for the group’s overall understanding and progress.…”

“…For example, Eitzel [ 105 ] documents how a member of an interdisciplinary working group decided to stop attending group meetings because of a mix of logistical burdens (heavy teaching load and a conference deadline) and personal burdens (always being called upon to teach the group about the basics of their discipline while simultaneously having the discipline being undervalued by the group members from another discipline). Eitzel [ 105 ] perceived this group member’s missing contributions to the discussions after leaving to be a huge loss for the group’s overall understanding and progress. Valuing the group member’s expertise both in attitude and compensation would have been helpful in maintaining their contributions to the overall group.…”

“…Finding a framework for prompting reflection that works for a team can be challenging. Eitzel [ 105 , 113 ] presents one such framework for autoethnographic assessment of best practices. Eitzel [ 105 , 113 ] provides an example of working through the process herself on her own modeling best practices that includes social science considerations as well as more traditional modeling practices.…”

“…Eitzel [ 105 , 113 ] presents one such framework for autoethnographic assessment of best practices. Eitzel [ 105 , 113 ] provides an example of working through the process herself on her own modeling best practices that includes social science considerations as well as more traditional modeling practices. This process helped Eitzel not only grapple with issues of technical reproducibility (by transparently providing errata to a previously published paper when a mistake was found in the analysis) but also helped improve her modeling practices by working to engage with community stakeholders.…”