Mapping the Iceberg: The Impact of Data Sources on the Study of District Courts

Abstract

Three decades ago, Siegelman and Donohue aptly characterized research about courts and litigation that relied only on published opinions as “studying the iceberg from its tip.” They implored researchers to view published district court opinions “with greater sensitivity to the ways in which such cases are unrepresentative of all cases”. The dynamic, multistage nature of trial court litigation makes a focus solely on published opinions particularly ill‐suited to the study of federal district courts. Expanded electronic access to court documents now allows more precise analysis of the ways in which published cases are unrepresentative and what differences that makes for conclusions about the work of district courts. Heeding Siegelman and Donohue's admonition, this study seeks to map the iceberg, exploring the extent to which the visible part misrepresents what lies below the surface. Using a supplemented version of the Kim, Schlanger, and Martin EEOC Litigation Project data, this article examines the varying extent to which cases and judicial activity are visible in the several data sources commonly used by district court researchers. More specifically, we analyze how the work of federal district courts looks different depending on whether research relies on published opinions, on opinions available on Westlaw or Lexis (both “published” and “unpublished”), or on more comprehensive data available on PACER (Public Access to Court Electronic Documents). Our results reveal vast variation in visibility of cases and motions, depending on the data source used. We also demonstrate that these differences in case and motion visibility can affect the results of empirical analyses relating to, for example, the success rates of litigants and whether the party of the appointing president affects judicial behavior. Our findings mean that utilizing docket sheets, now available electronically, to gather data will often be required to draw accurate conclusions about the nature of district court litigation and the behavior of district court judges.

Ideology, Disadvantage, and Federal District Court Inmate Civil Rights Filings: The Troubling Effects of Pro Se Status

“…Finally, scholars may also be considering different outcomes of interest , with some studying published opinions and others investigating unpublished opinions or all court decisions, regardless of whether an opinion was published. Some scholars use published opinions to interrogate the effect of ideology on case outcomes (e.g., Sunstein et al 2006), but there is reason to believe published opinions are not representative of all judicial opinions (and, looking at only published opinions introduces posttreatment bias into estimates of the effect of judicial ideology on case outcomes; Boyd et al 2020). Some scholars suggest ideology may factor into decisions that result in published opinions , but not in those cases that do not produce an opinion (Keele et al 2009).…”

“…The primary challenge in estimating the effect of a characteristic like judicial ideology or pro se status on case outcomes in the district courts is one of data availability. Though some scholars study published opinions of district courts (e.g., Sunstein et al 2006), that approach is incomplete as it does not consider all decisions levied by the courts and can result in posttreatment bias in the estimates of the effect of variables like ideology and pro se status on case outcomes (Kim et al 2009; Hubert & Copus 2019; Boyd et al 2020). This is especially problematic in the inmate lawsuit context, as these kinds of cases are much less likely to result in published opinions (Olson 1992; Feierman 2006; Hoffman et al 2007).…”

Ideological Determinants of Citations to Supreme Court Precedent Across the Federal Judiciary

“…How frequently an opinion is invoked in the lower courts, where the vast majority of litigation occurs, is a key measure of precedential impact (Hitt 2016). Due to the volume of cases heard at lower levels of the hierarchy, there is the greatest potential for variation in the rate of citation by district courts (Boyd, Kim, and Schlanger 2020). Thus, citations to Supreme Court precedent by district courts contain a wealth of information on the applicability of precedent.…”

Scientific Disciplines and the Admissibility of Expert Evidence in Courts

“…However, not all decisions are reported. E-filing has substantially improved the coverage of conventional databases, yet they may contain only the “tip of the iceberg” of expert admissibility decisions (Boyd, Kim, and Schlanger 2020). In addition to all reported admissibility decisions, Daubert Tracker contains many unreported admissibility decisions gathered from docket sheets, litigation reports, jury verdict reports, court Web sites, and other sources.…”