The Problem of Data Bias in the Pool of Published U.S. Appellate Court Opinions

Carlson, Keith Thor; Livermore, Michael A.; Rockmore, Daniel N.

doi:10.1111/jels.12253

Cited by 18 publications

(10 citation statements)

References 60 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…11 We preprocess the text by decomposing it into individual words, and then remove numbers, punctuation, proper names, and a list of commonly used and customized "stop-words." 12 We then rely on Natural Language Processing (NLP) tools to analyze the content of these written reviews (Carlson et al, 2020;Fagan, 2021;7 See Appendix Table A1 for a summary of the information available from each of the five rating websites.…”

Section: Methods: Text Analysis Of Physician Written Reviewsmentioning

confidence: 99%

“…We use text analysis to quantitively analyze the written reviews. We pre‐process the text by decomposing it into individual words, and then remove numbers, punctuation, proper names, and a list of commonly used and customized “stop‐words.” We then rely on Natural Language Processing (NLP) tools to analyze the content of these written reviews (Carlson et al, 2020; Fagan, 2021; Frankenreiter & Livermore, 2020). First, we simply count the most frequently used words in reviews and Questionable and control physicians.…”

Section: Theory Data and Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Should patients use online reviews to pick their doctors and hospitals?

Hyman¹,

Liu

Black

2022

J Empirical Legal Studies

View full text Add to dashboard Cite

We compare the online reviews of 221 “Questionable” Illinois and Indiana physicians with multiple paid medical malpractice claims and disciplinary sanctions with matched control physicians with clean records. Across five prominent online rating services, we find small, mostly insignificant differences in star ratings and written reviews for Questionable versus control physicians. Only one rating service (Healthgrades) reports on paid medical malpractice claims and disciplinary actions and it misses more than 90% of these actions. We also evaluate the online ratings of 171 Illinois hospitals and find that their ratings are largely uncorrelated with the share of hospital‐affiliated physicians with paid medical malpractice claims and disciplinary sanctions. Online ratings have limited utility in helping patients avoid physicians with troubled medical malpractice and disciplinary records, and steering patients away from hospitals at which more physicians have paid medical malpractice claims and disciplinary sanctions.

show abstract

Section: Methods: Text Analysis Of Physician Written Reviewsmentioning

confidence: 99%

Section: Theory Data and Methodsmentioning

confidence: 99%

Should patients use online reviews to pick their doctors and hospitals?

Hyman¹,

Liu

Black

2022

J Empirical Legal Studies

View full text Add to dashboard Cite

show abstract

“…mated effects. For example, recent research on the U.S. Courts of Appeals has uncovered evidence that the characteristics of published opinions appear to be correlated with the partisan make-up of the panels issuing those opinions (Carlson, Livermore, and Rockmore 2020). In the context of district courts, Hübert and Copus (2022) show that subsetting to cases that end in certain ways biases the effect of judge partisanship toward zero, potentially causing scholars to mistakenly conclude that political ideology matters less in district courts than other levels of the federal judiciary.…”

Section: The Case-level All Else Equal Conditionmentioning

confidence: 99%

Trading Diversity? Judicial Diversity and Case Outcomes in Federal Courts

Copus¹,

Hübert²,

Pellaton³

2022

Preprint

View full text Add to dashboard Cite

Does racial and gender diversity on the federal judicial bench affect legal outcomes? Prior studies do not provide a clear answer, in part because they focus on how judges from different backgrounds---after controlling for other judicial characteristics---decide cases differently. In this article, we ask a simpler, but vital, question: do presidents' more diverse appointees, taken as they are, resolve cases differently than those presidents' other appointees? Using an original dataset of around 100,000 civil rights cases filed in 12 federal district courts, we find that the assignment of a case to a judge who is non-White or a woman has no statistically significant effect on case outcomes among Democratic appointees but causes more pro-defendant outcomes among Republican appointees. Our results are consistent with a theory of bargaining over judicial appointments in which Republican presidents take advantage of Democrats' concern over diversity on the bench to appoint more conservative judges.

show abstract

“…There is also evidence that the strength of the effect varies depending on publication status of opinions. (e.g., Carlson et al 2020). For district courts, however, research results examining judge identity are much more mixed (compare, e.g., Rowland & Carp 1996and Boyd 2016to Ashenfelter et al 1995.…”

Section: Conclusion About Judicial Behaviormentioning

confidence: 99%

Mapping the Iceberg: The Impact of Data Sources on the Study of District Courts

Boyd

Kim

Schlanger

2020

J Empirical Legal Studies

View full text Add to dashboard Cite

Three decades ago, Siegelman and Donohue aptly characterized research about courts and litigation that relied only on published opinions as “studying the iceberg from its tip.” They implored researchers to view published district court opinions “with greater sensitivity to the ways in which such cases are unrepresentative of all cases”. The dynamic, multistage nature of trial court litigation makes a focus solely on published opinions particularly ill‐suited to the study of federal district courts. Expanded electronic access to court documents now allows more precise analysis of the ways in which published cases are unrepresentative and what differences that makes for conclusions about the work of district courts. Heeding Siegelman and Donohue's admonition, this study seeks to map the iceberg, exploring the extent to which the visible part misrepresents what lies below the surface. Using a supplemented version of the Kim, Schlanger, and Martin EEOC Litigation Project data, this article examines the varying extent to which cases and judicial activity are visible in the several data sources commonly used by district court researchers. More specifically, we analyze how the work of federal district courts looks different depending on whether research relies on published opinions, on opinions available on Westlaw or Lexis (both “published” and “unpublished”), or on more comprehensive data available on PACER (Public Access to Court Electronic Documents). Our results reveal vast variation in visibility of cases and motions, depending on the data source used. We also demonstrate that these differences in case and motion visibility can affect the results of empirical analyses relating to, for example, the success rates of litigants and whether the party of the appointing president affects judicial behavior. Our findings mean that utilizing docket sheets, now available electronically, to gather data will often be required to draw accurate conclusions about the nature of district court litigation and the behavior of district court judges.

show abstract

The Problem of Data Bias in the Pool of Published U.S. Appellate Court Opinions

Cited by 18 publications

References 60 publications

Should patients use online reviews to pick their doctors and hospitals?

Should patients use online reviews to pick their doctors and hospitals?

Trading Diversity? Judicial Diversity and Case Outcomes in Federal Courts

Mapping the Iceberg: The Impact of Data Sources on the Study of District Courts

Contact Info

Product

Resources

About