Word Embeddings: What Works, What Doesn’t, and How to Tell the Difference for Applied Research

Rodríguez, Pedro L.; Spirling, Arthur

doi:10.1086/715162

Cited by 66 publications

(46 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our sample of mTurkers were drawn from a variety of English-speaking contexts and instructed to carry out the task, focusing on word use in legal and courtroom settings. While we see little reason ex ante that coders from Kenya specifically would employ different intuitive word mappings from coders of other nationalities, future research in text analysis might consider how validation exercises like the one in Rodriguez and Spirling (2022) might vary from context to context to better understand asymmetries between the context of the written language and the nationalities of coders. a new dataset of almost 10,000 criminal cases from the Kenyan High Court.…”

Section: Discussionmentioning

confidence: 94%

“…We provide evidence of a coethnic bonus in appeals decisions using 26 In the Appendix, we illustrate that our main results are consistent using different dictionaries and embedding models. In addition, Appendix F presents the word-embeddings validation exercise described in Rodriguez and Spirling (2022). This shows that human coders from mTurk are unable to distinguish between word relations generated by the word embeddings we employ and human-generated word relationships.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Ethnic Bias in Judicial Decision Making: Evidence from Criminal Appeals in Kenya

2022

View full text Add to dashboard Cite

Understanding sources of judicial bias is essential for establishing due process. To date, theories of judicial decision making are rooted in ranked societies with majority–minority group cleavages, leaving unanswered which groups are more prone to express bias and whether it is motivated by in-group favoritism or out-group hostility. We examine judicial bias in Kenya, a diverse society that features a more complex ethnic landscape. While research in comparative and African politics emphasizes instrumental motivations underpinning ethnic identity, we examine the psychological, implicit biases driving judicial outcomes. Using data from Kenyan criminal appeals and the conditional random assignment of judges to cases, we show that judges are 3 to 5 percentage points more likely to grant coethnic appeals than non-coethnic appeals. To understand mechanisms, we use word embeddings to analyze the sentiment of written judgments. Judges use more trust-related terms writing for coethnics, suggesting that in-group favoritism motivates coethnic bias in this context.

show abstract

Section: Discussionmentioning

confidence: 94%

Section: Discussionmentioning

confidence: 99%

Ethnic Bias in Judicial Decision Making: Evidence from Criminal Appeals in Kenya

2022

View full text Add to dashboard Cite

show abstract

“…There are various parameters in the modeling process that can be changed to identify the best model for a given dataset. For the purpose of our analyses, we followed the recommendations of Pennington et al ( 2014 ) and Rodriguez and Spirling ( 2022 ). To have enough context for each token, we kept a minimum occurrence of five tokens.…”

Section: Methodsmentioning

confidence: 99%

Examining Sentiment in Complex Texts. A Comparison of Different Computational Approaches

et al. 2022

View full text Add to dashboard Cite

Can we rely on computational methods to accurately analyze complex texts? To answer this question, we compared different dictionary and scaling methods used in predicting the sentiment of German literature reviews to the “gold standard” of human-coded sentiments. Literature reviews constitute a challenging text corpus for computational analysis as they not only contain different text levels—for example, a summary of the work and the reviewer's appraisal—but are also characterized by subtle and ambiguous language elements. To take the nuanced sentiments of literature reviews into account, we worked with a metric rather than a dichotomous scale for sentiment analysis. The results of our analyses show that the predicted sentiments of prefabricated dictionaries, which are computationally efficient and require minimal adaption, have a low to medium correlation with the human-coded sentiments (r between 0.32 and 0.39). The accuracy of self-created dictionaries using word embeddings (both pre-trained and self-trained) was considerably lower (r between 0.10 and 0.28). Given the high coding intensity and contingency on seed selection as well as the degree of data pre-processing of word embeddings that we found with our data, we would not recommend them for complex texts without further adaptation. While fully automated approaches appear not to work in accurately predicting text sentiments with complex texts such as ours, we found relatively high correlations with a semiautomated approach (r of around 0.6)—which, however, requires intensive human coding efforts for the training dataset. In addition to illustrating the benefits and limits of computational approaches in analyzing complex text corpora and the potential of metric rather than binary scales of text sentiment, we also provide a practical guide for researchers to select an appropriate method and degree of pre-processing when working with complex texts.

show abstract

“…Identification and scaling can be implemented through either a dictionary approach (i.e., matching target texts with a list of attribute keywords or another list of texts) or a machine learning approach. Although NLP methods are primarily developed in computational linguistics, they can also serve as robust instruments in social sciences (Rodriguez & Spirling, 2021).…”

Section: Natural Language Processingmentioning

confidence: 99%

Computational Social Science for Nonprofit Studies: Developing a Toolbox and Knowledge Base for the Field

Ebeid

Wit

et al. 2021

Voluntas

View full text Add to dashboard Cite

How can computational social science (CSS) methods be applied in nonprofit and philanthropic studies? This paper summarizes and explains a range of relevant CSS methods from a research design perspective and highlights key applications in our field. We define CSS as a set of computationally intensive empirical methods for data management, concept representation, data analysis, and visualization. What makes the computational methods “social” is that the purpose of using these methods is to serve quantitative, qualitative, and mixed-methods social science research, such that theorization can have a solid ground. We illustrate the promise of CSS in our field by using it to construct the largest and most comprehensive database of scholarly references in our field, the Knowledge Infrastructure of Nonprofit and Philanthropic Studies (KINPS). Furthermore, we show that through the application of CSS in constructing and analyzing KINPS, we can better understand and facilitate the intellectual growth of our field. We conclude the article with cautions for using CSS and suggestions for future studies implementing CSS and KINPS.

show abstract

Word Embeddings: What Works, What Doesn’t, and How to Tell the Difference for Applied Research

Cited by 66 publications

References 10 publications

Ethnic Bias in Judicial Decision Making: Evidence from Criminal Appeals in Kenya

Ethnic Bias in Judicial Decision Making: Evidence from Criminal Appeals in Kenya

Examining Sentiment in Complex Texts. A Comparison of Different Computational Approaches

Computational Social Science for Nonprofit Studies: Developing a Toolbox and Knowledge Base for the Field

Contact Info

Product

Resources

About