Researcher reasoning meets computational capacity: Machine learning for social science

Lundberg, Ian; Brand, Jennie E.; Jeon, Nanum

doi:10.1016/j.ssresearch.2022.102807

Cited by 11 publications

(10 citation statements)

References 92 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Breiman’s proposition was that more scientific research should use algorithmic modeling ( 12 ). Several responses to Breiman have argued for the value of merging the two cultures or moving iteratively between them ( 48 , 62 – 66 ).…”

Section: Module 1: Study Goalsmentioning

confidence: 99%

See 1 more Smart Citation

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Kapoor,

Cantrell,

Peng

et al. 2024

Sci. Adv.

View full text Add to dashboard Cite

Machine learning (ML) methods are proliferating in scientific research. However, the adoption of these methods has been accompanied by failures of validity, reproducibility, and generalizability. These failures can hinder scientific progress, lead to false consensus around invalid claims, and undermine the credibility of ML-based science. ML methods are often applied and fail in similar ways across disciplines. Motivated by this observation, our goal is to provide clear recommendations for conducting and reporting ML-based science. Drawing from an extensive review of past literature, we present the REFORMS checklist (recommendations for machine-learning-based science). It consists of 32 questions and a paired set of guidelines. REFORMS was developed on the basis of a consensus of 19 researchers across computer science, data science, mathematics, social sciences, and biomedical sciences. REFORMS can serve as a resource for researchers when designing and implementing a study, for referees when reviewing papers, and for journals when enforcing standards for transparency and reproducibility.

show abstract

Section: Module 1: Study Goalsmentioning

confidence: 99%

“…Considering the differences and similarities between Breiman’s two cultures may be useful to researchers when motivating the use of ML methods. In addition, many recent articles provide guidance on the value of ML methods for science ( 1 , 7 , 48 , 62 , 67 – 71 ).…”

Section: Module 1: Study Goalsmentioning

confidence: 99%

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Kapoor,

Cantrell,

Peng

et al. 2024

Sci. Adv.

View full text Add to dashboard Cite

show abstract

“…LLMs can also be used as a tool for putting analytical categories to the test and may work as an interlocutor for conceptual refinement (Pardo-Guerra & Pahwa, 2022). In sum, in using LLMs sociologists are capable of "extending" (Lundberg, Brand, & Jeon, 2022) and even "augmenting" their expertise (Do et al, 2022).…”

Section: Renewals Of Text Classification With Llmsmentioning

confidence: 99%

Machine Learning and the Analysis of Culture

Mützel,

Ollion

2024

Preprint

View full text Add to dashboard Cite

The focus of this chapter is on how machine learning (ML) impacts the analysis of culture in sociology. It shows how ML has greatly advanced the analysis of culture, with new tools enabling a massive and fine-grained extraction of information from textual and audiovisual troves as well as data analysis, operationalizing long-standing cultural sociology concepts. It also indicates that this renewed interest is building on already fertile ground, as sociologists of culture have long used and reflected on formal models when analyzing culture. The chapter suggests that as the toolbox of ML approaches expands, so will the need for methodological reflection on the datasets and algorithms used, analyzed, and interpreted. The chapter also suggests that ML techniques can serve as catalysts to generate new theoretical insights. The chapter’s conclusion discusses the potential of ML research to generate new theoretical insights abductively and advocates for methodological reflexivity.

show abstract

“…Artificial intelligence has been used for impact in variety of societal domains, from education to healthcare. While many of its applications have focused on prediction, such as that of educational outcomes (Tamhane et al 2014;Lakkaraju et al 2015;Xu et al 2017) and medical incidents and risk (Hosseinzadeh et al 2013;Ma et al 2018;Ballinger et al 2018;Optum 2024), practitioners and researchers invariably encounter the question of how to use these predictions for interventions to improve the outcomes that they care about.…”

Section: Introductionmentioning

confidence: 99%

“…The set of techniques and applications for causal inference and analysis is vast, mostly notably including program evaluation and randomized controlled trials (Stephenson and Imrie The Thirty-Eighth AAAI Conference on Artificial Intelligence 1998; Deaton and Cartwright 2018), observational studies (Rosenbaum, Rosenbaum, and Briskman 2010) using modern ML techniques (Athey and Imbens 2016), adaptive trial designs (Collins, Murphy, and Strecher 2007;Montoya et al 2022), individual treatment effect and counterfactual inference (Shalit, Johansson, and Sontag 2017;Lei and Candès 2021;Bynum, Loftus, and Stoyanovich 2023). Prior work informed by causal inference has discussed the gap between predictions and decisions (Athey 2017), and how the use of prediction in these cases is predicated upon critical causal assumptions (Prosperi et al 2020;Lundberg, Brand, and Jeon 2022). In the education domain, despite the prevalence of RCTs and causal analysis on the population impact of interventions (Cook et al 2014;Yeager et al 2019), instancelevel targeting and decision making in schools are still often driven by risk scores that predict academic outcomes without incorporating knowledge of plausible interventions (Bruce et al 2011;Knowles 2015;Perdomo et al 2023).…”

Section: Introductionmentioning

confidence: 99%

On the Actionability of Outcome Prediction

Liu,

Barocas,

Kleinberg

et al. 2024

AAAI

View full text Add to dashboard Cite

Predicting future outcomes is a prevalent application of machine learning in social impact domains. Examples range from predicting student success in education to predicting disease risk in healthcare. Practitioners recognize that the ultimate goal is not just to predict but to act effectively. Increasing evidence suggests that relying on outcome predictions for downstream interventions may not have desired results. In most domains there exists a multitude of possible interventions for each individual, making the challenge of taking effective action more acute. Even when causal mechanisms connecting the individual's latent states to outcomes are well understood, in any given instance (a specific student or patient), practitioners still need to infer---from budgeted measurements of latent states---which of many possible interventions will be most effective for this individual. With this in mind, we ask: when are accurate predictors of outcomes helpful for identifying the most suitable intervention? Through a simple model encompassing actions, latent states, and measurements, we demonstrate that pure outcome prediction rarely results in the most effective policy for taking actions, even when combined with other measurements. We find that except in cases where there is a single decisive action for improving the outcome, outcome prediction never maximizes "action value", the utility of taking actions. Making measurements of actionable latent states, where specific actions lead to desired outcomes, may considerably enhance the action value compared to outcome prediction, and the degree of improvement depends on action costs and the outcome model. This analysis emphasizes the need to go beyond generic outcome prediction in interventional settings by incorporating knowledge of plausible actions and latent states.

show abstract

Researcher reasoning meets computational capacity: Machine learning for social science

Cited by 11 publications

References 92 publications

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

REFORMS: Consensus-based Recommendations for Machine-learning-based Science

Machine Learning and the Analysis of Culture

On the Actionability of Outcome Prediction

Contact Info

Product

Resources

About