Estimating the Effect Of Exercising On Users’ Online Behavior

Falavarjani, Seyed Amin Mirlohi; Hosseini, Hawre; Noorian, Zeinab; Bagheri, Ebrahim

doi:10.1609/icwsm.v11i1.14975

Cited by 5 publications

(6 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Text representations used in propensity score models generally do not yet leverage recent breakthroughs in NLP, and roughly fall into three groups: those using uni-and bigram representations (De Choudhury et al 2016;Johansson, Shalit, and Sontag 2016;Olteanu, Varol, and Kiciman 2017), those using LDA or topic modeling (Falavarjani et al 2017;Roberts, Stewart, and Nielsen 2020;Sridhar et al 2018), and those using neural word embeddings such as GLoVe (Pham and Shen 2017), fastText (Joulin et al 2017;Chen, Montano-Campos, and Zadrozny 2020), or BERT (Veitch, Sridhar, and Blei 2019), (Pryzant et al 2018). Three classes of estimators are commonly used to compute the AT E: inverse probability of treatment weighting (IPTW), propensity score stratification, and matching, either using propensity scores or, less frequently, some other distance metric.…”

Section: Background and Related Workmentioning

confidence: 99%

“…Using the common setting of real social media histories (De Choudhury et al 2016;Olteanu, Varol, and Kiciman 2017;Veitch, Sridhar, and Blei 2019;Choudhury and Kiciman 2017;Falavarjani et al 2017;Kiciman, Counts, and Gasser 2018;Saha et al 2019;Roberts, Stewart, and Nielsen 2020), we identify five challenges consistently present when representing natural language for causal inference:…”

Section: Challenges For Causal Inference With Textmentioning

confidence: 99%

“…While natural language is far more complex than any finite set of challenges can capture, the five we have chosen to highlight are challenges that regularly need to be addressed in many causal inference applications that use natural language. This set of challenges was developed by reviewing a broad set of existing studies (Johansson, Shalit, and Sontag 2016;De Choudhury et al 2016;Choudhury and Kiciman 2017;Falavarjani et al 2017;Olteanu, Varol, and Kiciman 2017;Kiciman, Counts, and Gasser 2018;Sridhar et al 2018;Saha et al 2019;Veitch, Sridhar, and Blei 2019;Roberts, Stewart, and Nielsen 2020) and identifying commonalities. While the strength of selection effect, sample size, and placebo test challenges are not exclusive to causal inference with text, these challenges are present in most real world studies, and as such, a holistic evaluation framework must consider them.…”

Section: Challenges For Causal Inference With Textmentioning

confidence: 99%

“…Inspired by challenges from a wide range of studies (Johansson, Shalit, and Sontag 2016; De Choudhury et al 2016;Choudhury and Kiciman 2017;Falavarjani et al 2017;Olteanu, Varol, and Kiciman 2017;Kiciman, Counts, and Gasser 2018;Sridhar et al 2018;Saha et al 2019;Veitch, Sridhar, and Blei 2019;Roberts, Stewart, and Nielsen 2020), our framework consists of five tasks ( §5.2): Linguistic Complexity, Signal Intensity, Strength of Selection Effect, Sample Size, and Placebo Test. Each semi-synthetic task is generated from public social media users' profiles, perturbed with synthetic posts to create increasing levels of difficulty.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Adjusting for Confounders with Text: Challenges and an Empirical Evaluation Framework for Causal Inference

Weld

West

Glenski

et al. 2022

ICWSM

View full text Add to dashboard Cite

Causal inference studies using textual social media data can provide actionable insights on human behavior. Making accurate causal inferences with text requires controlling for confounding which could otherwise impart bias. Recently, many different methods for adjusting for confounders have been proposed, and we show that these existing methods disagree with one another on two datasets inspired by previous social media studies. Evaluating causal methods is challenging, as ground truth counterfactuals are almost never available. Presently, no empirical evaluation framework for causal methods using text exists, and as such, practitioners must select their methods without guidance. We contribute the first such framework, which consists of five tasks drawn from real world studies. Our framework enables the evaluation of any casual inference method using text. Across 648 experiments and two datasets, we evaluate every commonly used causal inference method and identify their strengths and weaknesses to inform social media researchers seeking to use such methods, and guide future improvements. We make all tasks, data, and models public to inform applications and encourage additional research.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Section: Challenges For Causal Inference With Textmentioning

confidence: 99%

Section: Challenges For Causal Inference With Textmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Adjusting for Confounders with Text: Challenges and an Empirical Evaluation Framework for Causal Inference

Weld

West

Glenski

et al. 2022

ICWSM

View full text Add to dashboard Cite

show abstract

“…When explicit randomization is not an option, there are numerous techniques for measuring causality from observational, historical data (Imbens and Rubin 2015), but many of these methods, such as matching (Falavarjani et al 2017;Ribeiro, Cheng, and West 2022), rely on the strong assumption of unconfoundedness, which stipulates that all relevant variables that affect both the treatment (e.g., civil interactions) and outcome (e.g., subsequent user engagement) are measured. Unconfoundedness is difficult to justify in user behavior studies, where there can be numerous latent attributes that can confound the treatment-outcome relationship of interest (Feder, Riehm, and Mojtabai 2020).…”

Section: Introductionmentioning

confidence: 99%

Measuring Causal Effects of Civil Communication without Randomization

Liu,

Ungar,

Kording

et al. 2024

ICWSM

View full text Add to dashboard Cite

Understanding the causal effects of civility is critical when analyzing online social communication, yet measuring causality is difficult. A/B tests and other randomized experiments are the gold standard for establishing causal effects but they are inapplicable in this setting due to 1) the inability to control civility levels in an experiment, and more importantly, 2) ethical constraints on intentionally randomizing civility levels. We develop a novel quasi-experimental approach to quantify the causal effect of civility in online communities on the Roblox social 3D platform without requiring explicit randomization. This method uses residual stochasticity in the "matchmaking" assignment of users to servers as a quasi-randomization mechanism in observational historical data. We find that assigning a user to a server with higher levels of civil communication could increase engagement time by as much as 1.5% in particular experiences. Given the 4.8B person hours spent monthly on the platform, this implies a potential increase of over 8,000 person years of social interaction every month. Furthermore, this effect is mis-estimated by non-causal methods. Quasi-experimental approaches promise new avenues for measuring the causal impact of user behavior in online communities without adversely affecting users through randomized experiments.

show abstract

Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates

Keith¹,

Jensen²,

O’Connor³

2020

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Many applications of computational social science aim to infer causal conclusions from nonexperimental data. Such observational data often contains confounders, variables that influence both potential causes and potential effects. Unmeasured or latent confounders can bias causal estimates, and this has motivated interest in measuring potential confounders from observed text. For example, an individual's entire history of social media posts or the content of a news article could provide a rich measurement of multiple confounders. Yet, methods and applications for this problem are scattered across different communities and evaluation practices are inconsistent. This review is the first to gather and categorize these examples and provide a guide to dataprocessing and evaluation decisions. Despite increased attention on adjusting for confounding using text, there are still many open problems, which we highlight in this paper.

show abstract

Estimating the Effect Of Exercising On Users’ Online Behavior

Cited by 5 publications

References 0 publications

Adjusting for Confounders with Text: Challenges and an Empirical Evaluation Framework for Causal Inference

Adjusting for Confounders with Text: Challenges and an Empirical Evaluation Framework for Causal Inference

Measuring Causal Effects of Civil Communication without Randomization

Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates

Contact Info

Product

Resources

About