Everything has its Price: Foundations of Cost-Sensitive Learning and its Application in Psychology

Sterner, Philipp; Goretzko, David; Pargent, Florian

doi:10.31234/osf.io/7asgz

Cited by 9 publications

(13 citation statements)

References 67 publications

(108 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…That could be particularly true for Adversity and Deception, both of which were "absent" in the great majority of situational snapshots in our sample, confirming the low variances repeatedly found for these dimensions in past studies (s., Jonason & Sherman, 2020;Rauthmann et al, 2014;Sherman et al, 2015). However, we countered this under-representation problem by applying specific weighting techniques for imbalanced class distributions (s., Sterner et al, 2021) in our data analyses. Thus, we can rule out the possibility that the differential predictability of the eight DIAMONDS dimensions was merely caused by methodological issues.…”

Section: Differential Prediction Performance Patterns For Situation C...supporting

confidence: 87%

“…We ran these models with class-dependent costs to account for imbalanced class distributions (e.g., Deception was only "present" in 3% of all sampled situations). Thereby, we assigned a class-dependent theoretical weight to each observation to increase the effect of the minority class and decrease the effect of the majority class observations (Sterner et al, 2021).…”

Section: Machine Learningmentioning

confidence: 99%

See 1 more Smart Citation

Snapshots of Daily Life: Situations Investigated Through the Lens of Smartphone Sensing

Schoedel¹,

Kunz²,

Bergmann³

et al. 2022

Preprint

View full text Add to dashboard Cite

Daily life unfolds in a sequence of situational contexts, which are pivotal for explaining people’s thoughts, feelings, and behaviors. While situational data were previously difficult to collect, the ubiquity of smartphones now opens up new opportunities for assessing situations in situ, that is, while they occur. Seizing this development, the present study demonstrates how smartphones can help establish associations between the psychological perception and the physical reality of situations. We employed an intensive longitudinal sampling design and investigated 9,790 situational snapshots experienced by 455 participants for 14 consecutive days. These snapshots combined self-reported situation characteristics from experience samplings with their corresponding objective situation cues obtained via smartphone sensing. To account for the complexity of real-world situations, we extracted a total of 1,356 granular situation cues from different sensing modalities. We applied linear and nonlinear machine learning algorithms to examine how well these cues predicted the perceived characteristics in terms of the Situational Eight DIAMONDS, finding significant out-of-sample predictions for the five dimensions capturing the situations’ Duty, Intellect, Mating, pOsitivity, and Sociality. Analyses of (grouped) feature importance revealed that these predictions relied on complex constellations of cues representing various situational information about the Persons/Interactions and Objects present, the Events/Activities happening, and the current Location and Time. Furthermore, a nomological network analysis provided evidence for the construct validity of our cue-based DIAMONDS predictions. We conclude by discussing how smartphone-based situational snapshots, in general, and our prediction models, in particular, advance psychological research on situations.

show abstract

Section: Differential Prediction Performance Patterns For Situation C...supporting

confidence: 87%

Section: Machine Learningmentioning

confidence: 99%

Snapshots of Daily Life: Situations Investigated Through the Lens of Smartphone Sensing

Schoedel¹,

Kunz²,

Bergmann³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This absence of situations high in Deception and Adversity may reflect an actual (fortunate) lack of such situations in everyday life or may be a methodological artifact introduced by a sampling bias if participants do not answer ESs in these situations across studies. Either way, we countered this underrepresentation by applying specific weighting techniques for imbalanced class distributions in our data analyses, as explained in our methods section (see also Sterner et al, 2021). Thus, we can rule out that the differential predictability of the eight DIAMONDS dimensions was merely a methodological artifact of the algorithms applied in our study.…”

Section: Discussionmentioning

confidence: 99%

Snapshots of daily life: Situations investigated through the lens of smartphone sensing.

Schoedel¹,

Kunz²,

Bergmann³

et al. 2023

Journal of Personality and Social Psychology

View full text Add to dashboard Cite

Daily life unfolds in a sequence of situational contexts, which are pivotal for explaining people’s thoughts, feelings, and behaviors. While situational data were previously difficult to collect, the ubiquity of smartphones now opens up new opportunities for assessing situations in situ, that is, while they occur. Seizing this opportunity, the present study demonstrates how smartphones can help establish associations between the psychological perception and physical reality of situations. We employed an intensive longitudinal sampling design and investigated 9,790 situational snapshots experienced by 455 participants for 14 consecutive days. These snapshots combined self-reported situation characteristics from experience samplings with their corresponding objective cues obtained via smartphone sensing. More precisely, we extracted a total of 1,356 granular cues from different sensing modalities to account for the complexity of real-world situations. We applied linear and nonlinear machine learning algorithms to examine how well these cues predicted the perceived characteristics in terms of the Situational Eight Duty, Intellect, Adversity, Mating, pOsitivity, Negativity, Deception, Sociality (DIAMONDS), finding significant out-of-sample predictions for the five dimensions reflecting the situations’ Duty, Intellect, Mating, pOsitivity, and Sociality. In a series of follow-up analyses, we further explored the data patterns captured by our models, revealing, for example, that those cues related to time and location were particularly informative of the respective situation characteristics. We conclude by interpreting the mapping between cues and characteristics in real-world situations and discussing how smartphone-based situational snapshots may push the boundaries of psychological research on situations.

show abstract

“…When looking at the results, we notice that while M M CE is very similar for RF and LASSO, the models slightly differ in their respective tradeoff of SEN S and SP EC. This finding exemplifies the need to consider other performance measures beyond mean classification error or accuracy in many applied classification settings, in which the practical cost of false positive and false negative predictions are not the same (Sterner et al, 2021).…”

Section: Practical Exercise Iii: Model Comparisons With Benchmark Exp...mentioning

confidence: 77%

“…AUC is based on the receiver operating curve (ROC) which plots 1 − SP EC against SEN S. Each combination results from a different prediction threshold, that means we predict class 1 if the predicted probability for class 1 is greater than the threshold (0.5 by default). For more advanced techniques, Sterner et al (2021) give an introduction to cost-sensitive learning for psychologists with mlr3.…”

Section: Table 2 Confusion Matrixmentioning

confidence: 99%

Best Practices in Supervised Machine Learning: A Tutorial for Psychologists

Pargent¹,

Schoedel²,

Stachl³

2022

Preprint

View full text Add to dashboard Cite

Supervised machine learning (ML) is becoming an influential research method in psychology and other social sciences. However, theoretical ML concepts and predictive modeling techniques are not yet widely taught in psychology programs. This tutorial is intended to provide a low-barrier, non-technical entrance to supervised ML for psychologists in four consecutive modules. After introducing the basic idea of supervised ML, Module I covers performance evaluation of ML models with resampling methods (performance measures, bias-variance tradeoff, k-fold cross-validation). Module II introduces nonlinear, tree-based algorithms, focusing on random forests and their components, regression and classification trees. Module III is about performing empirical benchmark experiments (comparing the performance of several ML algorithms on multiple datasets). Finally, Module IV discusses the interpretation of ML models, including permutation variable importance measures, effect plots (partial dependence plots, individual conditional expectation profiles, accumulated local effect plots), and the concept of model fairness. Throughout the tutorial, intuitive descriptions of theoretical concepts (with as few mathematical formulas as possible) are followed by code examples, using the mlr3 and companion packages in R. Key practical analysis steps are demonstrated on the publicly available PhoneStudy dataset (N = 624), which includes over 1800 variables from smartphone sensing to predict Big Five personality trait scores. The manuscript contains a checklist to be used as a reminder on important aspects when performing, reporting, or reviewing ML analyses in psychology. Additional examples and more advanced concepts are demonstrated in extensive online materials (https://osf.io/9273g/).

show abstract

Everything has its Price: Foundations of Cost-Sensitive Learning and its Application in Psychology

Cited by 9 publications

References 67 publications

Snapshots of Daily Life: Situations Investigated Through the Lens of Smartphone Sensing

Snapshots of Daily Life: Situations Investigated Through the Lens of Smartphone Sensing

Snapshots of daily life: Situations investigated through the lens of smartphone sensing.

Best Practices in Supervised Machine Learning: A Tutorial for Psychologists

Contact Info

Product

Resources

About