An Evaluation of Interrater Reliability Measures on Binary Tasks Using <i>d-Prime</i>

Grant, Malcolm J.; Button, Cathryn M.; Snook, Brent

doi:10.1177/0146621616684584

Cited by 28 publications

(11 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We did not code as suspicious anyone who generally thought that the study was strange (they thought it was weird, odd, or unusual that the answers were provided or that they were told not to look at the answers; they were not sure if the provided answers were correct). Between-coder agreement was high, κ = .904 (Cohen, 1960; Grant, Button, & Snook, 2017), and all disagreements were resolved by the team leads at each site.…”

Section: Resultsmentioning

confidence: 99%

Many Labs 5: Registered Replication of Vohs and Schooler (2008), Experiment 1

Buttrick

Aczél

Aeschbach

et al. 2020

Advances in Methods and Practices in Psychological Science

View full text Add to dashboard Cite

Does convincing people that free will is an illusion reduce their sense of personal responsibility? Vohs and Schooler (2008) found that participants reading from a passage “debunking” free will cheated more on experimental tasks than did those reading from a control passage, an effect mediated by decreased belief in free will. However, this finding was not replicated by Embley, Johnson, and Giner-Sorolla (2015), who found that reading arguments against free will had no effect on cheating in their sample. The present study investigated whether hard-to-understand arguments against free will and a low-reliability measure of free-will beliefs account for Embley et al.’s failure to replicate Vohs and Schooler’s results. Participants ( N = 621) were randomly assigned to participate in either a close replication of Vohs and Schooler’s Experiment 1 based on the materials of Embley et al. or a revised protocol, which used an easier-to-understand free-will-belief manipulation and an improved instrument to measure free will. We found that the revisions did not matter. Although the revised measure of belief in free will had better reliability than the original measure, an analysis of the data from the two protocols combined indicated that free-will beliefs were unchanged by the manipulations, d = 0.064, 95% confidence interval = [−0.087, 0.22], and in the focal test, there were no differences in cheating behavior between conditions, d = 0.076, 95% CI = [−0.082, 0.22]. We found that expressed free-will beliefs did not mediate the link between the free-will-belief manipulation and cheating, and in exploratory follow-up analyses, we found that participants expressing lower beliefs in free will were not more likely to cheat in our task.

show abstract

Section: Resultsmentioning

confidence: 99%

Many Labs 5: Registered Replication of Vohs and Schooler (2008), Experiment 1

Buttrick

Aczél

Aeschbach

et al. 2020

Advances in Methods and Practices in Psychological Science

View full text Add to dashboard Cite

show abstract

“…The third possibility is a statistical artifact. The expert group (11-year-old group) showed quite few instances of nonproficiency, which, combined with the small sample size for the analysis, biased the statistics to provide this result (Grant et al, 2017).…”

Section: Discussionmentioning

confidence: 99%

Development and Validation of a Checklist to Assess Proficient Performance of Basketball Straight Speed Dribbling Skill

Santos

Pacheco

Basso

et al. 2020

Journal of Human Kinetics

View full text Add to dashboard Cite

The aim of this study was to develop and validate a checklist to assess proficient performance of basketball straight speed dribbling skill. The sample was composed of 100 children and adolescents between 7 and 15 years of age with and without structured practice in basketball. The validation process tested the validity domain, decision, tendencies, reliability, responsiveness, and objectivity. The results show that the checklist contains criteria that represent the speed dribbling skill and is sensible to distinguish between different proficiency levels of performance. The results also expressed high reliability and objectivity (intra and inter-rater). In light of the findings, we concluded that the checklist can be used to reliably analyze performance and evaluate the process of learning and development of the straight speed dribbling skill.

show abstract

“…A recent paper by Grant, Button, and Snook (2017) suggested one resolution of this debate. In a series of Monte Carlo simulations, Grant and his colleagues used a novel criterion, d-prime, to assess the performance of five reliability measure indices.…”

Section: (Dis)agreement On Inter-rater Agreementmentioning

confidence: 99%

“…In Panel c, both Chris and Laura say "yes" more often than "no" but differ in the ratio. Grant et al (2017) referred to this kind of sensitivity as Observer Expertise, and it is this factor that is central to reliability. Second, agreement might occur because both Chris and Laura come to the task with similar assumptions about the prevalence of cooperation.…”

Section: An Inter-rater Reliability Dilemmamentioning

confidence: 99%

“…Second, agreement might occur because both Chris and Laura come to the task with similar assumptions about the prevalence of cooperation. Grant et al (2017) called this factor Observer Bias. Finally, agreement might occur purely by chance; that is, Chris and Laura could have been blindfolded and agreement would still be high.…”

Section: An Inter-rater Reliability Dilemmamentioning

confidence: 99%

See 1 more Smart Citation