Interrater agreement for journal manuscript reviews.

Whitehurst, Grover J.

doi:10.1037/0003-066x.39.1.22

Cited by 178 publications

(98 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unfortunately, the ICC is dependent on the distribution of the couples' responses to the MCRPR. Finn's r (Finn, 1970) is an agreement index similar to the ICC that does not have this undesirable property and is recommended in cases in which the distribution of scores may not be normally distributed (Whitehurst, 1984). In this sample, the mean index was .52 (SD = .28).…”

Section: Methodsmentioning

confidence: 85%

Parenting Similarity and Children with Attention-Deficit/Hyperactivity Disorder

Harvey¹

2000

Child & Family Behavior Therapy

View full text Add to dashboard Cite

ABSTRACT. This study examined the relation between parenting similarity and comorbid internalizing and disruptive behavior problems among 70 children with Attention-Deficit/Hyperactivity Disorder (ADHD). General child rearing similarity and discipline similarity were found to be associated with fewer disruptive behavior problems but not related to internalizing problems. Parenting similarity was also found to be associated with greater marital adjustment and less marital conflict among parents of ADHD children. In contrast to previous studies in non-clinical populations, controlling for parental effectiveness these relations are significant. Parenting similarity, related to fathers' reports of parenting alliance and discipline similarity, is also found to be associated with lower parenting stress for mothers. Implications for parent training for ADHD children are discussed.

show abstract

Section: Methodsmentioning

confidence: 85%

Parenting Similarity and Children with Attention-Deficit/Hyperactivity Disorder

Harvey¹

2000

Child & Family Behavior Therapy

View full text Add to dashboard Cite

show abstract

“…Many manuscripts are rejected on this criterion, even if the reviewers identify the research as sound and reported effectively. Despite evidence of the unreliability of the review process for evaluation and identifying importance (Bornmann, Mutz, & Daniel, 2010;Cicchetti, 1991;Gottfredson, 1978;Marsh & Ball, 1989;Marsh, Jayasinghe, & Bond, 2008;Peters & Ceci, 1982;Petty, Fleming, & Fabrigar, 1999;Whitehurst, 1984), this is a reasonable criterion given that journals have limited space and desires to be prestigious outlets. However, in the digital age, page limits are an anachronism (Nosek & Bar-Anan, 2012).…”

Section: Journals With Peer Review Standards Focused On the Soundnessmentioning

confidence: 99%

Scientific Utopia

Nosek

Spies

Motyl

2012

Perspect Psychol Sci

1,065

421

View full text Add to dashboard Cite

An academic scientist's professional success depends on publishing. Publishing norms emphasize novel, positive results. As such, disciplinary incentives encourage design, analysis, and reporting decisions that elicit positive results and ignore negative results. Prior reports demonstrate how these incentives inflate the rate of false effects in published science. When incentives favor novelty over replication, false results persist in the literature unchallenged, reducing efficiency in knowledge accumulation. Previous suggestions to address this problem are unlikely to be effective. For example, a journal of negative results publishes otherwise unpublishable reports. This enshrines the low status of the journal and its content. The persistence of false findings can be meliorated with strategies that make the fundamental but abstract accuracy motive-getting it right-competitive with the more tangible and concrete incentive-getting it published. This article develops strategies for improving scientific practices and knowledge accumulation that account for ordinary human motivations and biases.

show abstract

“…The Finn coefficient is recommended when variance between raters is low (Finn 1970). Whitehurst (1984) suggests Finn as an alternative to problems with Kappa, and affirms that it is the most reasonable index for agreement.…”

Section: Analysis Proceduresmentioning

confidence: 99%

“…We adopted Finn coefficient because of problems identified in Kappa coefficient, by data analysis researchers (Whitehurst 1984;Feinstein and Cicchetti 1990;Gwet 2002;Powers 2012). The Kappa test is done in two phases.…”

Section: Analysis Proceduresmentioning

confidence: 99%

Investigating factors that affect the human perception on god class detection: an analysis based on a family of four controlled experiments

Santos

Rocha-Junior

Mendonça

2017

J Softw Eng Res Dev

View full text Add to dashboard Cite

Context:Evaluation of design problems in object oriented systems, which we call code smells, is mostly a human-based task. Several studies have investigated the impact of code smells in practice. Studies focusing on human identification of code smells have shown low agreement among developers. Unfortunately, those studies do not attempt to investigate the reasons behind this phenomenon. Objective: This paper aims to investigate factors affecting human perception of code smells. Specifically, it focuses on factors affecting god class detection, one of the most known code smells. Method: The investigation encompassed a family of four controlled experiments, covering potential factors affecting human detection of code smells. The method is incremental. In other words, each experiment produces insights to the next one. This allows the investigators to control specific factors affecting the agreement on god class detection. The factors addressed in this study are: i) developer experience, ii) developer knowledge, iii) developer training, iv) tool support for design comprehension, and v) software size. Result: Our findings show that tool support for design comprehension is the only factor that does not affect the human perception of god class. The other factors impact this perception in some way. Conclusion: The area still needs more investigation and discussion on what we call the code smell conceptualization problem, to ensure similar criteria and thresholds on human-based code smell detection.

show abstract

Interrater agreement for journal manuscript reviews.

Cited by 178 publications

References 19 publications

Parenting Similarity and Children with Attention-Deficit/Hyperactivity Disorder

Parenting Similarity and Children with Attention-Deficit/Hyperactivity Disorder

Scientific Utopia

Investigating factors that affect the human perception on god class detection: an analysis based on a family of four controlled experiments

Contact Info

Product

Resources

About