People who judge people

Weiss, David J.; Shanteau, James; Harries, Priscilla

doi:10.1002/bdm.529

Cited by 40 publications

(55 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Building on the aforementioned studies and results, we posit that conceptualizing the cognitive processes of raters is an essential part of increasing the quality of PBA (Beck et al 1995;Cross et al 2001;Govaerts et al 2007;Weiss et al 2006;Wood 2014). For this study we were informed by the Theory of Expertise and the Dual-Process Theories of Reasoning.…”

Section: Purpose Of the Studymentioning

confidence: 98%

Expectations, observations, and the cognitive processes that bind them: expert assessment of examinee performance

St‐Onge

Chamberland

Lévesque

et al. 2015

Adv in Health Sci Educ

View full text Add to dashboard Cite

Performance-based assessment (PBA) is a valued assessment approach in medical education, be it in a clerkship, residency, or practice context. Raters are intrinsic to PBA and the increased use of PBA has lead to an increased interest in rater cognition. Although several researchers have tackled factors that may influence the variability in rater judgment, the critical examination of rater observation of performance and the translation of that data into judgements are being investigated. The purpose of this study was to qualitatively investigate the cognitive processes of raters, and to create a framework that conceptualizes those processes when raters assess a complex performance. We conducted semi-structured interviews with 11 faculty members (nominated as excellent assessors) from a Department of Medicine to investigate how raters observe, interpret, and translate performance into judgments. The transcribed verbal protocols were analyzed using Constructivist Grounded Theory in order to develop a theoretical model of raters' assessment processes. Several themes emerged from the data and were grouped according to three macro-level themes describing how the raters balance two sources of data [(1) external sources of information and (2) internal/personal sources of information] by relying on specific cognitive processes to assess an examinee performance. The results from our study demonstrate that assessment is a difficult cognitive task that involves nuance using specific cognitive processes to weigh external and internal data against each other. Our data clearly draws attention to the constant struggle between objectivity and subjectivity that is observed in assessment as illustrated by the importance given to nuancing the examinee's observed performance.

show abstract

Section: Purpose Of the Studymentioning

confidence: 98%

Expectations, observations, and the cognitive processes that bind them: expert assessment of examinee performance

St‐Onge

Chamberland

Lévesque

et al. 2015

Adv in Health Sci Educ

View full text Add to dashboard Cite

show abstract

“…Using a multifaceted battery to examine the level of expertise has an advantage over previous criteria: it is more complex and is able to capture the multiple dimensions to expertise. However, the study by King et al (14) did not include measures of clinicians' actual performance (16).…”

Section: Cws 0 Discrimination Inconsistencymentioning

confidence: 99%

“…In most clinical circumstances, there is no definitive or right answer that can be used to evaluate claims to expertise in a field. Therefore, assessing clinicians' level of expertise and identifying experts is a more complex task than previously thought (16).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Is length of experience an appropriate criterion to identify level of expertise?

Rassafiani¹

2009

Scandinavian Journal of Occupational Therapy

View full text Add to dashboard Cite

Clinical experience plays an important role in the development of expertise, particularly when coupled with reflection on practice. There is debate, however, regarding the amount of clinical experience that is required to become an expert. Various lengths of practice have been suggested as suitable for determining expertise, ranging from five years to 15 years. This study aimed to investigate the association between length of experience and therapists' level of expertise in the field of cerebral palsy with upper limb hypertonicity using an empirical procedure named Cochrane-Weiss-Shanteau (CWS). The methodology involved re-analysis of quantitative data collected in two previous studies. In Study 1, 18 experienced occupational therapists made hypothetical clinical decisions related to 110 case vignettes, while in Study 2, 29 therapists considered 60 case vignettes drawn randomly from those used in Study 1. A CWS index was calculated for each participant's case decisions. Then, in each study, Spearman's rho was calculated to identify the correlations between the duration of experience and level of expertise. There was no significant association between these two variables in both studies. These analyses corroborated previous findings of no association between length of experience and judgemental performance. Therefore, length of experience may not be an appropriate criterion for determining level of expertise in relation to cerebral palsy practice.

show abstract

“…Our approach to understanding how analysts employ item information in deciding an item's subsequent status can be traced to well-established findings that a prediction model, based on an expert's behaviors, tend to outperform the expert himself (viz., Armstrong, 2001;Dawes, 1979;Dawes et al, 1989;Einhorn, 1974;Goldberg, 1968;Libby, 1976;Weiss, Shanteau, & Harries, 2006). In this approach predictive models are developed that effectively serve as substitutes for the decision maker's actual judgments.…”

mentioning

confidence: 99%

Using a Model of Analysts’ Judgments to Augment an Item Calibration Process

Hauser

Thum

et al. 2014

Educational and Psychological Measurement

View full text Add to dashboard Cite

When conducting item reviews, analysts evaluate an array of statistical and graphical information to assess the fit of a field test (FT) item to an item response theory model. The process can be tedious, particularly when the number of human reviews (HR) to be completed is large. Furthermore, such a process leads to decisions that are susceptible to human errors. A key finding from behavioral decision-making research has shown that a parametric model of human decision making often outperforms the decision maker himself. We exploit this finding by seeking a model to mimic how analysts integrate FT item level statistics and graphical performance plots to predict the analyst's assignment of the item's status. The procedure suggests a set of rules that achieves a desired level of classification accuracy, separating situations in which the evidence supports firm decisions from those situations that would likely benefit from HRs. Implementation of the decision rules accounts for an estimated 65% reduction in calibrations requiring HRs.Keywords item calibration, item fit statistics, human decision making, logistic regression, item review Good item parameter estimates are critical to the validity of a test based on item response theory (IRT). Approaches to assessing the quality of estimates obtained during an item calibration process have been a subject of numerous studies for well over 1 Northwest Evaluation Association,

show abstract

People who judge people

Cited by 40 publications

References 29 publications

Expectations, observations, and the cognitive processes that bind them: expert assessment of examinee performance

Expectations, observations, and the cognitive processes that bind them: expert assessment of examinee performance

Is length of experience an appropriate criterion to identify level of expertise?

Using a Model of Analysts’ Judgments to Augment an Item Calibration Process

Contact Info

Product

Resources

About