Can training improve the quality of inferences made by raters in competency modeling? A quasi-experiment.

Lievens, Filip; Sánchez, Juan I.

doi:10.1037/0021-9010.92.3.812

Cited by 55 publications

(63 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Even though it has been shown for mini-CEX that rater training had no effect on the reliability of the exam scores [30] it is an unconditional requirement to reach solid reliability for the assessment of competences [11]. Another study also showed that rater training had a positive effect on the quality of inferences made by raters in competence modeling [31]. Additionally, a qualitative study pointed out that participation of clinicians in a performance dimension rater training and in a frame of reference rater training equipped participants with assessment skills, which were congruent with principles of criterion-referenced assessment and entrustment, and basic principles of competency-based education [32].…”

Section: Discussionmentioning

confidence: 94%

Assessing Competences in Medical Students with a Newly Designed 360- Degree Examination of a Simulated First Day of Residency: A Feasibility Study

Harendza¹,

Berberat²,

Kadmon³

2017

J Community Med Health Educ

View full text Add to dashboard Cite

Background: Many medical schools worldwide are in the process of curricular change towards competencebased undergraduate medical education to ease the transition to residency. Aspects of patient management, interprofessional interaction and the concept of entrustable professional activities are in the focus of medical educators. Such curricular changes require an assessment with valid and reliable measurements of competences feasible for large numbers of medical students.

show abstract

Section: Discussionmentioning

confidence: 94%

Assessing Competences in Medical Students with a Newly Designed 360- Degree Examination of a Simulated First Day of Residency: A Feasibility Study

Harendza¹,

Berberat²,

Kadmon³

2017

J Community Med Health Educ

View full text Add to dashboard Cite

show abstract

“…These recommendations are not meant to be exhaustive; we urge researchers and practitioners to build on them and conduct much-needed empirical research. In the past, we successfully engaged in similar endeavors to formulate evidence-based recommendations for increasing the quality of the data gathered in unproctored Internet testing (e.g., Bartram, 2008;Lievens & Burke, 2011) and in competency determinations (e.g., Campion et al, 2011;Lievens & Sanchez, 2007). There is no reason why we could not do this again this time.…”

Section: Epiloguementioning

confidence: 99%

Reducing the Noise From Scraping Social Media Content: Some Evidence-Based Recommendations

Lievens

Iddekinge

2016

Ind. Organ. Psychol.

Self Cite

View full text Add to dashboard Cite

Chamorro-Premuzic, Winsborough, Sherman, and Hogan (2016) describe a variety of new selection approaches (e.g., "scraping" of social media information, gamified assessments) in the staffing domain that might provide new sources of information about people. The authors also mention advantages and downsides of these potentially "new talent signals."We suggest that the next step is to identify conditions under which these new approaches might be best used to increase their probability of providing accurate job-related information on candidates' knowledge, skills, abilities, and other characteristics (KSAOs). Although there has been little scientific research on these new assessment methods, we posit that some guidance may be found in the over 100 years of research on personnel selection. This makes sense because, as noted in the focal article, these new tools are mainly technologically enhanced versions of traditional assessment methods. Therefore, we draw on existing personnel selection knowledge to delineate a set of general recommendations to make these new talent signals less weak and "noisy" (i.e., more reliable and valid). We focus primarily on scraping social media information, although we show how some of these recommendations may be relevant for gamified assessment as well.

show abstract

“…From these weights, one can also determine the relative evidence for Hypothesis m compared to m . For instance, in the example of Lievens and Sanchez (2007), H 1 is 0.44/0.14 ≈ 3.18 more likely than H u . Therefore, it is not a weak hypothesis.…”

Section: The Goric Weightsmentioning

confidence: 99%

“…In this section, we will illustrate the GORIC supported by real data for which the descriptive statistics are available in Lievens and Sanchez (2007). They investigated the effect of training on the quality of ratings made by consultants.…”

Section: Analysis Of Variance (Anova)mentioning

confidence: 99%

AFortran90 Program for the Generalized Order-Restricted Information Criterion

Kuiper¹,

Hoijtink²

2013

J. Stat. Soft.

View full text Add to dashboard Cite

The generalized order-restricted information criterion (GORIC) is a generalization of the Akaike information criterion such that it can evaluate hypotheses that take on specific, but widely applicable, forms (namely, closed convex cones) for multivariate normal linear models. It can examine the traditional hypotheses H 0 : β 1,1 = · · · = β t,k and H u : β 1,1 , . . . , β t,k and hypotheses containing simple order restrictions H m : β 1,1 ≥ . . . ≥ β t,k , where any "≥" may be replaced by "=" and m is the model/hypothesis index; with β h,j the parameter for the h-th dependent variable and the j-th predictor in a t-variate regression model with k predictors (which might include the intercept). But, the GORIC can also be applied to restrictions of the form H m : R 1 β = r 1 , R 2 β ≥ r 2 , with β a vector of length tk, R 1 a c m1 × tk matrix, r 1 a vector of length c m1 , R 2 a c m2 × tk matrix, and r 2 a vector of length c m2 . It should be noted that [R 1 , R 2 ] should be of full rank when [r 1 , r 2 ] = 0. In practice, this implies that one cannot examine range restrictions (e.g., 0 < β 1,1 < 2 or β 1,2 < β 1,1 < 2β 1,2 ) with the GORIC. A Fortran 90 program is presented, which enables researchers to compute the GORIC for hypotheses in the context of multivariate regression models. Additionally, an R package called goric is made by Daniel Gerhard and the first author.

show abstract

Can training improve the quality of inferences made by raters in competency modeling? A quasi-experiment.

Cited by 55 publications

References 32 publications

Assessing Competences in Medical Students with a Newly Designed 360- Degree Examination of a Simulated First Day of Residency: A Feasibility Study

Assessing Competences in Medical Students with a Newly Designed 360- Degree Examination of a Simulated First Day of Residency: A Feasibility Study

Reducing the Noise From Scraping Social Media Content: Some Evidence-Based Recommendations

AFortran90 Program for the Generalized Order-Restricted Information Criterion

Contact Info

Product

Resources

About