Job analysis ratings and criterion‐related validity: Are they related and can validity be used as a measure of accuracy?

Abstract

Job analysis data are largely judgements from subject matter experts (SMEs), judgements with unknown accuracy. To date, accuracy has been inferred largely based on inter‐rater reliability or agreement between SMEs and without reference to an external criterion. The current research examined job analysis rating accuracy by comparing SME importance ratings of knowledge, skills, abilities, and other requirements (KSAOs) with the validity of measures of these same KSAOs in predicting job performance. We tested hypotheses about whether SME judgement accuracy is moderated by SME job tenure, industry experience, role, self‐reported knowledge of the job, and data scrubbing. Four independent tests involving 48 separate validation studies were conducted. In three of the four samples, there was a large (r = .50 range) relationship between trait importance and trait validity, showing that job analysis ratings can be directly related to test validities and serve as a measure of job analysis accuracy. Moderator analyses showed that the best results may come from supervisors, rather than incumbents, and those who know the job extremely well (there were no differences due to SME job tenure, industry experience, or deletion of outliers). Showing a direct relationship between SME judgements and actual criterion‐related validity provides a new lens for operationalizing accuracy in job analysis research. Practitioner Points This study demonstrates that test validities can serve as a measure of accuracy, providing a new avenue for job analysis research. The most accurate job analysis ratings came from supervisors and those who reported knowing the job extremely well.

Can you crowdsource expertise? Comparing expert and crowd-based scoring keys for three situational judgment tests

“…Not only can there be inconsistency among SMEs, but several studies have reported little to no differences in the accuracy of job analysis ratings between high and low performers (Conley & Sackett, 1987) and between job incumbents and college students (Smith & Hakel, 1979). Moreover, past research has found conflicting evidence for the ability of SMEs to identify the relative validity of traits or personal characteristics based on specific job requirements (e.g., Murphy, Deckert, Kinney, & Kung, 2013;Weekley, Labrador, Campion, & Frye, 2019). If there is uncertainty regarding whether SMEs can provide valid ratings of job performance or job requirements, how can we be sure that they are uniquely able to provide valid judgments about SJT item content?…”

Prospektive Kompetenzanalyse (ProKA) – Ein Verfahren zur Einschätzung von zukünftigen Kompetenzveränderungen

“…Dies können die Stelleninhaber*innen sowie weitere Personengruppen sein, die Auskunft über die Kompetenzanforderun-K gen geben können und die Stelle oder Tätigkeit wesentlich beeinflussen (Führungskräfte, Personalentwicklung, ggf. Kolleg*innen oder Kund*innen) (Weekley et al 2019). Um ein umfangreiches Bild der Arbeitsstelle und -tätigkeit zu bekommen und ein eingehendes Verständnis zu gewährleisten, wird durch gezieltes Nachfragen individuell auf die Antworten der Interviewten eingegangen.…”

Development of a New Personality-Oriented Work Analysis Questionnaire: First Steps Towards Validation

“…The evidence so far, shows us that there is also a linear personality-job performance relationship, even in a job analysis context, where the SMEs rate a certain trait in regards to its importance for performance in a certain job (Walmsley et al, 2018). It should also be noted that although subjective rating scales, such as those which assess the importance or difficulty regarding a characteristic or a task, produce higher levels of interrater agreement compared to objective ones, but at the same time are more prone to lower interrater reliability and discriminability between jobs (DuVernet et al, 2015), while Weekley et al (2019) reported that the use of an "importance" rating scale when using personality traits as items, is valid enough to be used in the work analysis process.…”

“…Out of all the demographic information of the SMEs, Weekley et al (2019) revealed that the most relevant one related to rating the importance of personality traits required for job performance was provided by individuals, both incumbents and supervisors, who reported that they "knew the job extremely well", regardless of either job experience in the field of activity or the job tenure. This last aspect was also strengthened by the results provided by DuVernet et al (2015), respectively, job tenure was associated with a higher inflation of the mean ratings, and also by Morgeson et al (2016) namely, a greater job experience was associated with careless responding and a lower convergence in terms of decomposed and holistic ratings.…”

“…Also, we decided not to collect other demographic information, because gender-based characteristics in terms of sample heterogeneity have produced lower factor structure confirmation and other aspects such as race or age did not generate any significant effects concerning data quality (DuVernet et al, 2015). Instead, we opted to add a four step Likert type of scale in which SMEs can assess their own level of job knowledge (Hunter, 1983) of the activity occurred within the rated job on the following continuum: (1) extremely well (2) very well, (3) somewhat and ( 4) not at all (Weekley et al, 2019).…”