Comparability of Self-Ratings and Observer Ratings in Occupational Psychosocial Risk Assessments: Is There Agreement?

Abstract

Objective. The suitability of self-ratings and observer ratings within organisational management approaches is controversial. The aim of this study was to compare the degree of agreement between self-rated and observer-rated occupational psychosocial demands. The comparison took place within a work-activity and not worker-centred assessment, according to official policies for psychosocial risk assessment. Through simultaneous application of two versions of the same instrument, we aimed to reduce the rating bias to a minimum demonstrating the suitability of self-ratings and observer ratings in companies of all kinds. Methods. A multimethod online assessment of 22 different work activities was conducted in Germany from October 2016 to October 2017. Workers (self-ratings) and occupational safety and health (OSH) committees (observer ratings) rated the occupational psychosocial risks of each activity with the same instrument (N = 669). The instrument measured psychosocial risk conditions at work. Reliability and agreement indices were computed. Results. The within-group agreement (WGA; rwg,mean = .42) of the workers’ self-ratings was good for each psychosocial risk and the interrater reliability (IRR) was excellent on average (ICC 2 = .77) with a medium effect size of ICC 1 = .15. The interrater agreement (IRA) between the two groups varied across the activities depending on rating group and activity composition (from ICCunjust,mean = .39 to ICCunjust,mean = .86) but was good to excellent on average (ICCunjust,mean = .71). Conclusion. The reasonable agreement and excellent reliability in workers’ self-ratings justify aggregation of item means at the group level. Furthermore, if the work activities are homogenous and the committee consists of members from different OSH specialties, observer ratings and self-ratings provide comparable results. According to this study’s results, both methods are reliable assessment strategies in the context of psychosocial risk assessment. The observer rating approach is especially suitable for small-to-medium enterprises that do not have access to a large anonymous survey assessment.

The Assessment of Psychosocial Work Conditions and Their Relationship to Well-Being: A Multi-Study Report

“…The only exception is skill utilization (item 10), which does not relate to well-being, but positive strain-balance (data not shown). Nevertheless, we decided to keep this item in the practitioners' version of the questionnaire from the 2019th validation [12], as it seems to be a resource if present during task operation. One item that was removed from the practitioners' version was the question of a fixed place of work (item 16) because it delivered contradictory results.…”

“…Future research should focus on validating the new instruments in relation to more objective work-related health outcomes. However, a previous study has shown that the subjective outcomes correspond to observer-ratings [12] and thus could be regarded as a rather "objective" assessment of the working conditions. Regarding objective health outcomes, physiological variables such as hair cortisol as a biomarker for chronic stress [38] or organizational outcome variables (e.g., sick leave days) could be considered.…”

“…In addition, the majority of those questionnaires assessed work demands from a person-centered perspective, potentially increasing the subjectivity bias [11]. Therefore, we have developed a new instrument that assesses working conditions from a third-person perspective [12].…”

“…PsyHealth is an objective and reliable instrument to assess psychosocial work conditions at item level in true work-settings [12]. We would like to examine whether the instrument has psychometric properties for use in research (Study 1) and test if PsyHealth measures psychosocial work conditions reliably over time and if it provides incremental validity above existing instruments in explaining variance in employee well-being (Study 2).…”

Corrigendum to “Comparability of Self-Ratings and Observer Ratings in Occupational Psychosocial Risk Assessments: Is There Agreement?”

“…In the article titled “Comparability of Self-Ratings and Observer Ratings in Occupational Psychosocial Risk Assessments: Is There Agreement?” [ 1 ], References [5, 8] should be removed since they were recommended during the review process and do not contribute essentially to the topic.…”