Unfolding the phenomenon of interrater agreement: a multicomponent approach for in-depth examination was proposed

Abstract

Objective: The overall objective was to unfold the phenomenon of inter-rater agreement: to identify potential sources of variation in agreement data and to explore how they can be statistically accounted for. The ultimate aim was to propose recommendations for in-depth examination of agreement, in order to improve the reliability of assessment instruments.Study Design and Setting: Utilizing a sample where 10 rater pairs had assessed the presence/absence of 188 environmental barriers by a systematic rating form, a raters items dataset was generated (N=1,880). In addition to common agreement indices, relative shares of agreement variation were calculated. Multilevel regression analysis was carried out, using rater and item characteristics as predictors of agreement variation.Results: Following a conceptual decomposition, the agreement variation was statistically disentangled into relative shares. The raters accounted for 6-11%, the items for 32-33% and the residual for 57-60% of the variation. Multilevel regression analysis showed barrier prevalence and raters" familiarity with using standardized instruments to have the strongest impact on agreement.Conclusion: Supported by a conceptual analysis we propose an approach of in-depth examination of agreement variation, as a strategy for increasing the level of inter-rater agreement. By identifying and limiting the most important sources of disagreement, ultimately instrument reliability can be improved.

Test–retest reliability and agreement of the Satisfaction with the Assistive Technology Services (SATS) instrument in two Nordic countries

“…In order to explore this, as recommended by Slaug et al (2012) an analysis of the characteristics of the interviewers and the settings in which the interviews took place would have been be useful. Unfortunately, we did not collect any data for such an analysis.…”

Assessment and analysis of housing accessibility: adapting the environmental component of the housing enabler to United States applications

“…According to Slaug et al (2012), a number of factors may have contributed to the less than ideal levels of agreement attained. That is, there are many methodological challenges to analyzing inter-rater agreement using multiple pairs of raters and varying home environments.…”

Assessing environmental barriers by means of the Swedish craig hospital inventory of environmental factors among people post-stroke

“…The CHIEF-S [8] comprises five sub-scales with 25 items in total. The sub-scales are Attitude/support (5 items), Service/Assistance (7 items), Physical/structural (6 items), Work/school (3 items) and Policy (4 items).…”

“…However, assessing environmental features and its impact on occupational engagement is associated with major methodological challenges because of its complexity [7][8][9]. For instance, occupational engagement can be viewed from a taxonomic hierarchy where the level of complexity range from voluntary movement and mental processes, actions, tasks, activities to engagement in entire occupations [1].…”

“…Environmental features that influence e.g. occupational engagement might also be difficult to assess due to the dynamic of the physical environment [8], e.g. changing light and weather conditions or due to other dimensions such as attitudes and service [9].…”