Inter-rater reliability, intra-rater reliability and internal consistency of the Brisbane Evidence-Based Language Test

Abstract

Purpose: To examine the inter-rater reliability, intra-rater reliability, internal consistency and practice effects associated with a new test, the Brisbane Evidence-Based Language Test. Methods: Reliability estimates were obtained in a repeated-measures design through analysis of clinician video ratings of stroke participants completing the Brisbane Evidence-Based Language Test. Inter-rater reliability was determined by comparing 15 independent clinicians' scores of 15 randomly selected videos. Intra-rater reliability was determined by comparing two clinicians' scores of 35 videos when re-scored after a two-week interval. Results: Intraclass correlation coefficient (ICC) analysis demonstrated almost perfect inter-rater reliability (0.995; 95% confidence interval: 0.990-0.998), intra-rater reliability (0.994; 95% confidence interval: 0.989-0.997) and internal consistency (Cronbach's a ¼ 0.940 (95% confidence interval: 0.920-1.0)). Almost perfect correlations (0.998; 95% confidence interval: 0.995-0.999) between face-to-face and video ratings were obtained. Conclusion: The Brisbane Evidence-Based Language Test demonstrates almost perfect inter-rater reliability, intra-rater reliability and internal consistency. High correlation coefficients and narrow confidence intervals demonstrated minimal practice effects with scoring or influence of years of clinical experience on test scores. Almost perfect correlations between face-to-face and video scoring methods indicate these reliability estimates have direct application to everyday practice. The test is available from brisbanetest.org. ä IMPLICATIONS FOR REHABILITATION The Brisbane Evidence-Based Language Test is a new measure for the assessment of acquired language disorders. The Brisbane Evidence-Based Language Test demonstrated almost perfect inter-rater reliability, intrarater reliability and internal consistency. High reliability estimates and narrow confidence intervals indicated that test ratings vary minimally when administered by clinicians of different experience levels, or different levels of familiarity with the new measure. The test is a reliable measure of language performance for use in clinical practice and research.

Untitled

“…Based on Farhady et al's (2001) guideline, high (r = 0.844-0.92) and almost moderate (r = 0.69-0.77) correlations were found for the first and second-raters, respectively. Furthermore, as indicated by the findings of the previous studies (Kayapınar, 2014;Rohde et al, 2020), a longer period is deemed to reduce the intra-rater reliability (Kayapınar, 2014). Likewise, in the case of our study, the first rater's higher reliability scores suggest that time may negatively affect intra-rater reliability; the longer the interval between the two ratings, the lower the reliability.…”

“…From the batch of 110 samples, 50 were randomly selected and given to the first and second-raters to be scored after six-week and nine-week intervals, respectively, to test the intra-rater reliability achieved by the raters using ASAW. Various intervals have been suggested in the literature ranging from two weeks (Rohde et al, 2020) to 10 weeks (Kayapınar, 2014). We did not opt for a small interval to allow enough time for a wash-out period.…”

Cross‐cultural adaptation and psychometric properties of the Indonesian version of theory of planned behavior questionnaire to measure dental attendance of children with hearing loss: A pilot study

“…11,20,22 This pilot study revealed that 90% of the ICC results are perfectly correlated, indicating the reliability and stability of this questionnaire. 23 It can also be concluded that this questionnaire has no floor and/or ceiling effect, based on the participants' minimum and maximum total score for each variable of attitude, subjective norm, behavioral perception, and intention (TPB model) less than 15%. 24 In other words, the questionnaire is sensitive and precise to measure the minimum and maximum scores, allowing for measurement of data variation in a proper manner.…”

What is the usage of the Brisbane Evidence Based Language Test in clinical practice?: A speech language therapy survey

“…The development and diagnostic validation of the language test are described in the Rohde et al (2020a) multicentre cross-sectional study. A second psychometric investigation of interrater reliability, intra-rater reliability and internal consistency has also been published, with findings supporting that the EBLT is a diagnostically sensitive assessment (Rohde et al, 2020b). Clinician expertise and experiences are an essential component of evidencebased healthcare (Bate et al, 2010).…”