A comparative judgment approach to assessing Chinese Sign Language interpreting

Abstract

The quality of sign language interpreting (SLI) is a gripping construct among practitioners, educators and researchers, calling for reliable and valid assessment. There has been a diverse array of methods in the extant literature to measure SLI quality, ranging from traditional error analysis to recent rubric scoring. In this study, we want to expand the terrain of SLI assessment, by exploring and evaluating a novel method, known as comparative judgment (CJ), to assess SLI quality. Briefly, CJ involves judges to compare two like objects/items and make a decision by choosing the one with higher quality. The binary outcomes from repeated comparisons by a group of judges are then modelled statistically to produce standardized estimates of perceived quality for each object/item. We recruited 12 expert judges to operationalize CJ via a computerized system to assess the quality of Chinese Sign Language interpreting produced by 36 trainee interpreters. Overall, our analysis of quantitative and qualitative data provided preliminary evidential support for the validity and utility of CJ in SLI assessment. We discussed these results in relation to previous SLI literature, and suggested future research to cast light on CJ’s usefulness in applied assessment contexts.

Assessing intelligibility as conceptualised within the CEFR-companion volume (CV) framework using Adaptive Comparative Judgement

“…In addition, the SSRs reached a relatively high level for both judge groups, exceeding the established lower limit of 0.90 for high-stakes assessments (Han & Xiao, 2022;Verhavert et al, 2019). Although the "inflated" reliability of ACJ was calibrated in RM Compare (Kimbell, 2021), without a comparison with traditional CJ results, whether there is inflation in the SSRs of this study remains unclear.…”

“…Furthermore, various cutoffs used to standardise infit statistic values can lead to differing outcomes. For example, some judges' infit statistics in this study might fall outside the standard range of 0.5 to 1.5, as used in Han and Xiao (2022). Thus, while results from RM Compare are generally precise, adopting ACJ in high-stakes settings demands careful consideration.…”

“…The current study expands this approach by adopting Weir's (2005) sociocognitive validation framework, which includes scoring and perceived validity. This framework, previously applied in Han (2022) and Han and Xiao (2022), assesses whether judges rely on irrelevant criteria during evaluations and evaluates how they perceive the usefulness of ACJ based on their experience.…”

Assessing intelligibility as conceptualised within the CEFR-companion volume (CV) framework using Adaptive Comparative Judgement

“…In addition, the SSRs reached a relatively high level for both judge groups, exceeding the established lower limit of 0.90 for high-stakes assessments (Han & Xiao, 2022;Verhavert et al, 2019). Although the "inflated" reliability of ACJ was calibrated in RM Compare (Kimbell, 2021), without a comparison with traditional CJ results, whether there is inflation in the SSRs of this study remains unclear.…”

“…Furthermore, various cutoffs used to standardise infit statistic values can lead to differing outcomes. For example, some judges' infit statistics in this study might fall outside the standard range of 0.5 to 1.5, as used in Han and Xiao (2022). Thus, while results from RM Compare are generally precise, adopting ACJ in high-stakes settings demands careful consideration.…”

“…The current study expands this approach by adopting Weir's (2005) sociocognitive validation framework, which includes scoring and perceived validity. This framework, previously applied in Han (2022) and Han and Xiao (2022), assesses whether judges rely on irrelevant criteria during evaluations and evaluates how they perceive the usefulness of ACJ based on their experience.…”

Quality assessment in multilingual, multimodal, and multiagent translation and interpreting (QAM 3 T&I): Proposing a unifying framework for research

“…The multimodality analysis indicates a significant under-representation of inter-modal translational practices (e.g., sign-to-voice interpreting) and auditory-mediated communication (e.g., spoken-language interpreting), compared to written/typed translation and MT. Research should explore under-examined modalities, including sign language interpreting, audio description, subtitling, respeaking, and dubbing (Fryer, 2019;Han & Xiao, 2021;Pedersen, 2017;Romero-Fresco & Pöchhacker, 2017;Yan & Luo, 2023) and compare them with intra-lingual modalities, thus broadening the scope of research on multimodal T&I-mediated communication and its implications.…”

Large language models as zero-shot evaluators of English–Chinese interpreting: A comparison of GPT-4o and DeepSeek-R1

“…Another widely adopted method is rubric-based scoring (Shafiei, 2024). More recently, comparative judgment has been introduced as an alternative approach (Han & Xiao, 2022), where a group of judges conduct repeated and randomized pairwise comparisons of interpreted renditions, selecting the higher-quality version in each comparison (Sickinger et al, 2025).…”

Assessing intelligibility as conceptualised within the CEFR-companion volume (CV) framework using Adaptive Comparative Judgement

“…In addition, the SSRs reached a relatively high level for both judge groups, exceeding the established lower limit of 0.90 for high-stakes assessments (Han & Xiao, 2022;Verhavert et al, 2019). Although the "inflated" reliability of ACJ was calibrated in RM Compare (Kimbell, 2021), without a comparison with traditional CJ results, whether there is inflation in the SSRs of this study remains unclear.…”

“…Furthermore, various cutoffs used to standardise infit statistic values can lead to differing outcomes. For example, some judges' infit statistics in this study might fall outside the standard range of 0.5 to 1.5, as used in Han and Xiao (2022). Thus, while results from RM Compare are generally precise, adopting ACJ in high-stakes settings demands careful consideration.…”

“…The current study expands this approach by adopting Weir's (2005) sociocognitive validation framework, which includes scoring and perceived validity. This framework, previously applied in Han (2022) and Han and Xiao (2022), assesses whether judges rely on irrelevant criteria during evaluations and evaluates how they perceive the usefulness of ACJ based on their experience.…”

Quality assessment in multilingual, multimodal, and multiagent translation and interpreting (QAM 3 T&I): Proposing a unifying framework for research

“…The multimodality analysis indicates a significant under-representation of inter-modal translational practices (e.g., sign-to-voice interpreting) and auditory-mediated communication (e.g., spoken-language interpreting), compared to written/typed translation and MT. Research should explore under-examined modalities, including sign language interpreting, audio description, subtitling, respeaking, and dubbing (Fryer, 2019;Han & Xiao, 2021;Pedersen, 2017;Romero-Fresco & Pöchhacker, 2017;Yan & Luo, 2023) and compare them with intra-lingual modalities, thus broadening the scope of research on multimodal T&I-mediated communication and its implications.…”

Large language models as zero-shot evaluators of English–Chinese interpreting: A comparison of GPT-4o and DeepSeek-R1

“…Another widely adopted method is rubric-based scoring (Shafiei, 2024). More recently, comparative judgment has been introduced as an alternative approach (Han & Xiao, 2022), where a group of judges conduct repeated and randomized pairwise comparisons of interpreted renditions, selecting the higher-quality version in each comparison (Sickinger et al, 2025).…”

Assessing intelligibility as conceptualised within the CEFR-companion volume (CV) framework using Adaptive Comparative Judgement

“…In addition, the SSRs reached a relatively high level for both judge groups, exceeding the established lower limit of 0.90 for high-stakes assessments (Han & Xiao, 2022;Verhavert et al, 2019). Although the "inflated" reliability of ACJ was calibrated in RM Compare (Kimbell, 2021), without a comparison with traditional CJ results, whether there is inflation in the SSRs of this study remains unclear.…”

“…Furthermore, various cutoffs used to standardise infit statistic values can lead to differing outcomes. For example, some judges' infit statistics in this study might fall outside the standard range of 0.5 to 1.5, as used in Han and Xiao (2022). Thus, while results from RM Compare are generally precise, adopting ACJ in high-stakes settings demands careful consideration.…”

“…The current study expands this approach by adopting Weir's (2005) sociocognitive validation framework, which includes scoring and perceived validity. This framework, previously applied in Han (2022) and Han and Xiao (2022), assesses whether judges rely on irrelevant criteria during evaluations and evaluates how they perceive the usefulness of ACJ based on their experience.…”

Quality assessment in multilingual, multimodal, and multiagent translation and interpreting (QAM 3 T&I): Proposing a unifying framework for research

“…The multimodality analysis indicates a significant under-representation of inter-modal translational practices (e.g., sign-to-voice interpreting) and auditory-mediated communication (e.g., spoken-language interpreting), compared to written/typed translation and MT. Research should explore under-examined modalities, including sign language interpreting, audio description, subtitling, respeaking, and dubbing (Fryer, 2019;Han & Xiao, 2021;Pedersen, 2017;Romero-Fresco & Pöchhacker, 2017;Yan & Luo, 2023) and compare them with intra-lingual modalities, thus broadening the scope of research on multimodal T&I-mediated communication and its implications.…”

Large language models as zero-shot evaluators of English–Chinese interpreting: A comparison of GPT-4o and DeepSeek-R1

“…Another widely adopted method is rubric-based scoring (Shafiei, 2024). More recently, comparative judgment has been introduced as an alternative approach (Han & Xiao, 2022), where a group of judges conduct repeated and randomized pairwise comparisons of interpreted renditions, selecting the higher-quality version in each comparison (Sickinger et al, 2025).…”