Breaking down the RECIST 1.1 double read variability in lung trials: What do baseline assessments tell us?

Abstract

BackgroundIn clinical trials with imaging, Blinded Independent Central Review (BICR) with double reads ensures data blinding and reduces bias in drug evaluations. As double reads can cause discrepancies, evaluations require close monitoring which substantially increases clinical trial costs. We sought to document the variability of double reads at baseline, and variabilities across individual readers and lung trials.Material and methodsWe retrospectively analyzed data from five BICR clinical trials evaluating 1720 lung cancer patients treated with immunotherapy or targeted therapy. Fifteen radiologists were involved. The variability was analyzed using a set of 71 features derived from tumor selection, measurements, and disease location. We selected a subset of readers that evaluated ≥50 patients in ≥two trials, to compare individual reader’s selections. Finally, we evaluated inter-trial homogeneity using a subset of patients for whom both readers assessed the exact same disease locations. Significance level was 0.05. Multiple pair-wise comparisons of continuous variables and proportions were performed using one-way ANOVA and Marascuilo procedure, respectively.ResultsAcross trials, on average per patient, target lesion (TL) number ranged 1.9 to 3.0, sum of tumor diameter (SOD) 57.1 to 91.9 mm. MeanSOD=83.7 mm. In four trials, MeanSOD of double reads was significantly different. Less than 10% of patients had TLs selected in completely different organs and 43.5% had at least one selected in different organs. Discrepancies in disease locations happened mainly in lymph nodes (20.1%) and bones (12.2%). Discrepancies in measurable disease happened mainly in lung (19.6%). Between individual readers, the MeanSOD and disease selection were significantly different (p<0.001). In inter-trials comparisons, on average per patient, the number of selected TLs ranged 2.1 to 2.8, MeanSOD 61.0 to 92.4 mm. Trials were significantly different in MeanSOD (p<0.0001) and average number of selected TLs (p=0.007). The proportion of patients having one of the top diseases was significantly different only between two trials for lung. Significant differences were observed for all other disease locations (p<0.05).ConclusionsWe found significant double read variabilities at baseline, evidence of reading patterns and a means to compare trials. Clinical trial reliability is influenced by the interplay of readers, patients and trial design.

Reproducing RECIST lesion selection via machine learning: Insights into intra and inter-radiologist variation

“…In other words, in situations where it is not clear how to measure a lesion, the radiologist may rely on their own experience, which will vary between radiologists, resulting in variability of measurements and, consequently, lesion selection. This phenomenon, where radiologists have their own reading pattern, dictated by differing experience, background, and interpretation of the guidelines, has been observed before by Iannessi and Beumount [6] in target lesion selection.…”

“…This observation could still be valid when considering that most of the disagreement will happen around cutoff values, where lesion size fails to provide a reproducible answer, while in general, the trend for larger lesions remains the same. There is a limited amount of research available on the variability of measurable lesion selection: Iannessi and Beaumont [6] found a noticeable discrepancy in differentiating between measurable and non-measurable disease, which was suggested to be linked to the small tumor burden.…”

Can we predict discordant RECIST 1.1 evaluations in double read clinical trials?

“…Based on our features set ( 14 ), and a pre-processing of feature selection for RF algorithm, classification performances for response-related KoDs are summarized in Table 5 . For each independent clinical trial or in pooled data, feature selection did not improve classification performances.…”

“…Comprehensive description of all the risk factors are provided in Annex A . By means of odds ratios (ODDs), we performed a univariate analysis testing associations between KoDs and a set of predefined features ( 14 ) ( Annex A ) derived from risk factors. These features applied to target lesions (TLs) and non-target lesions (NTLs) and were stratified according to the different diseased organs (See Annex B ).…”