Examining Bias and Reporting in Oral Health Prediction Modeling Studies

Abstract

Recent efforts to improve the reliability and efficiency of scientific research have caught the attention of researchers conducting prediction modeling studies (PMSs). Use of prediction models in oral health has become more common over the past decades for predicting the risk of diseases and treatment outcomes. Risk of bias and insufficient reporting present challenges to the reproducibility and implementation of these models. A recent tool for bias assessment and a reporting guideline—PROBAST (Prediction Model Risk of Bias Assessment Tool) and TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis)—have been proposed to guide researchers in the development and reporting of PMSs, but their application has been limited. Following the standards proposed in these tools and a systematic review approach, a literature search was carried out in PubMed to identify oral health PMSs published in dental, epidemiologic, and biostatistical journals. Risk of bias and transparency of reporting were assessed with PROBAST and TRIPOD. Among 2,881 papers identified, 34 studies containing 58 models were included. The most investigated outcomes were periodontal diseases (42%) and oral cancers (30%). Seventy-five percent of the studies were susceptible to at least 4 of 20 sources of bias, including measurement error in predictors ( n = 12) and/or outcome ( n = 7), omitting samples with missing data ( n = 10), selecting variables based on univariate analyses ( n = 9), overfitting ( n = 13), and lack of model performance assessment ( n = 24). Based on TRIPOD, at least 5 of 31 items were inadequately reported in 95% of the studies. These items included sampling approaches ( n = 15), participant eligibility criteria ( n = 6), and model-building procedures ( n = 16). There was a general lack of transparent reporting and identification of bias across the studies. Application of the recommendations proposed in PROBAST and TRIPOD can benefit future research and improve the reproducibility and applicability of prediction models in oral health.

Reporting Quality of Studies Developing and Validating Melanoma Prediction Models: An Assessment Based on the TRIPOD Statement

“…Even though this analysis related to a special segment of prediction studies, our finding of inadequate reporting in the field of melanoma risk prediction is comparable to other studies [ 64 , 65 , 66 , 67 ]. Heus et al [ 65 ] included 146 publications across 37 clinical domains and reported a median TRIPOD adherence of 44%, which is even lower than the median of the studies included in our analysis (57%).…”

A critical appraisal of the prognostic predictive models for patients with sepsis: Which model can be applied in clinical practice?

“…Of note in the studies of other diseases consulted 13,59‐63 was that analysis was the domain that presented the greatest risk of bias, as in various reviews either no or very few studies among those included did this correctly. The other domains, however, both for risk of bias and applicability, were quite satisfactory, surpassing 70% in almost all the reviews.…”

“…Among the 14 articles included, only one had eight questions assessed positively (the article failed to indicate how it handled missing data, unclear), 48 another had seven questions assessed satisfactorily while the other two were negative (missing data from combining the missing data with the reference category and using univariate analysis to select the model predictors), 58 and three articles had six positive ques- did not adequately handle continuous predictors, used univariate analysis to select model predictors and we could not assess missing data 54 and (3) Lagu et al (2011): did not take into account overfitting, and there were two items for which we had no information for their assessment, the first being missing data and the second the questionable use of all patients to develop the model since when making exclusions after starting the follow-up, this was unclear. 57 Of note in the studies of other diseases consulted 13,[59][60][61][62][63] was that analysis was the domain that presented the greatest risk of bias, as in various reviews either no or very few studies among those included did this correctly. The other domains, however, both for risk of bias and applicability, were quite satisfactory, surpassing 70% in almost all the reviews.…”

“…When comparing our results (risk of bias and applicability), we had to analyse systematic reviews with similar methodologies that used PROBAST, and we found great variability in patient types and outcomes. 13,[59][60][61][62][63] The percentages of low risk and low concern regarding applicability of all these studies are indicated in Table 3.…”

Comparison of the Tree-Based Machine Learning Algorithms to Cox Regression in Predicting the Survival of Oral and Pharyngeal Cancers: Analyses Based on SEER Database

“…These clinical decision support tools are all developed based on clinical prediction modelling research, which aims to yield the most accurate outcome prediction by capturing patterns in the available data (known as data-generating mechanisms) and minimizing the difference between the predicted and observed outcome (known as bias). However, following the up-to-date bias assessment criteria (PROBAST-Prediction model Risk Of Bias ASsessment Tool) [ 6 ] and reporting guidelines (TRIPOD-Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis) [ 7 ], the overall quality of oral health prediction modelling studies was found to be less than optimal due to the presence of multiple sources of bias (e.g., measurement error, unmeasured predictors) and lack of reporting transparency [ 8 ].…”