BackgroundFindings from randomized trials (RCTs) on cervical pessary treatment to prevent spontaneous preterm birth are inconsistent.ObjectivesOur hypothesis suggests that adhering to the European Medical Device Regulation (MDR) and following the instructions for use are essential prerequisites for successful therapy. Conversely, the non‐adherence to these guidelines will probably contribute to its failure.Search Strategy and Selection CriteriaBased on validated criteria from integrity assessments we performed a systematic review identifying 14 RCTs evaluating the effect of cervical pessaries.Data Collection and AnalysisWe analyzed the implications of 14 criteria each accounting for 0–2 points of a score reflecting the clinical evaluation plan (CEP) as proposed by the MDR to evaluate the risk–benefit ratio of medical devices.Main ResultsSeven RCTs in each singleton and twin pregnancies (5193 “cases”) were included, detecting a high heterogeneity within control groups (I2 = 85% and 87%, respectively, P < 0.01). The CEP score varied from 11 to 26 points for all studies. The most common reasons for low scores and potential data compromise were poor recruitment rates, no (completed) power analysis, and no pre‐registration, but mainly non‐adherence to technical, biological, and clinical equivalence to the instructions for use as required by the MDR. All trials with score values greater than 20 had applied audit procedures. Within this group we found significantly reduced rates of spontaneous preterm birth at less than 34 weeks within the pessary group in singleton (odds ratio 0.28; 95% confidence interval 0.12–0.65) and twin pregnancies (odds ratio 0.30; 95% confidence interval 0.13–0.67). Similarly, there was a significant reduction in the composite poor neonatal outcome in singleton (odds ratio 0.25; 95% confidence interval 0.10–0.61) and twin pregnancies (odds ratio 0.54; 95% confidence interval 0.35–0.82) after a pessary as compared with controls.ConclusionNon‐audited RCTs and meta‐analyses mixing studies of different clinical quality as pre‐defined by a CEP and the MDR pose the risk for erroneous conclusions.