Assessing the Reproducibility of Research Based on the Food and Drug Administration Manufacturer and User Facility Device Experience Data

Abstract

Objective This article aims to assess the reproducibility of Manufacturer and User Facility Device Experience (MAUDE) data-driven studies by analyzing the data queries used in their research processes. Methods Studies using MAUDE data were sourced from PubMed by searching for “MAUDE” or “Manufacturer and User Facility Device Experience” in titles or abstracts. We manually chose articles with executable queries. The reproducibility of each query was assessed by replicating it in the MAUDE Application Programming Interface. The reproducibility of a query is determined by a reproducibility coefficient that ranges from 0.95 to 1.05. This coefficient is calculated by comparing the number of medical device reports (MDRs) returned by the reproduced queries to the number of reported MDRs in the original studies. We also computed the reproducibility ratio, which is the fraction of reproducible queries in subgroups divided by the query complexity, the device category, and the presence of a data processing flow. Results As of August 8, 2022, we identified 523 articles from which 336 contained queries, and 60 of these were executable. Among these, 14 queries were reproducible. Queries using a single field like product code, product class, or brand name showed higher reproducibility (50%, 33.3%, 31.3%) compared with other fields (8.3%, P = 0.037). Single-category device queries exhibited a higher reproducibility ratio than multicategory ones, but without statistical significance (27.1% versus 8.3%, P = 0.321). Studies including a data processing flow had a higher reproducibility ratio than those without, although this difference was not statistically significant (42.9% versus 17.4%, P = 0.107). Conclusions Our findings indicate that the reproducibility of queries in MAUDE data-driven studies is limited. Enhancing this requires the development of more effective MAUDE data query strategies and improved application programming interfaces.