Flagging incorrect nucleotide sequence reagents in biomedical papers: To what extent does the leading publication format impede automatic error detection?

Abstract

In an idealised vision of science the scientific literature is error-free. Errors reported during peer review are supposed to be corrected prior to publication, as further research establishes new knowledge based on the body of literature. It happens, however, that errors pass through peer review, and a minority of cases errata and retractions follow. Automated screening software can be applied to detect errors in manuscripts and publications. The contribution of this paper is twofold. First, we designed the erroneous reagent checking (ERC) benchmark to assess the accuracy of fact-checkers screening biomedical publications for dubious mentions of nucleotide sequence reagents. It comes with a test collection comprised of 1679 nucleotide sequence reagents that were curated by biomedical experts. Second, we benchmarked our own screening software called Seek&Blastn with three input formats to assess the extent of performance loss when operating on various publication formats. Our findings stress the superiority of markup formats (a 79% detection rate on XML and HTML) over the prominent PDF format (a 69% detection rate at most) regarding an error flagging task. This is the first published baseline on error detection involving reagents reported in biomedical scientific publications. The ERC benchmark is designed to facilitate the development and validation of software bricks to enhance the reliability of the peer review process.

Identification of human gene research articles with wrongly identified nucleotide sequences

“…The 100 flagged C + G articles were published between 2008 and 2019 across 48 journals ( Table 1 ) and referred to a median of 2 (range 0–4) human genes across their titles ( ). The 100 flagged C + G articles examined 13 human cancer types, where most (96/100, 96%) examined a single cancer type, typically pancreatic (35/100, 35%) or lung (22/100, 22%) cancer ), reflecting the clinical use of cisplatin and gemcitabine ( 27 , 28 ). Most (90/100, 90%) C + G articles were published by authors from China, where most (82/90, 91%) were also affiliated with hospitals ( Table 1 and ).…”

“…Although there have been previous reports of wrongly identified PCR primers and gene knockdown reagents in single articles or small cohorts ( 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 21 ), the present study is the first to systematically fact-check the identities of nucleotide sequences in more than 3,400 research articles. Our supported application of S&B ( 12 , 27 ) ( https://www.protocols.io/view/seek-amp-blastn-standard-operating-procedure-bjhpkj5n ) to screen three targeted corpora and two journals identified 712 articles published across 78 journals that described more than 1,500 wrongly identified sequences. These problematic articles have received >17,000 citations, including citations by human clinical trials, where approximately one quarter of problematic articles could misinform the future development of human therapies.…”

“…Our discovery of frequent wrongly identified nucleotide sequence reagents in human gene research articles led us to develop a semi-automated tool Seek & Blastn (S&B) to fact-check the reported identities of nucleotide sequence reagents in human research articles ( 12 , 27 ). The S&B tool scans text to identify and extract nucleotide sequences and their associated text descriptors, submits extracted nucleotide sequences to Blastn analysis ( 28 ) to predict their genetic identities and hence their targeting or non-targeting status, and then compares the predicted status of each nucleotide sequence reagent with the claimed status within the text ( 12 ).…”

The thin ret(raction) line: biomedical journal responses to incorrect non-targeting nucleotide sequence reagents in human gene knockdown publications

“…Our subsequent notifications of these and other papers with incorrect non-targeting controls mean that journal responses are more likely to concern papers with this error type. Thirdly, subsequent analyses suggest that incorrect non-targeting reagents represent a robustly detected error type (Labbé et al 2020), as such reagents have "gained" a gene targeting function that is unambiguous and therefore easy to detect (Fig. 1b, c).…”

Recommendations from the Reducing the Inadvertent Spread of Retracted Science: Shaping a Research and Implementation Agenda Project

“…Issues pertaining to image data reporting are highly represented in retractions (Bik et al, 2016), and have received significant attention (Abbott, 2019;Bik et al, 2016, as have straightforwardly plagiarized text Foltýnek et al, 2019) and statistically impossible or improbable data (Carlisle, 2012(Carlisle, , 2017Pandit, 2012). Factors such as data availability (Dafoe, 2014) and publication file formats (Labbé et al, 2020) also impact the ease of detection. By contrast, another well-known problem, misidentified cell lines, is described as widespread, but rarely results in retractions Horbach & Halffman, 2017;.…”