Semi-automated fact-checking of nucleotide sequence reagents in biomedical research publications: The Seek & Blastn tool

Abstract

Nucleotide sequence reagents are verifiable experimental reagents in biomedical publications, because their sequence identities can be independently verified and compared with associated text descriptors. We have previously reported that incorrectly identified nucleotide sequence reagents are characteristic of highly similar human gene knockdown studies, some of which have been retracted from the literature on account of possible research fraud. Because of the throughput limitations of manual verification of nucleotide sequences, we developed a semi-automated fact checking tool, Seek & Blastn, to verify the targeting or non-targeting status of published nucleotide sequence reagents. From previously described and unknown corpora of 48 and 155 publications, respectively, Seek & Blastn correctly extracted 304/342 (88.9%) and 1066/1522 (70.0%) nucleotide sequences and a predicted targeting/ non-targeting status. Seek & Blastn correctly predicted the targeting/ non-targeting status of 293/304 (96.4%) and 988/1066 (92.7%) of the correctly extracted nucleotide sequences. A total of 38/39 (97.4%) or 31/79 (39.2%) Seek & Blastn predictions of incorrect nucleotide sequence reagent use were correct in the two literature corpora. Combined Seek & Blastn and manual analyses identified a list of 91 misidentified nucleotide sequence reagents, which could be built upon through future studies. In summary, incorrect nucleotide sequence reagents represent an under-recognized source of error within the biomedical literature, and fact checking tools such as Seek & Blastn may help to identify papers and manuscripts affected by these errors.

Digital magic, or the dark arts of the 21stcentury—how can journals and peer reviewers detect manuscripts and publications from paper mills?

“…). We have reported highly similar papers studying the functions of human genes in cancer cell lines that describe wrongly identified nucleotide sequence reagents . These reagents are not affected by minor typographic errors, but instead show genetic identities that are completely different from their text descriptions (Figs and ).…”

“…Explanation of Seek & Blastn output for a retracted publication that described the effects of TPD52L2 gene knockdown in gastric cancer cell lines. Columns (left to right) are as follows: ‘Tested file’; ‘Nearest dist’, intertextual distance description, and PubMed identifier of the reference paper with the closest intertextual distance to the test publication; ‘Genes’, gene and species identifiers extracted by Seek & Blastn; ‘Cont.…”

“…However, cell line identifiers resemble nucleotide sequences in that the biological identity of a given cell line may not be obvious from its identifier, and in some cases, different cell lines can share very similar identifiers [19]. In a high-throughput setting, manuscript writers could copy identifiers corresponding to misidentified or A B [24] results (green, yellow, or red hypertext) [11]. The test publication [22] is part of the Seek & Blastn reference publication cohort [10,11], and its nearest text match is therefore the same publication.…”

“…In a high-throughput setting, manuscript writers could copy identifiers corresponding to misidentified or A B [24] results (green, yellow, or red hypertext) [11]. The test publication [22] is part of the Seek & Blastn reference publication cohort [10,11], and its nearest text match is therefore the same publication. Two nucleotide sequences represented previously reported sequences (Seq A and Seq D) [10].…”

Flagging incorrect nucleotide sequence reagents in biomedical papers: To what extent does the leading publication format impede automatic error detection?

“…Although individuals manage to visually screen biomedical images from thousands of papers (Baker 2016;Christopher 2018;Van Noorden 2015), the enormous worldwide scientific production calls for automated approaches relying on machine learning, as described by Acuna et al (2018), for instance. These automated approaches include the detection of plagiarism (Citron and Ginsparg 2015), p-value hacking (Nuijten et al 2016), and discrepancies between the claims presented and current knowledge on the topic (Labbé et al 2019).…”