Similarity corpus on microbial transcriptional regulation

Abstract

Background The ability to express the same meaning in different ways is a well-known property of natural language. This amazing property is the source of major difficulties in natural language processing. Given the constant increase in published literature, its curation and information extraction would strongly benefit from efficient automatic processes, for which corpora of sentences evaluated by experts are a valuable resource. Results Given our interest in applying such approaches to the benefit of curation of the biomedical literature, specifically that about gene regulation in microbial organisms, we decided to build a corpus with graded textual similarity evaluated by curators and that was designed specifically oriented to our purposes. Based on the predefined statistical power of future analyses, we defined features of the design, including sampling, selection criteria, balance, and size, among others. A non-fully crossed study design was applied. Each pair of sentences was evaluated by 3 annotators from a total of 7; the scale used in the semantic similarity assessment task within the Semantic Evaluation workshop (SEMEVAL) was adapted to our goals in four successive iterative sessions with clear improvements in the agreed guidelines and interrater reliability results. Alternatives for such a corpus evaluation have been widely discussed. Conclusions To the best of our knowledge, this is the first similarity corpus—a dataset of pairs of sentences for which human experts rate the semantic similarity of each pair—in this domain of knowledge. We have initiated its incorporation in our research towards high-throughput curation strategies based on natural language processing.

Protocol for a reproducible experimental survey on biomedical sentence similarity

“…On the other hand, the development of sentence similarity benchmarks for the biomedical domain is much more recent. Currently, there are only three datasets for the evaluation of methods on biomedical sentence similarity, called BIOSSES [ 20 ], MedSTS [ 49 ], and CTR [ 50 ]. BIOSSES was introduced in 2017 and it is limited to 100 sentence pairs with their corresponding similarity scores, whilst MedSTS full is made up by 1,068 scored sentence pairs of the MedSTS dataset [ 100 ], which contains 174,629 sentence pairs gathered from a clinical corpus on biomedical sentence similarity.…”

“…Fig 3 shows the workflow for running the experiments that will be carried out for this work. Given an input dataset, such as BIOSSES [ 20 ], MedSTS [ 49 ], or CTR [ 50 ], the first step is to pre-process all of the sentences, as shown in Fig 4 . For each sentence in the dataset (named S1 and S2), the preprocessing phase will be divided into four stages as follows: (1.a) named entity recognition of UMLS [ 120 ] concepts, using different state-of-the-art NER tools, such as MetaMap [ 107 ] or cTAKES [ 108 ]; (1.b) tokenize the sentence, using well-known tokenizers, such as the Stanford CoreNLP tokenizer [ 117 ], BioCNLPTokenizer [ 118 ], or WordPieceTokenizer [ 33 ] for BERT-based methods; (1.c) lower-case normalization; (1.d) character filtering, which allows the removal of punctuation marks or special characters; and finally, (1.e) the removal of stop-words, following different approximations evaluated by other authors like Blagec et al [ 28 ] or Sogancioglu et al [ 20 ].…”

“…A second motivation is the implementation of a set of unexplored methods which are based on adaptations from other methods proposed for the general language domain. A third motivation is the evaluation in the same software platform of the benchmarks on biomedical sentence similarity reported in the literature as follows: Biomedical Semantic Similarity Estimation System (BIOSSES) [ 20 ] and Medical Semantic Textual Similarity (MedSTS) [ 49 ] datasets, as well as the evaluation for the first time of the Microbial Transcriptional Regulation (CTR) [ 50 ] dataset in a sentence similarity task, despite it having been previously evaluated in other related tasks, such as the curation of gene expressions from scientific publications [ 51 ]. A fourth motivation is a study on the impact of the pre-processing stage and NER tools on the performance of the sentence similarity methods, such as that done by Gerlach et al [ 52 ] for stop-words in topic modeling task.…”

L-Regulon: A novel “soft-curation” approach supported by a semantic enriched reading for RegulonDB literature

“…The quality of the semantic-interlinks was evaluated indirectly by 303 measuring the precision of the similarity metrics. This was done through a 304 10-fold cross-validation over a Similarity Corpus [13], which is a graded 305 textual similarity corpus that was specifically designed to be used for 306 training similarity models.…”

“…10 Protein-protein interactions, regulatory interactions identification, entity 11 association to ontologies, or even directed searches, are just some of aided 12 curation examples [6,17,23]. It is important to emphasize that they are 13 focused in facilitating access to specific information patterns. This kind of 14 curation is vital and very helpful but it is designed on the premise that 15 what is searched fits in a set of predefined and foreseen model and criteria, 16 thus it only targets a fraction of the potential knowledge.…”