CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark

Chen, Mosha; Bi, Zhongnan; Liang, Xiaozhuan; Li, Lei; Shang, Xin; Kangping, Yin,; Tan, Chuanqi; Xu, Jing; Huang, Fei; Si, Luo; Yuan, Naichang; Sui, Zhifang; Chang, Baobao; Zong, Hui; Zheng, Yi; Li, Linfeng; Yan, Jun; Zan, Hongying; Zhang, Kunli; Tang, Buzhou; Chen, Qingcai

doi:10.48550/arxiv.2106.08087

Cited by 14 publications

(18 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to evaluate the effectiveness of the model on the NER and FRE task, we selected the NER dataset CMeEE [8] and the FRE dataset CMeEE [9] from the public benchmark CBLUE [38]. For each task, we selected the SOTA methods to compare with our model.…”

Section: Experiments Results On Ner Task and Fre Taskmentioning

confidence: 99%

A Unified Knowledge Extraction Method Based on BERT and Handshaking Tagging Scheme

Yang

Pun²,

Vai

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

In the actual knowledge extraction system, different applications have different entity classes and relationship schema, so the generalization and migration ability of knowledge extraction are very important. By training a knowledge extraction model in the source domain and applying the model to an arbitrary target domain directly, open domain knowledge extraction technology becomes crucial to mitigate the generalization and migration ability issues. Traditional knowledge extraction models cannot be directly transferred to new domains and also cannot extract undefined relation types. In order to deal with the above issues, in this paper, we proposed an end-to-end Chinese open-domain knowledge extraction model, TPORE (Extract Open-domain Relations through Token Pair linking), which combined BERT with a handshaking tagging scheme. TPORE can alleviate the nested entities and nested relations issues. Additionally, a new loss function that conducts a pairwise comparison of target category score and non-target category score to automatically balance the weight was adopted, and the experiment results indicate that the loss function can bring speed and performance improvements. The extensive experiments demonstrate that the proposed method can significantly surpass strong baselines. Specifically, our approach can achieve new state-of-the-art Chinese open Relation Extraction (ORE) benchmarks (COER and SAOKE). In the COER dataset, F1 increased from 66.36% to 79.63%, and in the SpanSAOKE dataset, F1 increased from 46.0% to 54.91%. In the medical domain, our method can obtain close performance compared with the SOTA method in the CMeIE and CMeEE datasets.

show abstract

Section: Experiments Results On Ner Task and Fre Taskmentioning

confidence: 99%

A Unified Knowledge Extraction Method Based on BERT and Handshaking Tagging Scheme

Yang

Pun²,

Vai

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…The goal of intent detection is to identify query intent and classify them into specific categories (Chen et al, 2012;Howard and Cambria, 2013;Guo et al, 2014;Cai et al, 2017). With artificial intelligence gradually changing the landscape of healthcare and biomedical research (Yu et al, 2018), medical intent detection (Zhang et al, 2021;Chen et al, 2020a) becomes an important task. In medical domain, query intent can be divided into many categories, such as disease description, medical fees, treatment plan, precautions, and so on, which are domain-specific with highly specialized medical knowledge (Zhang et al, 2021).…”

Section: Medical Intent Detectionmentioning

confidence: 99%

“…With artificial intelligence gradually changing the landscape of healthcare and biomedical research (Yu et al, 2018), medical intent detection (Zhang et al, 2021;Chen et al, 2020a) becomes an important task. In medical domain, query intent can be divided into many categories, such as disease description, medical fees, treatment plan, precautions, and so on, which are domain-specific with highly specialized medical knowledge (Zhang et al, 2021). Understanding medical can assist medical question answer systems and significantly improve the relevance of medical search results (Wu et al, 2020;Mrini et al, 2021).…”

Section: Medical Intent Detectionmentioning

confidence: 99%

Incremental Intent Detection for Medical Domain with Contrast Replay Networks

Bai¹,

He²,

Liu³

et al. 2022

Findings of the Association for Computational Linguistics: ACL 2022

View full text Add to dashboard Cite

Conventional approaches to medical intent detection require fixed pre-defined intent categories. However, due to the incessant emergence of new medical intents in the real world, such requirement is not practical. Considering that it is computationally expensive to store and re-train the whole data every time new data and intents come in, we propose to incrementally learn emerged intents while avoiding catastrophically forgetting old intents. We first formulate incremental learning for medical intent detection. Then, we employ a memorybased method to handle incremental learning. We further propose to enhance the method with contrastive replay networks, which use multilevel distillation and contrastive objective to address training data imbalance and medical rare words respectively. Experiments show that the proposed method outperforms the state-of-the-art model by 5.7% and 9.1% of accuracy on two benchmarks respectively.

show abstract

“…All the above-discussed works refer to tasks in English. Recently similar works appeared for other languages, for example, a Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark [26]. However, that cannot be said for Russian, where only a few separate datasets exist.…”

Section: Related Workmentioning

confidence: 99%

RuMedBench: A Russian Medical Language Understanding Benchmark

Blinov,

Reshetnikova,

Nesterov

et al. 2022

Preprint

View full text Add to dashboard Cite

The paper describes the open Russian medical language understanding benchmark covering several task types (classification, question answering, natural language inference, named entity recognition) on a number of novel text sets. Given the sensitive nature of the data in healthcare, such a benchmark partially closes the problem of Russian medical dataset absence. We prepare the unified format labeling, data split, and evaluation metrics for new tasks. The remaining tasks are from existing datasets with a few modifications. A single-number metric expresses a model's ability to cope with the benchmark. Moreover, we implement several baseline models, from simple ones to neural networks with transformer architecture, and release the code. Expectedly, the more advanced models yield better performance, but even a simple model is enough for a decent result in some tasks. Furthermore, for all tasks, we provide a human evaluation. Interestingly the models outperform humans in the large-scale classification tasks. However, the advantage of natural intelligence remains in the tasks requiring more knowledge and reasoning.

show abstract

CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark

Cited by 14 publications

References 0 publications

A Unified Knowledge Extraction Method Based on BERT and Handshaking Tagging Scheme

A Unified Knowledge Extraction Method Based on BERT and Handshaking Tagging Scheme

Incremental Intent Detection for Medical Domain with Contrast Replay Networks

RuMedBench: A Russian Medical Language Understanding Benchmark

Contact Info

Product

Resources

About