“…We can see that the exact match (EM) score is 80.567, and the F1 score is 88.117. {"exact": 80.56764427625355, "f1": 88.11721947565059, "total": 10570, "HasAns_exact": 80.56764427625355, "HasAns_f1": 88.11721947565059, "HasAns_total": 10570} Note: As the main goal of our previous work ( Cui et al., 2022 ) was to provide robust and comprehensive analyses of machine reading comprehension models, we carried out each experiment five times with different random seeds, and their average scores were used. However, to minimize the training time, we only train one model in this protocol, and it can be easily generalized to multiple runs as well by running steps 1–3 multiple times.…”
Section: Step-by-step Methods Detailsmentioning
confidence: 99%
“…In this context, we propose to visualize the attention by using a multilingual and multi-aspect way to comprehensively understand whether these attentions can be explainable ( Cui et al., 2022 ). Instead of analyzing the attention matrix as a whole, we decompose the attention matrix into four different attention zones to explicitly analyze their behaviors.…”
Section: Before You Beginmentioning
confidence: 99%
“…This approach can be generalized to other pretrained language models. For complete details on the use and execution of this protocol, please refer to Cui et al. (2022) .…”
“…We can see that the exact match (EM) score is 80.567, and the F1 score is 88.117. {"exact": 80.56764427625355, "f1": 88.11721947565059, "total": 10570, "HasAns_exact": 80.56764427625355, "HasAns_f1": 88.11721947565059, "HasAns_total": 10570} Note: As the main goal of our previous work ( Cui et al., 2022 ) was to provide robust and comprehensive analyses of machine reading comprehension models, we carried out each experiment five times with different random seeds, and their average scores were used. However, to minimize the training time, we only train one model in this protocol, and it can be easily generalized to multiple runs as well by running steps 1–3 multiple times.…”
Section: Step-by-step Methods Detailsmentioning
confidence: 99%
“…In this context, we propose to visualize the attention by using a multilingual and multi-aspect way to comprehensively understand whether these attentions can be explainable ( Cui et al., 2022 ). Instead of analyzing the attention matrix as a whole, we decompose the attention matrix into four different attention zones to explicitly analyze their behaviors.…”
Section: Before You Beginmentioning
confidence: 99%
“…This approach can be generalized to other pretrained language models. For complete details on the use and execution of this protocol, please refer to Cui et al. (2022) .…”
“…For the explainability studies in MRC, [26] propose a method to extract evidence sentences from multi-choice MRC tasks. [27] propose to use system performance rather than visualizing attention score to better reveal the model's explainability. [28] investigate a few black-box attacks at the character, word, and sentence level for MRC systems.…”
“…Transformer Pruning Previous studies (Michel et al, 2019;Voita et al, 2019) have shown that not all attention heads are equally important in the transformers, and some of the attention heads can be pruned without performance loss (Cui et al, 2022). Thus, Identifying and removing the least important attention heads can reduce the model size and have a small impact on performance.…”
Pre-trained language models have been prevailed in natural language processing and become the backbones of many NLP tasks, but the demands for computational resources have limited their applications. In this paper, we introduce TextPruner, an open-source model pruning toolkit designed for pre-trained language models, targeting fast and easy model compression. TextPruner offers structured post-training pruning methods, including vocabulary pruning and transformer pruning, and can be applied to various models and tasks. We also propose a self-supervised pruning method that can be applied without the labeled data. Our experiments with several NLP tasks demonstrate the ability of TextPruner to reduce the model size without re-training the model. 1
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
