Pre-trained Transformer-based language models have become a key building block for natural language processing (NLP) tasks. While these models are extremely accurate, they can be too large and computationally intensive to run on standard deployments. A variety of compression methods, including distillation, quantization, structured and unstructured pruning are known to be applicable to decrease model size and increase inference speed. In this context, this paper's contributions are two-fold. We begin with an in-depth study of the accuracycompression trade-off for unstructured weight pruning in the context of BERT models, and introduce Optimal BERT Surgeon (O-BERT-S), an efficient and accurate weight pruning method based on approximate second-order information, which we show to yield state-of-theart results in terms of the compression/accuracy trade-off. Specifically, Optimal BERT Surgeon extends existing work on second-order pruning by allowing for pruning blocks of weights, and by being applicable at BERT scale. Second, we investigate the impact of this pruning method when compounding compression approaches for Transformer-based models, which allows us to combine state-of-the-art structured and unstructured pruning together with quantization, in order to obtain highly compressed, but accurate models. The resulting compression framework is powerful, yet general and efficient: we apply it to both the fine-tuning and pre-training stages of language tasks, to obtain state-of-the-art results on the accuracycompression trade-off with relatively simple compression recipes. For example, we obtain 10x model size compression with < 1% relative drop in accuracy to the dense BERT-base, 10x end-to-end CPU-inference speedup with < 2% relative drop in accuracy, and 29x inference speedups with < 7.5% relative accuracy drop.
A very high early strength portland cement-based concrete (VHES) is used to prevent long closures of highway lanes by completing any necessary repairs overnight. This technology has been embraced by several DOT agencies, including those in areas where the concrete will be subjected to freezing and thawing cycles. Flexural strength of VHES concrete is the critical parameter to permit traffic flow without damage to the concrete. It is possible to produce air-entrained rapid repair concrete, but this places higher requirements on the mixture proportions to overcome strength reduction caused by the presence of air. This article presents data showing that it is possible to attain the necessary strength and achieve adequate resistance to freezing and thawing cycles. The evidence presented includes accelerated testing using ASTM C 666, Procedure A, and scaling resistance using ASTM C 672. Additional evidence is presented showing that VHES concrete of low w/c can be durable with larger than acceptable spacing factor levels or without air entrainment.
This paper focuses on evaluating the impact of friction and contact pressure on helical steel tubes. The initial gaps between steel tubes and adjacent layers, friction coefficients and the contact stiffness are the main factors that affect such investigation. A novel meth odology by using UFLEX2D (a MARINTEK product) has been applied for modeling com plex umbilical cross sections and for the study of these parameters. Two cross sections for the same subsea application but with different designs have been investigated in the study. It has been shown how fatigue damage can be significantly impacted by different cross-sectional design. For this study, nonlinear moment/curvature relationship has been included in the analyses. Based on the findings of this study, more realistic results can be achieved by including the nonlinear behavior in global analysis for fatigue damage cal culations instead of using nominal bending stiffness supplied by umbilical manufacturer.
No abstract
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
customersupport@researchsolutions.com
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.