Recently pre-trained models have achieved state-of-the-art results in various language understanding tasks. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring information, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entities, semantic closeness and discourse relations. In order to extract the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which incrementally builds pre-training tasks and then learn pre-trained models on these constructed tasks via continual multi-task learning. Based on this framework, we construct several tasks and train the ERNIE 2.0 model to capture lexical, syntactic and semantic aspects of information in the training data. Experimental results demonstrate that ERNIE 2.0 model outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several similar tasks in Chinese. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.

We present a novel language representation model enhanced by knowledge called ERNIE (Enhanced Representation through kNowledge IntEgration). Inspired by the masking strategy of BERT (Devlin et al., 2018), ERNIE is designed to learn language representation enhanced by knowledge masking strategies, which includes entity-level masking and phrase-level masking. Entity-level strategy masks entities which are usually composed of multiple words. Phrase-level strategy masks the whole phrase which is composed of several words standing together as a conceptual unit. Experimental results show that ERNIE outperforms other baseline methods, achieving new state-of-the-art results on five Chinese natural language processing tasks including natural language inference, semantic similarity, named entity recognition, sentiment analysis and question answering. We also demonstrate that ERNIE has more powerful knowledge inference capacity on a cloze test.

Experimental SectionMeasurements. 1 H NMR (500 MHz) spectra were acquired in CDCl 3 using a Varian INOVA-500 spectrometer at 25 °C. Tetramethylsilane (TMS) was used as an internal reference for 1 H NMR spectroscopy. FT-IR spectra were obtained on a Bruker Tensor 27 system using attenuated total reflectance (ATR) sampling accessories. High resolution mass spectrum was obtained on a ThermoFinnigan MAT XL spectrometer.Matrix-assisted laser desorption/ionization time-of-flight mass spectrometery (MALDI-TOF) was measured on a Bruker Biflex IV (Billerica, MA) MALDI mass spectrometer equipped with a nitrogen laser (λ = 337 nm). Mass spectrum was acquired in the reflection mode with a mass range of 2000-12000 m/z, and the mass scale was calibrated externally using the peaks of peptide calibration standard II purchased from Bruker. Trans-2-[3-(4-tert-butylphenyl)-2-methyl-2-propenylidene]malononitrile (DCTB, Aldrich; ≥99%) served as matrix and was dissolved in CHCl 3 at a concentration of 20 mg/mL. Sodium trifluoroacetate (NaTFA, Aldrich; ~98%) served as cationizing agent and was dissolved in MeOH/CHCl 3 (1/3, v/v) at a concentration of 10 mg/mL. The polymer was dissolved in CHCl 3 at a concentration of 10 mg/mL. The matrix solution, polymer solution, and NaTFA solution were mixed in the ratio of 10/1/1 (v/v/v). The sample preparation involved depositing 1 μL of the mixture on the steel plate, and allowing the spot to dry.Gel permeation chromatography (GPC) was conducted using Viscotek GPC system equipped with a VE-3580 refractive index (RI) detector, a 270 dual detector system having a viscometer detector and a dual-angle (7 º and 90 º) laser light scattering (LS) detector, a VE 1122 pump, and two mixed-bed organic columns (PAS-103M with exclusion limit of 70 kDa and PAS-105M with exclusion limit of 4 MDa). N,N'-dimethylformamide (HPLC grade) with 0.1 M

Functional polylactide-g-paclitaxel–poly(ethylene glycol), a novel graft polymer–drug conjugate (GPDC) with paclitaxel (PTXL) as the divalent agent to bridge between the degradable polylactide (PLA)-based backbone and hydrophilic poly(ethylene glycol) (PEG) side chains, were prepared by the copper-catalyzed azide–alkyne cycloaddition reaction of acetylene-functionalized polylactide (PLA) with azide-functionalized PTXL–PEG conjugate. The acetylene-functionalized PLA was prepared by ring-opening copolymerization (ROCP) of acetylene-functionalized LA monomer with l-lactide (LA). The azide-functionalized PTXL–PEG conjugate was prepared by multistep organic synthesis. The well-controlled chemical structures of the GPDC and its precursors were verified by 1H NMR and GPC characterizations. DLS analysis indicated that GPDC molecules assembled in water to form nanoparticles with sizes of 8–40 nm. GPC analysis of buffer solutions (pH = 5.5 and 7.4) of the GPDC suggested the occurrence of multiple hydrolysis reactions under the experimental conditions, which resulted in the release of PTXL moieties and the cleavage of PLA-based backbone.

Amphiphilic double-brush copolymers (DBCs) with each graft site quantitatively carrying both a hydrophilic poly(ethylene oxide) (PEO) graft and a hydrophobic polylactide (PLA) graft were synthesized, characterized, and further utilized as surfactants for the stabilization of miniemulsions. Well-defined PEO-b-PLA-based diblock macromonomers (MMs) with exo-norbornene (NB)-functionalized diblock junction were prepared by the synthesis of a PEO-based NB-functionalized alcohol via polymeric reaction, followed by ringopening polymerization (ROP) of lactide (LA) initiated by the alcohol. Ring-opening metathesis polymerization (ROMP) of the MMs yielded DBCs. The well-controlled structures of the MMs and the DBCs were verified through rigorous instrumental characterizations. As compared with the MMs, the corresponding DBCs had lower crystallinities and melting temperatures (T m s) for both PEO and PLA phases and showed a negligible tendency for intermolecular self-assembly in solutions. With nanoscopic dimensions and novel amphiphilic architectures, these DBCs represent a new type of giant polymeric surfactant. Relative to the precursor MMs, the DBCs resulted in miniemulsions with remarkably enhanced stability.