Although end-to-end neural text-to-speech (TTS) methods (such as Tacotron2) are proposed and achieve state-of-theart performance, they still suffer from two problems: 1) low efficiency during training and inference; 2) hard to model long dependency using current recurrent neural networks (RNNs). Inspired by the success of Transformer network in neural machine translation (NMT), in this paper, we introduce and adapt the multi-head attention mechanism to replace the RNN structures and also the original attention mechanism in Tacotron2. With the help of multi-head self-attention, the hidden states in the encoder and decoder are constructed in parallel, which improves training efficiency. Meanwhile, any two inputs at different times are connected directly by a self-attention mechanism, which solves the long range dependency problem effectively. Using phoneme sequences as input, our Transformer TTS network generates mel spectrograms, followed by a WaveNet vocoder to output the final audio results. Experiments are conducted to test the efficiency and performance of our new network. For the efficiency, our Transformer TTS network can speed up the training about 4.25 times faster compared with Tacotron2. For the performance, rigorous human tests show that our proposed model achieves state-of-the-art performance (outperforms Tacotron2 with a gap of 0.048) and is very close to human quality (4.39 vs 4.44 in MOS).

No abstract

Current knowledge about the relationships between ruminal bacterial communities and metabolite profiles in the yak rumen is limited. This is due to differences in the nutritional and metabolic features between yak and other ordinary cattle combined with difficulties associated with farm-based research and a lack of technical guidance. A comprehensive analysis of the composition and alterations in ruminal metabolites is required to advance the development of modern yak husbandry. In the current study, we characterized the effect of feed type on the ruminal fluid microbiota and metabolites in yak using 16S rRNA gene sequencing and liquid chromatography-mass spectrometry (LC-MS). Bacteroidetes and Firmicutes were the predominant bacterial phyla in the yak rumen. At the genus level, the relative abundance of Bacteroidales BS11 gut group, Prevotellaceae UCG-003, Ruminococcaceae UCG-011, Bacteroidales RF16 group and Ruminococcaceae UCG-010 was significantly ( P < 0.01) higher in the forage group compared to that in the concentrate group, while the concentrate group harbored higher proportions of Bacteroidales S24-7 group, Ruminococcaceae NK4A214, Succiniclasticum and Ruminococcus 2 . Yak rumen metabolomics analysis combined with enrichment analysis revealed that feed type altered the concentrations of ruminal metabolites as well as the metabolic pattern, and significantly ( P < 0.01) affected the concentrations of ruminal metabolites involved in protein digestion and absorption (e.g., L-arginine, ornithine, L-threonine, L-proline and β-alanine), purine metabolism (e.g., xanthine, hypoxanthine, deoxyadenosine and deoxyadenosine monophosphate) and fatty acid biosynthesis (e.g., stearic acid, myristic acid and arachidonic acid). Correlation analysis of the association of microorganisms with metabolite features provides us with a comprehensive understanding of the composition and function of microbial communities. Associations between utilization or production were widely identified between affected microbiota and certain metabolites, and these findings will contribute to the direction of future research in yak.

Benchmark datasets have a significant impact on accelerating research in programming language tasks. In this paper, we introduce CodeXGLUE, a benchmark dataset to foster machine learning research for program understanding and generation. CodeXGLUE includes a collection of 10 tasks across 14 datasets and a platform for model evaluation and comparison. CodeXGLUE also features three baseline systems, including the BERT-style, GPT-style, and Encoder-Decoder models, to make it easy for researchers to use the platform. The availability of such data and baselines can help the development and validation of new methods that can be applied to various program understanding and generation problems 1 .

Effective HER activity on layered double hydroxide nanosheets through a plasma etching strategy.

This paper proposes a novel hierarchical recurrent neural network language model (HRNNLM) for document modeling. After establishing a RNN to capture the coherence between sentences in a document, HRNNLM integrates it as the sentence history information into the word level RNN to predict the word sequence with cross-sentence contextual information. A two-step training approach is designed, in which sentence-level and word-level language models are approximated for the convergence in a pipeline style. Examined by the standard sentence reordering scenario, HRNNLM is proved for its better accuracy in modeling the sentence coherence. And at the word level, experimental results also indicate a significant lower model perplexity, followed by a practical better translation result when applied to a Chinese-English document translation reranking task.

Pre-trained models for programming language have achieved dramatic empirical improvements on a variety of code-related tasks such as code search, code completion, code summarization, etc. However, existing pre-trained models regard a code snippet as a sequence of tokens, while ignoring the inherent structure of code, which provides crucial code semantics and would enhance the code understanding process. We present GraphCodeBERT, a pre-trained model for programming language that considers the inherent structure of code. Instead of taking syntacticlevel structure of code like abstract syntax tree (AST), we use data flow in the pre-training stage, which is a semantic-level structure of code that encodes the relation of "where-the-value-comes-from" between variables. Such a semantic-level structure is neat and does not bring an unnecessarily deep hierarchy of AST, the property of which makes the model more efficient. We develop GraphCodeBERT based on Transformer. In addition to using the task of masked language modeling, we introduce two structure-aware pre-training tasks. One is to predict code structure edges, and the other is to align representations between source code and code structure. We implement the model in an efficient way with a graph-guided masked attention function to incorporate the code structure. We evaluate our model on four tasks, including code search, clone detection, code translation, and code refinement. Results show that code structure and newly introduced pre-training tasks can improve GraphCodeBERT and achieves state-of-the-art performance on the four downstream tasks. We further show that the model prefers structure-level attentions over token-level attentions in the task of code search.

Non-task oriented dialogue systems have achieved great success in recent years due to largely accessible conversation data and the development of deep learning techniques. Given a context, current systems are able to yield a relevant and fluent response, but sometimes make logical mistakes because of weak reasoning capabilities. To facilitate the conversation reasoning research, we introduce Mu-Tual, a novel dataset for Multi-Turn dialogue Reasoning, consisting of 8,860 manually annotated dialogues based on Chinese student English listening comprehension exams. Compared to previous benchmarks for non-task oriented dialogue systems, MuTual is much more challenging since it requires a model that can handle various reasoning problems. Empirical results show that state-of-the-art methods only reach 71%, which is far behind the human performance of 94%, indicating that there is ample room for improving reasoning ability. MuTual is available at https://github. com/Nealcly/MuTual. * Contribution during internship at MSRA. M: Ma'am, you forgot your phone. F: Oh, thanks, I couldn't live without this little thing. M: I know what you mean. It is of great significance to you. So did you enjoy your dinner? F: Oh yes, everything was just perfect. It's so hard to take the whole family out to eat, but your restaurant was perfect. Johnny had his own place to play in and I had time to talk with my sisters and their husbands. ✓ (A) M: Thanks for your compliment for the restaurant. ✘ (B) M: I'm sorry that you don't have a good time. ✘ (C) M: Goodbye brother! Love you. ✘ (D) M: Hurry up honey, or we will be late for the dinner.