The choice of token vocabulary affects the performance of machine translation. This paper aims to figure out what is a good vocabulary and whether one can find the optimal vocabulary without trial training. To answer these questions, we first provide an alternative understanding of the role of vocabulary from the perspective of information theory. Motivated by this, we formulate the quest of vocabularization -finding the best token dictionary with a proper size -as an optimal transport (OT) problem. We propose VOLT, a simple and efficient solution without trial training. Empirical results show that VOLT outperforms widely-used vocabularies in diverse scenarios, including WMT-14 English-German and TED's 52 translation directions. For example, VOLT achieves 70% vocabulary size reduction and 0.5 BLEU gain on English-German translation. Also, compared to BPE-search, VOLT reduces the search time from 384 GPU hours to 30 GPU hours on English-German translation. Codes are available at https: //github.com/Jingjing-NLP/VOLT.

In this paper, we investigate the problem of reasoning over natural language statements. Prior neural based approaches do not explicitly consider the inter-dependency among answers and their proofs. In this paper, we propose PROBR, a novel approach for joint answer prediction and proof generation. PROBR defines a joint probabilistic distribution over all possible proof graphs and answers via an induced graphical model. We then optimize the model using variational approximation on top of neural textual representation. Experiments on multiple datasets under diverse settings (fully supervised, few-shot and zero-shot evaluation) verify the effectiveness of PROBR, e.g., achieving 10%-30% improvement on QA accuracy in few/zero-shot evaluation. Our codes and models can be found at https://github.com/ changzhisun/PRobr/.

No abstract

It is well accepted that the choice of token vocabulary largely affects the performance of machine translation. However, due to expensive trial costs, most studies only conduct simple trials with dominant approaches (e.g BPE) and commonly used vocabulary sizes. In this paper, we find an exciting relation between an information-theoretic feature and BLEU scores. With this observation, we formulate the quest of vocabularization -finding the best token dictionary with a proper size -as an optimal transport problem. We then propose VOLT, a simple and efficient vocabularization solution without the full and costly trial training. We evaluate our approach on multiple machine translation tasks, including WMT-14 English-German translation, TED bilingual translation, and TED multilingual translation. Empirical results show that VOLT beats widely-used vocabularies on diverse scenarios. For example, VOLT achieves 70% vocabulary size reduction and 0.6 BLEU gain on English-German translation. Also, one advantage of VOLT lies in its low resource consumption. Compared to naive BPE-search, VOLT reduces the search time from 288 GPU hours to 0.5 CPU hours.1 Here BPE-32K refers to a vocabulary constructed by Byte Pair Encoding (BPE) with 32K tokens.

The Newlander-Nirenberg theorem says that a formally integrable complex structure is locally equivalent to the standard complex structure in the complex Euclidean space. In this paper, we consider two natural generalizations of the Newlander-Nirenberg theorem under the presence of a C 2 strictly pseudoconvex boundary. When a given formally integrable complex structure X is defined on the closure of a bounded strictly pseudoconvex domain with C 2 boundary D ⊂ C n , we show the existence of global holomorphic coordinate systems defined on D that transform X into the standard complex structure provided that X is sufficiently close to the standard complex structure. Moreover, we show that such closeness is stable under a small C 2 perturbation of ∂D. As a consequence, when a given formally integrable complex structure is defined on a one-sided neighborhood of some point in a C 2 real hypersurface M ⊂ C n , we prove the existence of local one-sided holomorphic coordinate systems provided that M is strictly pseudoconvex with respect to the given complex structure. We also obtain results when the structures are finite smooth.