Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning

Tamborrino, Alexandre; Pellicanò, Nicola; Pannier, B.; Voitot, Pascal; Naudin, Louise

doi:10.48550/arxiv.2004.14074

Cited by 4 publications

(5 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The pretraining model learns a context-dependent representation of each member of an input sentence using almost unlimited text, and it implicitly learns general syntactic semantic knowledge. It can migrate knowledge learned from the open domain to downstream tasks to improve low-resource tasks, and is also very helpful for low-resource language processing [13].…”

Section: Pretrainingmentioning

confidence: 99%

Automatic Scoring Model of Subjective Questions Based Text Similarity Fusion Model

Xie

Chen

2022

Lecture Notes in Electrical Engineering

View full text Add to dashboard Cite

AI In this era, scene based translation and intelligent word segmentation are not new technologies. However, there is still no good solution for long and complex Chinese semantic analysis. The subjective question scoring still relies on the teacher's manual marking. However, there are a large number of examinations, and the manual marking work is huge. At present, the labor cost is getting higher and higher, the traditional manual marking method can't meet the demand The demand for automatic marking is increasingly strong in modern society. At present, the automatic marking technology of objective questions has been very mature and widely used. However, by reasons of the complexity and the difficulty of natural language processing technology in Chinese text, there are still many shortcomings in subjective questions marking, such as not considering the impact of semantics, word order and other issues on scoring accuracy. The automatic scoring technology of subjective questions is a complex technology, involving pattern recognition, machine learning, natural language processing and other technologies. Good results have been seen in the calculation method-based deep learning and machine learning. The rapid development of NLP technology has brought a new breakthrough for subjective question scoring. We integrate two deep learning models based on the Siamese Network through bagging to ensure the accuracy of the results, the text similarity matching model based on the birth networks and the score point recognition model based on the named entity recognition method respectively. Combining with the framework of deep learning, we use the simulated manual scoring method to extract and match the score point sequence of students’ answers with standard answers. The score recognition model effectively improves the efficiency of model calculation and long text keyword matching. The loss value of the final training score recognition model is about 0.9, and the accuracy is 80.54%. The accuracy of the training text similarity matching model is 86.99%, and the fusion model is single. The scoring time is less than 0.8s, and the accuracy is 83.43%.

show abstract

Section: Pretrainingmentioning

confidence: 99%

Automatic Scoring Model of Subjective Questions Based Text Similarity Fusion Model

Xie

Chen

2022

Lecture Notes in Electrical Engineering

View full text Add to dashboard Cite

show abstract

“…This provides the background knowledge in a novel way, as an alternative to the use of knowledge graphs mentioned at the beginning of this section. In contrast, Tamborrino et al (2020) rephrase the causal reasoning task into an "A because B" format and calculate the likelihood of each token in the input by masking one at a time, leading to much higher performance when compared to the use of the same model for fine-tuning on the target task.…”

Section: Related Workmentioning

confidence: 99%

“…Deep approaches aim to gain more information about a particular input, for example by using knowledge graphs to learn more about the entities and events mentioned in the input or applying formal logic to deterministically find the relation that is sought for, in order to classify that input correctly (Furbach, Gordon, and Schon 2015;Furbach and Schon 2016;Blass and Forbus 2017;Siebert, Schon, and Stolzenburg 2019;Goodwin and Demner-Fushman 2019). On the other hand, the approaches of the broad type attempt to cover a wider range of instances in their method, for example by determining the types of syntactic or semantic features that are generally used in expressing a particular relation (Gordon, Bejan, and Sagae 2011;Goodwin et al 2012;Jabeen, Gao, and Andreae 2014;Rahimtoroghi, Hernandez, and Walker 2017;Tamborrino et al 2020;Iter et al 2020). This paper attempts to tackle the causal reasoning task with two approaches, one of the deep and one of the broad type.…”

Section: Introductionmentioning

confidence: 99%

Improving Commonsense Causal Reasoning by Adversarial Training and Data Augmentation

Staliūnaitė

Gorinski

Iacobacci

2021

AAAI

View full text Add to dashboard Cite

Determining the plausibility of causal relations between clauses is a commonsense reasoning task that requires complex inference ability. The general approach to this task is to train a large pretrained language model on a specific dataset. However, the available training data for the task is often scarce, which leads to instability of model training or reliance on the shallow features of the dataset. This paper presents a number of techniques for making models more robust in the domain of causal reasoning. Firstly, we perform adversarial training by generating perturbed inputs through synonym substitution. Secondly, based on a linguistic theory of discourse connectives, we perform data augmentation using a discourse parser for detecting causally linked clauses in large text, and a generative language model for generating distractors. Both methods boost model performance on the Choice of Plausible Alternatives (COPA) dataset, as well as on a Balanced COPA dataset, which is a modified version of the original data that has been developed to avoid superficial cues, leading to a more challenging benchmark. We show a statistically significant improvement in performance and robustness on both datasets, even with only a small number of additionally generated data points.

show abstract

“…Information Masking Previous work [35,36,37,38] on question-answering has shown that some models tend to learn from artificial or superficial patterns of the dataset, and they can still predict the correct answer after import clues (to human) in the premise are masked. Therefore, we challenge our well-trained model, a RoBERTa with MLM fine-tuned on the original training set, by masking context, question, and both, respectively, during inference.…”

Section: Analysesmentioning

confidence: 99%

Go Beyond Plain Fine-Tuning: Improving Pretrained Models for Social Commonsense

Chang

Liu

Gopalakrishnan

et al. 2021

2021 IEEE Spoken Language Technology Workshop (SLT)

View full text Add to dashboard Cite

Pretrained language models have demonstrated outstanding performance in many NLP tasks recently. However, their social intelligence, which requires commonsense reasoning about the current situation and mental states of others, is still developing. Towards improving language models' social intelligence, in this study we focus on the Social IQA dataset, a task requiring social and emotional commonsense reasoning. Building on top of the pretrained RoBERTa and GPT2 models, we propose several architecture variations and extensions, as well as leveraging external commonsense corpora, to optimize the model for Social IQA. Our proposed system achieves competitive results as those top-ranking models on the leaderboard. This work demonstrates the strengths of pretrained language models, and provides viable ways to improve their performance for a particular task.

show abstract

Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning

Cited by 4 publications

References 0 publications

Automatic Scoring Model of Subjective Questions Based Text Similarity Fusion Model

Automatic Scoring Model of Subjective Questions Based Text Similarity Fusion Model

Improving Commonsense Causal Reasoning by Adversarial Training and Data Augmentation

Go Beyond Plain Fine-Tuning: Improving Pretrained Models for Social Commonsense

Contact Info

Product

Resources

About