Representation Learning from Limited Educational Data with Crowdsourced Labels

Wang, Wentao; Xu, Guowei; Ding, Wenbiao; Huang, Yan; Li, Guoliang; Tang, Jiliang; Liu, Zitao

doi:10.1109/tkde.2020.3017122

Cited by 10 publications

(10 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many approaches have achieved promising results in many tasks such as face recognition (Schroff et al, 2015), person re-identification (Yi et al, 2014), and collaborative filtering (Hsieh et al, 2017) etc. Recently a body of works have attempted to learn effective embeddings from crowdsourced labels by using DML approaches (Xu et al, 2019;Wang et al, 2020b). For example, Xu et al estimated crowdsourced label confidence and adjust the DML loss function accordingly (Xu et al, 2019).…”

Section: Deep Metric Learningmentioning

confidence: 99%

“…In this paper we study and develop solutions that are applicable and can learn effective neural language representations from crowdsourced labels in an end-to-end manner. Our work focuses on the refinements of a popular deep language representation learning paradigm: the deep metric learning (DML) (Koch et al, 2015;Xu et al, 2019;Wang et al, 2020b). We aim to develop an algorithm to automatically learn a nonlinear language representation of the crowdsourced data from multiple workers using DNNs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Temporal-aware Language Representation Learning From Crowdsourced Labels

Yang¹,

Zhai²,

Ding³

et al. 2021

Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)

Self Cite

View full text Add to dashboard Cite

Learning effective language representations from crowdsourced labels is crucial for many real-world machine learning tasks. A challenging aspect of this problem is that the quality of crowdsourced labels suffer high intraand inter-observer variability. Since the highcapacity deep neural networks can easily memorize all disagreements among crowdsourced labels, directly applying existing supervised language representation learning algorithms may yield suboptimal solutions. In this paper, we propose TACMA, a temporal-aware language representation learning heuristic for crowdsourced labels with multiple annotators. The proposed approach (1) explicitly models the intra-observer variability with attention mechanism; (2) computes and aggregates per-sample confidence scores from multiple workers to address the inter-observer disagreements. The proposed heuristic is extremely easy to implement in around 5 lines of code. The proposed heuristic is evaluated on four synthetic and four real-world data sets. The results show that our approach outperforms a wide range of state-of-the-art baselines in terms of prediction accuracy and AUC. To encourage the reproducible results, we make our code publicly available at https://github.com/ CrowdsourcingMining/TACMA.

show abstract

Section: Deep Metric Learningmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Temporal-aware Language Representation Learning From Crowdsourced Labels

Yang¹,

Zhai²,

Ding³

et al. 2021

Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Large-scale Human Evaluation Results. Besides evaluations on the GT set, which is usually limited in educational scenarios (Xu et al, 2019;Wang et al, 2020), we conduct evaluations on the largescale generated results. We randomly create 100 valid linear equations and ensure that none of them appears in our training set.…”

Section: Makementioning

confidence: 99%

Mathematical Word Problem Generation from Commonsense Knowledge Graph and Equations

Liu¹,

Qian²,

Ding³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

Self Cite

View full text Add to dashboard Cite

There is an increasing interest in the use of mathematical word problem (MWP) generation in educational assessment. Different from standard natural question generation, MWP generation needs to maintain the underlying mathematical operations between quantities and variables, while at the same time ensuring the relevance between the output and the given topic. To address above problem, we develop an end-to-end neural model to generate diverse MWPs in real-world scenarios from commonsense knowledge graph and equations. The proposed model (1) learns both representations from edge-enhanced Levi graphs of symbolic equations and commonsense knowledge; (2) automatically fuses equation and commonsense knowledge information via a self-planning module when generating the MWPs. Experiments on an educational gold-standard set and a large-scale generated MWP set show that our approach is superior on the MWP generation task, and it outperforms the SOTA models in terms of both automatic evaluation metrics, i.e., BLEU-4, ROUGE-L, Self-BLEU, and human evaluation metrics, i.e., equation relevance, topic relevance, and language coherence. To encourage reproducible results, we make our code and MWP dataset public available at https:// github.com/tal-ai/MaKE_EMNLP2021.

show abstract

“…Hard Example Mining Strategy: Many instances that can be classified correctly by the model contribute little to the contrastive loss [18,21]. That is to say, a randomly selected instance xc j probably has been far away from an instance x i after epochs of training.…”

Section: Multi-task Learning Modulementioning

confidence: 99%

“…To address the above challenges, in this study, we propose an end-to-end multi-task framework for automatic dialogic instruction detection from online videos. Specifically, we (1) propose a contrastive loss based multi-task framework to distinguish instances by enlarging the distances between instances of di↵erent categories [12,18]; (2) utilize the pre-trained neural language model to robustly handle errors from ASR transcriptions without manual annotation e↵orts [5,15]; and (3) propose a strategy to select and exploit hard instances in the training process to achieve higher performance [21,18].…”

Section: Introductionmentioning

confidence: 99%

Recent Advances in Multimodal Educational Data Mining in K-12 Education

Liu¹,

Yang²,

Tang

et al. 2020

Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery &Amp; Data Mining

Self Cite

View full text Add to dashboard Cite

In this work, we study computational approaches to detect online dialogic instructions, which are widely used to help students understand learning materials, and build e↵ective study habits. This task is rather challenging due to the widely-varying quality and pedagogical styles of dialogic instructions. To address these challenges, we utilize pre-trained language models, and propose a multi-task paradigm which enhances the ability to distinguish instances of di↵erent classes by enlarging the margin between categories via contrastive loss. Furthermore, we design a strategy to fully exploit the misclassified examples during the training stage. Extensive experiments on a real-world online educational data set demonstrate that our approach achieves superior performance than other baselines. To encourage reproducible results, we make our code online available at https://github.com/AIED2021/multitaskdialogic-instruction.

show abstract

Representation Learning from Limited Educational Data with Crowdsourced Labels

Cited by 10 publications

References 56 publications

Temporal-aware Language Representation Learning From Crowdsourced Labels

Temporal-aware Language Representation Learning From Crowdsourced Labels

Mathematical Word Problem Generation from Commonsense Knowledge Graph and Equations

Recent Advances in Multimodal Educational Data Mining in K-12 Education

Contact Info

Product

Resources

About