Domain Knowledge Transferring for Pre-trained Language Model via Calibrated Activation Boundary Distillation

Choi, Dongha; Choi, Hongseok; Lee, Hyunju

doi:10.18653/v1/2022.acl-long.116

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This suggests that compared to MRC mechanisms, our prompt-based approach improves the reasoning ability of the bert-based model. Our model's accuracy in fine-grained emotion recognition is not able to match that 2 https://github.com/huggingface/transformers 3 https://github.com/Lightning-AI/lightning of large language models because the large model itself has a good emotion recognition ability (Bubeck et al 2023), and FG-RECCON is marked by ChatGPT. But, we achieve similar performance in identifying causal spans with ChatGPT, which indicates that we obtain the ability to perform causal reasoning through knowledge distillation from the relatively LLaMA model.…”

Section: Resultsmentioning

confidence: 91%

“…The architecture of teacher-student models has also been used as a special form of transfer learning for domain migration (Choi, Choi, and Lee 2022). Recently, Large Language Models (LLMs) have shown excellent performance in generalization across various tasks (Bubeck et al 2023). In order to improve the performance of models in specific domains, many research works have focused on distilling the knowledge of teacher LLMs into student models.…”

Section: Prompt Learning and Knowledge Distillationmentioning

confidence: 99%

See 1 more Smart Citation

From Coarse to Fine: A Distillation Method for Fine-Grained Emotion-Causal Span Pair Extraction in Conversation

Chen,

Yang,

Sun

et al. 2024

AAAI

View full text Add to dashboard Cite

We study the problem of extracting emotions and the causes behind these emotions in conversations. Existing methods either tackle them separately or jointly model them at the coarse-grained level of emotions (fewer emotion categories) and causes (utterance-level causes). In this work, we aim to jointly extract more fine-grained emotions and causes. We construct a fine-grained dataset FG-RECCON, includes 16 fine-grained emotion categories and span-level causes. To further improve the fine-grained extraction performance, we propose to utilize the casual discourse knowledge in a knowledge distillation way. Specifically, the teacher model learns to predict causal connective words between utterances, and then guides the student model in identifying both the fine-grained emotion labels and causal spans. Experimental results demonstrate that our distillation method achieves state-of-the-art performance on both RECCON and FG-RECCON dataset.

show abstract

Section: Resultsmentioning

confidence: 91%

Section: Prompt Learning and Knowledge Distillationmentioning

confidence: 99%