Dual-channel graph contrastive learning for self-supervised graph-level representation learning

Luo, Zhenfei; Dong, Youming; Zheng, Qinghua; Liu, Huan; Luo, Minnan

doi:10.1016/j.patcog.2023.109448

Cited by 8 publications

(1 citation statement)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They propose to derive positive and negative contrastive pairs from citation triplets and demonstrate the power of mining hard negatives. MICoL and CitationSum (Luo et al, 2023) adopt contrastive learning to multi-label classification and summarization of scientific papers, respectively. As for multi-task learning, Luan et al (2018) propose a multi-task scientific knowledge graph construction framework by jointly identifying entities, relations, and coreference; treat multiple biomedical named entity recognition datasets (with different types of entities annotated) as multiple tasks so that they can mutually benefit each other.…”

Section: Related Workmentioning

confidence: 99%

Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding

Zhang,

Cheng,

Shen

et al. 2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Scientific literature understanding tasks have gained significant attention due to their potential to accelerate scientific discovery. Pretrained language models (LMs) have shown effectiveness in these tasks, especially when tuned via contrastive learning. However, jointly utilizing pre-training data across multiple heterogeneous tasks (e.g., extreme multi-label paper classification, citation prediction, and literature search) remains largely unexplored. To bridge this gap, we propose a multi-task contrastive learning framework, SciMult, with a focus on facilitating common knowledge sharing across different scientific literature understanding tasks while preventing task-specific skills from interfering with each other. To be specific, we explore two techniques -taskaware specialization and instruction tuning. The former adopts a Mixture-of-Experts Transformer architecture with task-aware sub-layers; the latter prepends task-specific instructions to the input text so as to produce task-aware outputs. Extensive experiments on a comprehensive collection of benchmark datasets verify the effectiveness of our task-aware specialization strategy, where we outperform stateof-the-art scientific pre-trained LMs. Code, datasets, and pre-trained models can be found at https://scimult.github.io/.

show abstract

Section: Related Workmentioning

confidence: 99%