Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications

Xie, Han; Zheng, Da; Ma, Jun; Zhang, H. P.; Ioannidis, Vassilis N.; Song, Xiang; Ping, Qing; Wang, Sheng; Yang, Carl; Xu, Yi; Zeng, Belinda; Chilimbi, Trishul

doi:10.1145/3580305.3599833

Cited by 10 publications

(5 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work we propose a model powered by a static layer combining a GNN and an LLM, showing how a synergy between the two architectures can help to combine content and consumption patterns. In a similar fashion, Xie et al [19] in their work show how a graph-aware Language Model framework can help to improve performances on different downstream tasks on large scale industry data. The static layer proposed in our work can be seen as analogous to the pre-training architecture proposed by Xie et al It is worth noting that there are multiple previous efforts combining graphs with LLMs, but the focus has been less on creating a foundation model.…”

Section: Related Workmentioning

confidence: 96%

“…On the other hand, graph-based learning models, specifically Graph Neural Networks (GNNs), have emerged as a powerful technology for recommendation systems at scale, becoming a core functionality on different online and social platforms [4,9,19,22]. Moreover, only lately, GNNs have been showing relevant gains also for enabling discovery without loss in accuracy [3].…”

mentioning

confidence: 99%

See 1 more Smart Citation

Towards Graph Foundation Models for Personalization

Damianou,

Fabbri,

Gigioli

et al. 2024

Companion Proceedings of the ACM Web Conference 2024

View full text Add to dashboard Cite

In the realm of personalization, integrating diverse information sources such as consumption signals and content-based representations is becoming increasingly critical to build state-of-the-art solutions. In this regard, two of the biggest trends in research around this subject are Graph Neural Networks (GNNs) and Foundation Models (FMs). While GNNs emerged as a popular solution in industry for powering personalization at scale, FMs have only recently caught attention for their promising performance in personalization tasks like ranking and retrieval. In this paper, we present a graphbased foundation modeling approach tailored to personalization. Central to this approach is a Heterogeneous GNN (HGNN) designed to capture multi-hop content and consumption relationships across a range of recommendable item types. To ensure the generality required from a Foundation Model, we employ a Large Language Model (LLM) text-based featurization of nodes that accommodates all item types, and construct the graph using co-interaction signals, which inherently transcend content specificity. To facilitate practical generalization, we further couple the HGNN with an adaptation mechanism based on a two-tower (2T) architecture, which also operates agnostically to content type. This multi-stage approach ensures high scalability; while the HGNN produces general purpose embeddings, the 2T component models in a continuous space the sheer size of user-item interaction data. Our comprehensive approach has been rigorously tested and proven effective in delivering recommendations across a diverse array of products within a real-world, industrial audio streaming platform.

show abstract

Section: Related Workmentioning

confidence: 96%

mentioning

confidence: 99%

Towards Graph Foundation Models for Personalization

Damianou,

Fabbri,

Gigioli

et al. 2024

Companion Proceedings of the ACM Web Conference 2024

View full text Add to dashboard Cite

show abstract

“…• Pre-training data: Models pre-trained on large amounts of text data to learn language representations can be useful. 78,80 The pre-training data may include various sources of biomedical literature, clinical notes, EHRs, drug labels, and other healthcarerelated text. These datasets can range from millions to billions of tokens.…”

Section: Parameters Used In the Development Of Gpt For Medicinementioning

confidence: 99%

Artificial Intelligence in Newborn Medicine

Huisman,

Huisman

2024

Newborn

View full text Add to dashboard Cite

“…Graph-Aware LLM Finetuning SPECTER [51], SciNCL [52], Touchup-G [54], TwHIN-BERT [56], MICoL [59], E2EG [60] LLM as Encoder Optimization One-step TextGNN [77], AdsGNN [78], GNN-LM [66] Two-step GIANT [58], LM-GNN [68], SimTeG [35], GaLM [80] Data Augmentation LLM-GNN [64], TAPE [70], ENG [71] Knowledge Distillation AdsGNN [78], GraD [69] LLM as Aligner Prediction Alignment LTRN [57], GLEM [62] Latent Space Alignment ConGrat [53], GRENADE [55], G2P2 [63], THLM [33] Text-Paired Graphs…”

Section: Graph As Sequencementioning

confidence: 99%

“…They further find that using the efficient fine-tuning method, e.g., LoRA [40] to tune the LLM can alleviate overfitting issues. GaLM [80] explores ways to pretrain the LLM-GNN cascaded architecture. The twostep strategy can effectively alleviate the insufficient training of the LLM which contributes to higher text representation quality but is more computationally expensive and timeconsuming than the one-step training strategy.…”

Section: Two-step Training Means First Adaptingmentioning

confidence: 99%

The Road Towards 6G: A Comprehensive Survey

Jiang¹,

Han²,

Habibi³

et al. 2021

Preprint

View full text Add to dashboard Cite

<p>As of today, the fifth generation (5G) mobile communication system has been rolled out in many countries and the number of 5G subscribers already reaches a very large scale. It is time for academia and industry to shift their attention towards the next generation. At this crossroad, an overview of the current state of the art and a vision of future communications are definitely of interest. This article thus aims to provide a comprehensive survey to draw a picture of the sixth generation (6G) system in terms of drivers, use cases, usage scenarios, requirements, key performance indicators (KPIs), and enabling technologies. First, we attempt to answer the question of “Is there any need for 6G?” by shedding light on the key driving factors of 6G, in which we predict the explosive growth of mobile traffic until 2030, and envision potential use cases and usage scenarios. Second, the technical requirements of 6G are discussed and compared with those of 5G with respect to a set of KPIs in a quantitative manner. Third, the state-of-the-art 6G research efforts and activities from representative institutions and countries are summarized, and a tentative roadmap of definition, specification, standardization, and regulation is projected. Then, we identify a dozen of potential technologies and introduce their principles, advantages, challenges, and open research issues. Finally, the conclusions are drawn to paint a picture of “What 6G may look like?”. This survey is intended to serve as an enlightening guideline to spur interests and further investigations for subsequent research and development of 6G communications systems.</p>

show abstract

Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications

Cited by 10 publications

References 16 publications

Towards Graph Foundation Models for Personalization

Towards Graph Foundation Models for Personalization

Artificial Intelligence in Newborn Medicine

The Road Towards 6G: A Comprehensive Survey

Contact Info

Product

Resources

About