GenePT: A Simple But Effective Foundation Model for Genes and Cells Built From ChatGPT

Abstract

There has been significant recent progress in leveraging large-scale gene expression data to develop foundation models for single-cell transcriptomes such as Geneformer [1], scGPT [2], and scBERT [3]. These models infer gene functions and interrelations from the gene expression profiles of millions of cells, which requires extensive data curation and resource-intensive training. Here, we explore a much simpler alternative by leveraging ChatGPT embeddings of genes based on literature. Our proposal, GenePT, uses NCBI text descriptions of individual genes with GPT-3.5 to generate gene embeddings. From there, GenePT generates single-cell embeddings in two ways: (i) by averaging the gene embeddings, weighted by each gene's expression level; or (ii) by creating a sentence embedding for each cell, using gene names ordered by the expression level. Without the need for dataset curation and additional pretraining, GenePT is efficient and easy to use. On many downstream tasks used to evaluate recent single-cell foundation models --- e.g., classifying gene properties and cell types --- GenePT achieves comparable, and often better, performance than Geneformer and other methods. GenePT demonstrates that large language model embedding of literature is a simple and effective path for biological foundation models.

Evaluating the Utilities of Foundation Models in Single-cell Data Analysis

“…Exploring the contribution of multimodal data to develop a multimodal FM is also a possible track. For example, incorporating text-based biological information [87] or multi-omic data with new tokens may help us further extend the functions of these FMs.…”

How do Large Language Models understand Genes and Cells

“…Cell2Sentence[16], through fine-tuning GPT-2, annotates cells by generating a textual representation for each one, based on the names of the top 100 genes ranked by their transcriptional expression values in the cell. GenePT[5] uses a similar approach for textualizing cells and employs GPT-3.5 to generate embedding vectors, which are then combined with other supervised learning models for various downstream tasks. The scarcity of efforts in fine-tuning LLMs for gene-related issues is apparent, with only Cell2Sentence being a notable attempt at cell annotation.…”

“…LLMs can be applied to various scientific research fields with just appropriate textual input (prompt), without the need for additional complex modeling. Nonetheless, the application of LLMs in addressing gene-related issues has been sparse: GenePT[5] utilizes LLMs to generate embedding vectors for genes and cells while Cell2Sentence[16] has fine-tuned GPT-2 for cell annotation tasks. To alleviate this issue, this paper focuses on exploring the performance of LLMs across a spectrum of gene-related problems and evaluates the effectiveness of several mainstream LLMs in these contexts.…”

Deep generative AI models analyzing circulating orphan non-coding RNAs enable accurate detection of early-stage non-small cell lung cancer

“…In recent years, various classes of neural networks have provided robust and customizable frameworks for guided representation learning. Deep generative models can leverage variational inference (Lopez et al, 2018) or pre-training on masked data (Cui et al, 2023; Chen and Zou, 2023; Rosen et al, 2023) to facilitate a variety of downstream tasks. Given the over-parameterized nature of these networks, a large number of samples is required for the adaptation of these models for clinical genomics applications.…”