Galactica: A Large Language Model for Science

Abstract

Information overload is a major obstacle to scientific progress. The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information. Today scientific knowledge is accessed through search engines, but they are unable to organize scientific knowledge alone. In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. We outperform existing models on a range of scientific tasks. On technical knowledge probes such as LaTeX equations, Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It also sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community 1 .Computing has indeed revolutionized how research is conducted, but information overload remains an overwhelming problem (Bornmann and Mutz, 2014). In May 2022, an average of 516 papers per day were submitted to arXiv (arXiv, 2022). Beyond papers, scientific data is also growing much more quickly than our ability to process it (Marx, 2013). As of August 2022, the NCBI GenBank contained 1.49 × 10 12 nucleotide bases (GenBank, 2022). Given the volume of information, it is impossible for a single person to read all the papers in a given field; and it is likewise challenging to organize data on the underlying scientific phenomena.Search engines are the current interface for accessing scientific knowledge following the Licklider paradigm. But they do not organize knowledge directly, and instead point to secondary layers such as Wikipedia,

AI chatbots not yet ready for clinical use

“…This is an advancement not seen in previous smaller language models and likely reflects the model's ability to learn and extract more knowledge from its training data. Transformer-based LLMs have demonstrated close to human-level performances in medical question and answering benchmarks and summarisation tasks ( 7 – 9 ), and with techniques like self-consistency ( 9 ), chain of thought prompting ( 10 ), and reinforcement learning from human feedback ( 11 ) the model performance can be further enhanced. Given their rapid rate of advancement, it is probable that LLM based conversational AI (chatbots) will soon be developed for healthcare use.…”

Studying psychosis using Natural Language Generation: A review of emerging opportunities

“…Such weighting is more "pliable" than a rule-based generation and mimics human-like statistical learning to some extent by enabling associations in the representation space. However, this same blending contributes to falsehoods (45).…”

Natural Language Processing in the Legal Domain

“…As the number of new publications over time grew and methodical and linguistic diversity increased, the task has become constantly more challenging. Similar developments in science in general 63 have inspired a series of innovative attempts to solve the problem of scientific knowledge discovery and education 36 by using of large language models to support both research 64 and writing of scientific publications 65 albeit with limited success and apparent upper bounds despite large numbers of parameters at present. 66 Given current technical limitations and employing an established design mechanism in machine learning, 67,68 we suggest a hybrid, human-in-the-loop approach to information management, combining state-of-the-art natural language processing tools to find and curate publications for review with subject matter expert human control.…”