Large language models help facilitate the automated synthesis of information on potential pest controllers

Abstract

The body of ecological literature, which informs much of our knowledge of the global loss of biodiversity, has been experiencing rapid growth in recent decades. The increasing difficulty to synthesise this literature manually has simultaneously resulted in a growing demand for automated text mining methods. Within the domain of deep learning, large language models (LLMs) have been the subject of considerable attention in recent years by virtue of great leaps in progress and a wide range of potential applications, however, quantitative investigation into their potential in ecology has so far been lacking. In this work, we analyse the ability of GPT-4 to extract information about invertebrate pests and pest controllers from abstracts of a body of literature on biological pest control, using a bespoke, zero-shot prompt. Our results show that the performance of GPT-4 is highly competitive with other state-of-the-art tools used for taxonomic named entity recognition and geographic location extraction tasks. On a held-out test set, we show that species and geographic locations are extracted with F1-scores of 99.8% and 95.3%, respectively, and highlight that the model is able to distinguish very effectively between the primary roles of interest (predators, parasitoids and pests). Moreover, we demonstrate the ability of the model to effectively extract and predict taxonomic information across various taxonomic ranks, and to automatically correct spelling mistakes. However, we do report a small number of cases of fabricated information (hallucinations). As a result of the current lack of specialised, pre-trained ecological language models, general-purpose LLMs may provide a promising way forward in ecology. Combined with tailored prompt engineering, such models can be employed for a wide range of text mining tasks in ecology, with the potential to greatly reduce time spent on manual screening and labelling of the literature.

Fast tracking tool selection for sustainability decisions

“…Crucially, it also allows exposure to other fields AI is set to play a powerful role in accelerating the development and integration of the taxonomy of tools approach, and in making the pre-assessment process we outline more time and cost effective. Off-the-shelf AI tools are already becoming increasingly competent at summarizing and extracting information from the scientific literature (de la Torre-López et al, 2023;Richards et al, 2024;Scheepens et al, 2024) and with effective prompt engineering (i.e. crafting queries for tools such as ChatGPT Clavié et al, 2023) they could help to come up quickly with an initial list of tools that could be supplemented by expert inputreducing the time spent searching for tools (i.e.…”

The changing landscape of text mining - a review of approaches for ecology and evolution

“…Some models have been pre-trained using scientific corpora such as SciBERT, trained on a random sample of 1. Despite their demonstrated use in biomedical sciences, large language models are just beginning to be adopted in ecology and evolution [1,15,53,54], and to our knowledge there is currently only one large language model, BiodivBERT, trained explicitly on biodiversity-related texts [53].…”

“…The training and fine-tuning of LLMs is a computationally expensive task and relies on the availability of large amounts of training data. Developing domain specific language models may be necessary for specialised tasks, but as language models are adapted to answer questions and respond directly to Natural Language prompts, it may be more efficient to directly prompt LLMs to identify, extract, and harmonise data directly from scientific literature [54]. As an alternative to training domain-specific models, prompt-based learning with a pre-trained, general-purpose LLM allows for the execution of domain-specific tasks without the need for pre-training or transfer learning [68].…”

“…By additionally training LLMs to act as generative question answering models (e.g. OpenAI's ChatGPT built on GPT, and Meta's Llama-2) [69,70], researchers can directly prompt models to perform more complex information extraction tasks [71], as been demonstrated for identifying potential pest controllers [54], and the locations of species observations [72]. In this context a 'prompt' refers to a piece of text to instruct a language model to perform a task in natural language rather than through computer code.…”

“…Least-to-most prompting [82] proposes to decompose a complex task into a series of subtasks, each as their own prompt, which are then queried in sequence. For example, in a task that involves both NER and RE, one could first prompt the LLM with an NER task (e.g., "list all species mentioned in the text"), and then detail the RE task and any other instructions (e.g., "return a table with two columns: If a listed species acts as a parasite of another species mentioned in the text [based on criteria x, y, z], then return this parasite in column one and its host(s) in column two") [54]. Least-to-most prompting has been shown to improve the ability of GPT-3 to solve difficult reasoning problems [82].…”