Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?

Wang, Shuai; Scells, Harrisen; Koopman, Bevan; Zuccon, Guido

doi:10.1145/3539618.3591703

Cited by 65 publications

(31 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experts found that ChatGPTs are helpful in writing research reviews. Interestingly, in similar views about ChatGPTs' usefulness in writing the theoretical part of research and types of research reviews, Wang et al (2023) found that ChatGPT demonstrated the potential of ChatGPT in generating useful Boolean queries for systematic review literature search. In contrast to many studies, Haman and S ˇkolnik (2023) found that ChatGPT is not supportive and useful in writing research reviews because the information given is based on fake papers, which makes it untrustworthy in research processes, and the papers are not based on academic and scholarly research steps.…”

Section: Chatgpt's Benefits and Reassurances In Research Levelmentioning

confidence: 91%

“…In recent studies, it has been reported that ChatGPT scored well in the law exam and achieved passing average grade (Choi et al, 2023). Similarly, Wang et al (2023) investigated the performance of ChatGPTs in two years' worth of the Chinese National Medical Licensing Examination that consists of four units to both ChatGPT and medical students and found that ChatGPTs produced responses that are at or near the passing threshold for all three parts of the US Medical Licensing Exam without any additional information or prompt engineering. The study recommended that medical education authorities and students to be familiar of the developments in the AI chatbot and consider its potential use in learning and education.…”

Section: Chatgpt's Benefits and Reassurances In Research Levelmentioning

confidence: 99%

See 1 more Smart Citation

ChatGPT in scientific and academic research: future fears and reassurances

Qasem

2023

LHTN

View full text Add to dashboard Cite

Purpose This purpose of this study is to examine future fears and reassurances about the nature of the recent artificial intelligence (AI) language model-based application, ChatGPT, use in the fields of scientific research and academic works and assignments. This study aims at exploring the positive and negative aspects of the use of ChatGPT by researchers and students. This paper recommends some practical academic steps and suggestions that help the researchers and publishers curtail the percentage of spread of unethical works such as plagiarism. Design/methodology/approach The emergence of OpenAI’s Generative Pre-Trained Transformer 3 (GPT-3) has recently sparked controversy and heated debate among academics worldwide about its use and application. The concern of experts and researchers about the GPT-3 platform entails how it would be of much support to the researchers and academic staff and how it might be used and misused to negatively affect academic and scholarly works. This research explored future fears and reassurances about the nature of Chat GPT-3 use at academic and scientific levels. The data for this research was obtained through the qualitative interviews of seven experts in AI, scientific research and academic works. The findings of the study showed that ChatGPT-3 has significant potential and is helpful if used wisely and ethically at scientific and academic levels. On the other hand, the results reported the experts' fears of the frequent use of ChatGPT including the misuse of ChatGPT as a tool to plagiarize and make the researchers dependent, not self-reliant and lazy. The widespread concern of many scholars is that ChatGPT would lead to an increase in the possibility of plagiarism and provide less control over research and writing ethics. This study proposed some stages and suggested that AI language model programs, including ChatGPT, should be integrated with widespread publishers and academic platforms to curtail the percentage of plagiarism and organize the process of publishing and writing scientific research and academic works to save the rights of researchers and writers. Findings The findings of the research presented that ChatGPT can act as a potential and useful tool if used wisely and ethically at scientific and academic fields. On contrast, the results also reported the negative aspects of the extensive ChatGPT's that leads to the spread of plagiarism and making the researchers and the students machine-dependent, not self-reliant and lazy. This study proposed some stages and suggested that AI language model programs, including ChatGPT, should be integrated with widespread publishers and academic platforms to curtail the percentage of plagiarism and organize the process of publishing and writing scientific research and academic works to save rights of researchers and writers. Originality/value To the best of the authors’ knowledge, this paper is the first of its kind to highlight the relationship between using ChatGPT and the spread of both positive and negative aspects of its extensive use in scientific research and academic work. The importance of this study lies in the fact that it presents the concerns and future fears of people in academia as they cope with and deal with the inevitable reality of AI language models such as ChatGPT.

show abstract

Section: Chatgpt's Benefits and Reassurances In Research Levelmentioning

confidence: 91%

Section: Chatgpt's Benefits and Reassurances In Research Levelmentioning

confidence: 99%

ChatGPT in scientific and academic research: future fears and reassurances

Qasem

2023

LHTN

View full text Add to dashboard Cite

show abstract

“…In a study prompting ChatGPT to provide answers to questions and supporting evidence in the form of references to external sources, Zuccon et al (2023) found that the majority of references do not actually exist even though they appear legitimate. Wang et al (2023) investigated the effectiveness of ChatGPT in generating effective Boolean queries for systematic review literature searches. Through a series of experiments, they found that use of ChatGPT compares favorably with current automated Boolean query generation methods in terms of precision, but at the expense of recall.…”

Section: Artificial Intelligence Technologies Employedmentioning

confidence: 99%

“…Wang et al (2023) investigated the effectiveness of ChatGPT in generating effective Boolean queries for systematic review literature searches. Through a series of experiments, they found that use of ChatGPT compares favorably with current automated Boolean query generation methods in terms of precision, but at the expense of recall.…”

Section: Impact Of Information Technologymentioning

confidence: 99%

Reviews and Reviewing: Approaches to Research Synthesis. An Annual Review of Information Science and Technology (ARIST) paper

Smith

2023

Asso for Info Science & Tech

View full text Add to dashboard Cite

Reviews have long been recognized as among the most important forms of scientific communication. The rapid growth of the primary literature has further increased the need for reviews to distill and interpret the literature. This review on Reviews and Reviewing: Approaches to Research Synthesis encompasses the evolution of the review literature, taxonomy of review literature, uses and users of reviews, the process of preparing reviews, assessment of review quality and impact, the impact of information technology on the preparation of reviews, and research opportunities for information science related to reviews and reviewing. In addition to providing a synthesis of prior research, this review seeks to identify gaps in the published research and to suggest possible future research directions.

show abstract

“…Nevertheless, ChatGPT has been shown to perform remarkable tasks many of which are similar to performing risk of bias assessments, including passing the United States Medical Licensing exams (15), performing accurate diagnoses (16), and offering medical advice comparable to physicians (17). Further, ChatGPT has been able to construct reasonable search strategies for systematic reviews (18) and other tasks for which it was not intentionally designed (19), suggesting that it may also be able to assess risk of bias despite not originally being designed for this task.…”

Section: Introductionmentioning

confidence: 99%

ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods study

Pitre,

Jassal,

Talukdar

et al. 2023

Preprint

View full text Add to dashboard Cite

BackgroundThe assessment of risk of bias is a critical component of systematic review methods. Assessing risk of bias, however, can be time- and resource-intensive. AI-based solutions may increase efficiency and reduce burden.ObjectiveTo evaluate the reliability of ChatGPT for performing risk of bias assessments of randomized trials.MethodsWe sampled recently published Cochrane systematic reviews of medical interventions (up to October 2023) that included randomized controlled trials and assessed risk of bias using the Cochrane-endorsed revised risk of bias tool for randomized trials (RoB 2.0). From each eligible review, we collected data on the risk of bias assessments for the first three reported outcomes. Using ChatGPT-4, we assessed the risk of bias for the same outcomes using three different prompts: a minimal prompt including limited instructions, a maximal prompt with extensive instructions, and an optimized prompt that was designed to yield the best risk of bias judgments. The agreement between ChatGPT’s assessments and those of the systematic reviewers was quantified using weighted kappa statistics.ResultsWe included 34 systematic reviews with 157 unique trials. We found the agreement between ChatGPT and systematic review authors for assessment of overall risk of bias to be 0.16 (95% CI: 0.01 to 0.3) for the maximal ChatGPT prompt, 0.17 (95% CI: 0.02 to 0.32) for the optimized prompt, and 0.11 (95% CI: -0.04 to 0.27) for the minimal prompt. For the optimized prompt, agreement ranged between 0.11 (95% CI: -0.11 to 0.33) to 0.29 (95% CI: 0.14 to 0.44) across risk of bias domains, with the lowest agreement for the deviations from the intended intervention domain and the highest agreement for the missing outcome data domain.ConclusionOur results suggest that ChatGPT and systematic reviewers only have “slight” to “fair” agreement in risk of bias judgments for randomized trials. ChatGPT is currently unable to reliably assess risk of bias of randomized trials. We recommend systematic reviewers avoid using ChatGPT to perform risk of bias assessments.

show abstract

Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?

Cited by 65 publications

References 36 publications

ChatGPT in scientific and academic research: future fears and reassurances

ChatGPT in scientific and academic research: future fears and reassurances

Reviews and Reviewing: Approaches to Research Synthesis. An Annual Review of Information Science and Technology (ARIST) paper

ChatGPT for assessing risk of bias of randomized trials using the RoB 2.0 tool: A methods study

Contact Info

Product

Resources

About