Towards Human-Like Educational Question Generation with Large Language Models

Wang, Zichao; Valdez, Jakob; Mallick, Debshila Basu; Baraniuk, Richard G.

doi:10.1007/978-3-031-11644-5_13

Cited by 30 publications

(16 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent work [39,40] has shown that GPT-3 is capable of generating natural questions that require long-form and informative answers, in comparison to existing approaches [41,42] that mainly generate questions designed for short and specific answers. In this work, we prompt GPT-3 to explicitly generate strategic questions about the event.…”

Section: Strategic Questions As Section Headingsmentioning

confidence: 99%

SmartBook: AI-Assisted Situation Report Generation

Reddy¹,

Fung²,

Zeng³

et al. 2023

Preprint

View full text Add to dashboard Cite

Emerging events, such as the COVID pandemic and the Ukraine Crisis, require a time-sensitive comprehensive understanding of the situation to allow for appropriate decision-making and effective action response. Automated generation of situation reports can significantly reduce the time, effort, and cost for domain experts when preparing their official human-curated reports. However, AI research toward this goal has been very limited, and no successful trials have yet been conducted to automate such report generation. Pre-existing natural language processing methods, large language model based text generation, and information retrieval techniques are insufficient to identify, locate, and summarize important information, and lack detailed, structured, and strategic awareness. We propose SmartBook, a novel task formulation targeting situation report generation, which consumes large volumes of news data to produce a structured situation report with multiple hypotheses (claims) summarized and grounded with rich links to factual evidence. We realize SmartBook for the Ukraine-Russia crisis by automatically generating intelligence analysis reports to assist expert analysts. The machine-generated reports are structured in the form of timelines, with each timeline organized by major events (or chapters), corresponding strategic questions (or sections) and their grounded summaries (or section content). Our proposed framework automatically detects realtime event-related strategic questions, which are more directed than manually-crafted analyst questions, which tend to be too complex, hard

show abstract

Section: Strategic Questions As Section Headingsmentioning

confidence: 99%

SmartBook: AI-Assisted Situation Report Generation

Reddy¹,

Fung²,

Zeng³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For few-shot prompting, we use the text-davinci-003 model (GPT-3.5) from OpenAI 5 with 128 as the maximum token output, 0.7 for temperature and 1.0 for nucleus sampling. Following previous recommendations (Wang et al, 2022;Elkins et al, 2023), we choose the 5-shot setting: beyond the query, 5 examples (each composed of text, question and answer) are incorporated into the prompt. The 5 examples have been randomly extracted from the train set based on the following criterion: the selected examples are consistent with the target attribute (either narrative element or explicitness) one aims to control in the generation process.…”

Section: Implementation Detailsmentioning

confidence: 99%

“…Linguistic Quality: To evaluate the linguistic quality of the generated questions and answers, we report perplexity, grammatical error, and diversity metrics. For perplexity, our motivation is that previous studies (Wang et al, 2022) claim there is a relation between perplexity and coherence, in a way that perplexity is inversely related to the coherence of the generated text: the lower the perplexity score, the higher the coherence. For the sake of computational efficiency, we use GPT-2 (Radford et al, 2019) to compute perplexity.…”

Section: Implementation Detailsmentioning

confidence: 99%

“…While our current approach uses 5 examples, aligned with previous recommendations (Wang et al, 2022;Elkins et al, 2023), we explore alternative numbers of examples. Figure 4 shows the impact on closeness results (for assessing narrative elements control) when using 1, 3, 5 and 7 prompt examples.…”

Section: Varying the Number Of Prompt Examplesmentioning

confidence: 99%

“…From a methodological perspective, some prior tecture, choice of hyperparameters) and substantial computational resources for training, which may deter practitioners who prefer a convenient "plug-andplay" AI-assisted approach, enabling them to effort-arXiv:2404.02800v1 [cs.CL] 3 Apr 2024 lessly interact with a QG system without the need to undergo model design and training (Wang et al, 2022). Motivated by this, we explore a few-shot prompting strategy to address the CQG task in the context of narrative comprehension.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Neural Question Generation for the Portuguese Language: A Preliminary Study

Leite

Cardoso

2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Question Generation aims to automatically generate questions based on a given input provided as context. A controllable question generation scheme focuses on generating questions with specific attributes, allowing better control. In this study, we propose a few-shot prompting strategy for controlling the generation of question-answer pairs from children's narrative texts. We aim to control two attributes: the question's explicitness and underlying narrative elements. With empirical evaluation, we show the effectiveness of controlling the generation process by employing few-shot prompting side by side with a reference model. Our experiments highlight instances where the few-shot strategy surpasses the reference model, particularly in scenarios such as semantic closeness evaluation and the diversity and coherency of question-answer pairs. However, these improvements are not always statistically significant. The code is publicly available at github.com/bernardoleite/few-shot-prompting-qg-control.

show abstract