Selecting Better Samples from Pre-trained LLMs: A Case Study on Question Generation

Abstract

Large Language Models (LLMs) have in recent years demonstrated impressive prowess in natural language generation. A common practice to improve generation diversity is to sample multiple outputs from the model. However, partly due to the inaccessibility of LLMs, there lacks a simple and robust way of selecting the best output from these stochastic samples. As a case study framed in the context of question generation, we propose two prompt-based approaches, namely round-trip and prompt-based score, to selecting high-quality questions from a set of LLM-generated candidates. Our method works without the need to modify the underlying model, nor does it rely on human-annotated references -both of which are realistic constraints for real-world deployment of LLMs. With automatic as well as human evaluations, we empirically demonstrate that our approach can effectively select questions of higher qualities than greedy generation.1 * Equal contribution. 1 We open-source all code and annotated data on github.

Evaluation of the Quality of Conversational Agents for the Creation of Evaluation Instruments in Bioelectric Signals Measurement

“…Once expertos evaluaron la calidad de las preguntas en términos de relevancia, gramática y utilidad, con resultados favorables [24] . Yuan utilizó GPT-3 para crear cinco preguntas abiertas para estudiantes de primaria y secundaria sobre diversos temas [25] . Su objetivo era evaluar un método para seleccionar preguntas de calidad entre un conjunto de preguntas generadas con IA.…”

“…Su objetivo era evaluar un método para seleccionar preguntas de calidad entre un conjunto de preguntas generadas con IA. Se realizó evaluación automática contra preguntas de referencia y evaluación por 87 expertos demostrando la eficacia de su método [25] . Finalmente, Xiao utilizó ChatGPT y GPT-2 para realizar 20 preguntas de opción múltiple sobre inglés nivel secundaria [26] .…”