Automatic computer science domain multiple-choice questions generation based on informative sentences

Maheen, Farah; Asif, Muhammad; Ahmad, Haseeb; Ahmad, Shahbaz; Alturise, Fahad; Asiry, Othman; Ghadi, Yazeed Yasin

doi:10.7717/peerj-cs.1010

Cited by 10 publications

(6 citation statements)

References 55 publications

(86 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Such evaluations often involved students differentiating AI‐generated from human‐generated content (Merine & Purkayastha, 2022; Song et al, 2022) and assessing student satisfaction with AI‐generated responses (Nguyen et al, 2021). Likewise, two studies have involved experts in evaluating specific features of the content generated by the LLMs‐based innovations, such as informativeness (Maheen et al, 2022) and cognitive level (Moore et al, 2022). Surveys have been used to evaluate students' experience with LLMs‐based innovations from multiple perspectives, such as the quality and difficulty of AI‐generated questions (Drori et al, 2022; Li & Xing, 2021) and potential learning benefits of the systems (Jayaraman & Black, 2022).…”

Section: Resultsmentioning

confidence: 99%

Practical and ethical challenges of large language models in education: A systematic scoping review

Yan,

Sha,

Zhao

et al. 2023

Brit J Educational Tech

View full text Add to dashboard Cite

Educational technology innovations leveraging large language models (LLMs) have shown the potential to automate the laborious process of generating and analysing textual content. While various innovations have been developed to automate a range of educational tasks (eg, question generation, feedback provision, and essay grading), there are concerns regarding the practicality and ethicality of these innovations. Such concerns may hinder future research and the adoption of LLMs‐based innovations in authentic educational contexts. To address this, we conducted a systematic scoping review of 118 peer‐reviewed papers published since 2017 to pinpoint the current state of research on using LLMs to automate and support educational tasks. The findings revealed 53 use cases for LLMs in automating education tasks, categorised into nine main categories: profiling/labelling, detection, grading, teaching support, prediction, knowledge representation, feedback, content generation, and recommendation. Additionally, we also identified several practical and ethical challenges, including low technological readiness, lack of replicability and transparency and insufficient privacy and beneficence considerations. The findings were summarised into three recommendations for future studies, including updating existing innovations with state‐of‐the‐art models (eg, GPT‐3/4), embracing the initiative of open‐sourcing models/systems, and adopting a human‐centred approach throughout the developmental process. As the intersection of AI and education is continuously evolving, the findings of this study can serve as an essential reference point for researchers, allowing them to leverage the strengths, learn from the limitations, and uncover potential research opportunities enabled by ChatGPT and other generative AI models. Practitioner notesWhat is currently known about this topic Generating and analysing text‐based content are time‐consuming and laborious tasks. Large language models are capable of efficiently analysing an unprecedented amount of textual content and completing complex natural language processing and generation tasks. Large language models have been increasingly used to develop educational technologies that aim to automate the generation and analysis of textual content, such as automated question generation and essay scoring. What this paper adds A comprehensive list of different educational tasks that could potentially benefit from LLMs‐based innovations through automation. A structured assessment of the practicality and ethicality of existing LLMs‐based innovations from seven important aspects using established frameworks. Three recommendations that could potentially support future studies to develop LLMs‐based innovations that are practical and ethical to implement in authentic educational contexts. Implications for practice and/or policy Updating existing innovations with state‐of‐the‐art models may further reduce the amount of manual effort required for adapting existing models to different educational tasks. The reporting standards of empirical research that aims to develop educational technologies using large language models need to be improved. Adopting a human‐centred approach throughout the developmental process could contribute to resolving the practical and ethical challenges of large language models in education.

show abstract

Section: Resultsmentioning

confidence: 99%

Practical and ethical challenges of large language models in education: A systematic scoping review

Yan,

Sha,

Zhao

et al. 2023

Brit J Educational Tech

View full text Add to dashboard Cite

show abstract

“…Such evaluations often involved students differentiating AI-generated from human-generated content [57,33] and assessing student satisfaction with AI-generated responses [37]. Likewise, two studies have involved experts in evaluating specific features of the content generated by the LLMsbased innovations, such as informativeness [31] and cognitive level [36]. Surveys have been used to evaluate students' experience with LLMs-based innovations from multiple perspectives, such as the quality and difficulty of AI-generated questions [17,30] and potential learning benefits of the systems [25].…”

Section: Ethical Challenges -Rq3mentioning

confidence: 99%

Practical and Ethical Challenges of Large Language Models in Education: A Systematic Literature Review

Yan¹,

Sha²,

Zhao³

et al. 2023

Preprint

View full text Add to dashboard Cite

Educational technology innovations that have been developed based on large language models (LLMs) have shown the potential to automate the laborious process of generating and analysing textual content. While various innovations have been developed to automate a range of educational tasks (e.g., question generation, feedback provision, and essay grading), there are concerns regarding the practicality and ethicality of these innovations. Such concerns may hinder future research and the adoption of LLMs-based innovations in authentic educational contexts. To address this, we conducted a systematic literature review of 118 peer-reviewed papers published since 2017 to pinpoint the current state of research on using LLMs to automate and support educational tasks. The practical and ethical challenges of LLMs-based innovations were also identified by assessing their technological readiness, model performance, replicability, system transparency, privacy, equality, and beneficence. The findings were summarised into three recommendations for future studies, including updating existing innovations with state-of-the-art models (e.g., GPT-3), embracing the initiative of open-sourcing models/systems, and adopting a human-centred approach throughout the developmental process. These recommendations could support future research to develop practical and ethical innovations for supporting diverse educational tasks and benefiting students, teachers, and institutions.

show abstract

“…Unstructured text sources often organize knowledge in the form of articles or paragraphs and are crucial in the field of question answering. In practice, multiple-answer questions play an important role in various assessment methods ( Maheen et al, 2022 ). Open-domain question answering based on multi-paragraph multi-answer reasoning challenges the ability to comprehensively utilize evidence from large-scale corpora.…”

Section: Related Workmentioning

confidence: 99%

Beyond top-k: knowledge reasoning for multi-answer temporal questions based on revalidation framework

Yao,

Yuan,

et al. 2023

PeerJ Computer Science

View full text Add to dashboard Cite

Answer sorting and filtering are two closely related steps for determining the answer to a question. Answer sorting is designed to produce an ordered list of scores based on Top-k and contextual criteria. Answer filtering optimizes the selection according to other criteria, such as the range of time constraints the user expects. However, the unclear number of answers and time constraints, as well as the high score of false positive results, indicate that the traditional sorting and selection methods cannot guarantee the quality of answers to multi-answer questions. Therefore, this study proposes MATQA, a component based on multi-answer temporal question reasoning, using a re-validation framework to convert the Top-k answer list output by the QA system into a clear number of answer combinations, and a new multi-answer based evaluation index is proposed for this output form. First, the highly correlated subgraph is selected by calculating the scores of the boot node and the related fact node. Second, the subgraph attention inference module is introduced to determine the initial answer with the highest probability. Finally, the alternative answers are clustered at the semantic level and the time constraint level. Meanwhile, the candidate answers with similar types and high scores but do not satisfy the semantic constraints or the time constraints are eliminated to ensure the number and accuracy of final answers. Experiments on the multi-answer TimeQuestions dataset demonstrate the effectiveness of the answer combinations output by MATQA.

show abstract

Automatic computer science domain multiple-choice questions generation based on informative sentences

Cited by 10 publications

References 55 publications

Practical and ethical challenges of large language models in education: A systematic scoping review

Practical and ethical challenges of large language models in education: A systematic scoping review

Practical and Ethical Challenges of Large Language Models in Education: A Systematic Literature Review

Beyond top-k: knowledge reasoning for multi-answer temporal questions based on revalidation framework

Contact Info

Product

Resources

About