Advanced Content Analysis

Franzen, Samuel R P; Quang, Cuong; Schweizer, Lukas; Budzier, Alexander; Hrstich, Peter; Reissfelder, Stéphane; Gold, Jenny; Vellez, Mercedes; Ramírez, Santiago R.; Raimondo, Estelle

doi:10.1596/37117

Cited by 2 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This may help to explain the relatively slow proliferation of ETs in evaluation. However, there are several indications of emerging use in this domain (i.e., Franzen et al., 2022).…”

Section: Resultsmentioning

confidence: 99%

“…The size and duration of the contract, set by the commissioners of evaluation studies, is likely to be of importance for how ET affects the evaluation market. The human capability, technology, and time invested in conducting algorithm‐based text analytics is significant (Franzen et al., 2022; Næss et al., n.d.). It may therefore prove too costly, time‐consuming, and cumbersome for small budget evaluations.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Disrupting evaluation? Emerging technologies and their implications for the evaluation industry

Nielsen

2023

New Drctns Evaluation

View full text Add to dashboard Cite

This article surveyed different emerging technologies (ET), in particular artificial intelligence, and their burgeoning application in the evaluation industry. Evidence suggests that evaluators have been relatively slow in adopting ET in their practice. However, more recent data suggest that ET adoption is increasing. This article then analyzed if, and how, ET affect the evaluation industry and evaluation practice. The article finds that program evaluation is one of several competing forms of knowledge production informing decision‐making, particularly in the government and not‐for‐profit sectors. Therefore, evaluation faces a number of challenges stemming from ET. In this article, it is argued that evaluators must, albeit critically, embrace ET. Most likely, ET will complement evaluation practice and, in some instances, displace human tasks.

show abstract

“…This may help to explain the relatively slow proliferation of ETs in evaluation. However, there are several indications of emerging use in this domain (i.e., Franzen et al., 2022).…”

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Disrupting evaluation? Emerging technologies and their implications for the evaluation industry

Nielsen

2023

New Drctns Evaluation

View full text Add to dashboard Cite

show abstract

“…Beyond the annotations step, any additional fine‐tuning or training for particular tasks, use cases, or knowledge domains represent an important step for improving the performance of LLMs, but are similarly vulnerable to error and bias with human involvement. Still, these are tasks that can also be completed with relatively less error, when done by well‐paid subject matter experts (Franzen et al., 2022).…”

Section: Introductionmentioning

confidence: 99%

Large language model applications for evaluation: Opportunities and ethical implications

Head,

Jasper,

McConnachie

et al. 2023

New Drctns Evaluation

View full text Add to dashboard Cite

Large language models (LLMs) are a type of generative artificial intelligence (AI) designed to produce text‐based content. LLMs use deep learning techniques and massively large data sets to understand, summarize, generate, and predict new text. LLMs caught the public eye in early 2023 when ChatGPT (the first consumer facing LLM) was released. LLM technologies are driven by recent advances in deep‐learning AI techniques, where language models are trained on extremely large text data from the internet and then re‐used for downstream tasks with limited fine‐tuning required. They offer exciting opportunities for evaluators to automate and accelerate time‐consuming tasks involving text analytics and text generation. We estimate that over two‐thirds of evaluation tasks will be affected by LLMs in the next 5 years. Use‐case examples include summarizing text data, extracting key information from text, analyzing and classifying text content, writing text, and translation. Despite the advances, the technologies pose significant challenges and risks. Because LLM technologies are generally trained on text from the internet, they tend to perpetuate biases (racism, sexism, ethnocentrism, and more) and exclusion of non‐majority languages. Current tools like ChatGPT have not been specifically developed for monitoring, evaluation, research, and learning (MERL) purposes, possibly limiting their accuracy and usefulness for evaluation. In addition, technical limitations and challenges with bias can lead to real world harm. To overcome these technical challenges and ethical risks, the evaluation community will need to work collaboratively with the data science community to co‐develop tools and processes and to ensure the application of quality and ethical standards.

show abstract

Advanced Content Analysis

Cited by 2 publications

References 0 publications

Disrupting evaluation? Emerging technologies and their implications for the evaluation industry

Disrupting evaluation? Emerging technologies and their implications for the evaluation industry

Large language model applications for evaluation: Opportunities and ethical implications

Contact Info

Product

Resources

About