Keyphrase Generation: A Multi-Aspect Survey

Çano, Erion; Bojar, Ondřej

doi:10.23919/fruct48121.2019.8981519

Cited by 14 publications

(14 citation statements)

References 77 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…They have been following a decreasing trend, falling from 32 % to 11 %. A similar trend is reported in other recent surveys [2,7]. We also observed that for the human evaluation process, existing studies usually involve a few human experts or some dozens of students.…”

Section: Nlg Evaluation Trendssupporting

confidence: 89%

Human or Machine: Automating Human Likeliness Evaluation of NLG Texts

Çano¹,

Bojar²

2020

Preprint

View full text Add to dashboard Cite

Automatic evaluation of various text quality criteria produced by data-driven intelligent methods is very common and useful because it is cheap, fast, and usually yields repeatable results. In this paper, we present an attempt to automate the human likeliness evaluation of the output text samples coming from natural language generation methods used to solve several tasks. We propose to use a human likeliness score that shows the percentage of the output samples from a method that look as if they were written by a human. Instead of having human participants label or rate those samples, we completely automate the process by using a discrimination procedure based on large pretrained language models and their probability distributions. As follow up, we plan to perform an empirical analysis of human-written and machinegenerated texts to find the optimal setup of this evaluation approach. A validation procedure involving human participants will also check how the automatic evaluation correlates with human judgments.

show abstract

Section: Nlg Evaluation Trendssupporting

confidence: 89%

Human or Machine: Automating Human Likeliness Evaluation of NLG Texts

Çano¹,

Bojar²

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…There was an opposite trend about the works that carry human evaluation only. They have been decreasing steadily, same as reported in [3] and [9]. We also noticed that most human evaluations involve a few domain experts or tens of university students.…”

Section: Text Quality Evaluationsupporting

confidence: 80%

Automating Text Naturalness Evaluation of NLG Systems

Çano,

Bojar

2020

Preprint

View full text Add to dashboard Cite

Automatic methods and metrics that assess various quality criteria of automatically generated texts are important for developing NLG systems because they produce repeatable results and allow for a fast development cycle. We present here an attempt to automate the evaluation of text naturalness which is a very important characteristic of natural language generation methods. Instead of relying on human participants for scoring or labeling the text samples, we propose to automate the process by using a human likeliness metric we define and a discrimination procedure based on large pretrained language models with their probability distributions. We analyze the text probability fractions and observe how they are influenced by the size of the generative and discriminative models involved in the process. Based on our results, bigger generators and larger pretrained discriminators are more appropriate for a better evaluation of text naturalness. A comprehensive validation procedure with human participants is required as follow up to check how well this automatic evaluation scheme correlates with human judgments.

show abstract

“…Keywords are a subset of words or phrases from a document that can describe con-cepts or topics covered in the document [57], [58]. They are commonly used to annotate articles or other documents, and are essential for the categorization and fast retrieval of such items in digital libraries [59].…”

Section: B: Methods 2: Automatic Annotationmentioning

confidence: 99%

Clone-Seeker: Effective Code Clone Search Using Annotations

et al. 2022

View full text Add to dashboard Cite

Source code search plays an important role in software development, e.g. for exploratory development or opportunistic reuse of existing code from a code base. Often, exploration of different implementations with the same functionality is needed for tasks like automated software transplantation, software diversification, and software repair. Code clones, which are syntactically or semantically similar code fragments, are perfect candidates for such tasks. Searching for code clones involves a given search query to retrieve the relevant code fragments. We propose a novel approach called Clone-Seeker that focuses on utilizing clone class features in retrieving code clones. For this purpose, we generate metadata for each code clone in the form of a natural language document. The metadata includes a pre-processed list of identifiers from the code clones augmented with a list of keywords indicating the semantics of the code clone. This keyword list can be extracted from a manually annotated general description of the clone class, or automatically generated from the source code of the entire clone class. This approach helps developers to perform code clone search based on a search query written either as source code terms, or as natural language. With various experiments, we show that (1) Clone-Seeker is effective in finding clones from BigCloneBench dataset by applying code queries and natural language queries; 2) Clone-Seeker has a higher recall when searching for semantic code clones (i.e., Type-4) in BigCloneBench than the state-of-the-art; 3) Clone-Seeker is a generalized technique as it is effective in finding clones in Project CodeNet dataset by applying code queries and natural language queries. 4) Clone-Seeker with manual annotation outperforms other variants in finding clones on the basis of natural language queries.

show abstract

Keyphrase Generation: A Multi-Aspect Survey

Cited by 14 publications

References 77 publications

Human or Machine: Automating Human Likeliness Evaluation of NLG Texts

Human or Machine: Automating Human Likeliness Evaluation of NLG Texts

Automating Text Naturalness Evaluation of NLG Systems

Clone-Seeker: Effective Code Clone Search Using Annotations

Contact Info

Product

Resources

About