Rediscovering 15 + 2 years of discoveries in language resources and evaluation

Mariani, Joseph; Paroubek, Patrick; Francopoulo, Gil; Hamon, Olivier

doi:10.1007/s10579-016-9352-9

Cited by 6 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Resources and Evaluation Conference (LREC) proceedings between 1998 and 2012 (Mariani et al, 2014a) then 15 + 2 years, adding LREC 2014 (Mariani et al, 2016).…”

Section: • Analysis Of 15 Years Of Research Contained In the Languagementioning

confidence: 99%

See 1 more Smart Citation

The NLP4NLP Corpus (I): 50 Years of Publication, Collaboration and Citation in Speech and Language Processing

Mariani

Francopoulo²,

Paroubek

2019

Front. Res. Metr. Anal.

Self Cite

View full text Add to dashboard Cite

This paper introduces the NLP4NLP corpus, which contains articles published in 34 major conferences and journals in the field of speech and natural language processing over a period of 50 years (1965-2015), comprising 65,000 documents, gathering 50,000 authors, including 325,000 references and representing ∼270 million words. Most of these publications are in English, some are in French, German, or Russian. Some are open access, others have been provided by the publishers. In order to constitute and analyze this corpus several tools have been used or developed. Many of them use Natural Language Processing methods that have been published in the corpus, hence its name. The paper presents the corpus and some findings regarding its content (evolution over time of the number of articles and authors, collaborations between authors, citations between papers and authors), in the context of a global or comparative analysis between sources. Numerous manual corrections were necessary, which demonstrated the importance of establishing standards for uniquely identifying authors, articles, or publications.

show abstract

“…Resources and Evaluation Conference (LREC) proceedings between 1998 and 2012 (Mariani et al, 2014a) then 15 + 2 years, adding LREC 2014 (Mariani et al, 2016).…”

Section: • Analysis Of 15 Years Of Research Contained In the Languagementioning

confidence: 99%

“…as published in the Language and Technology Conference (L&TC) from 1995 to 2015 (Mariani et al, 2015).…”

Section: • Analysis Of 20 Years Of Research In Language Technologymentioning

confidence: 99%

The NLP4NLP Corpus (I): 50 Years of Publication, Collaboration and Citation in Speech and Language Processing

Mariani

Francopoulo²,

Paroubek

2019

Front. Res. Metr. Anal.

Self Cite

View full text Add to dashboard Cite

show abstract

“…As mentioned in Section 1, shared tasks are competitions to which researchers or teams of researchers submit systems that address a particular challenge. In the field of NLP, the first shared tasks were initiated in the United States by NIST in collaboration with DARPA (Mariani et al, 2014). Paroubek et al (2007) report that the first shared tasks -then called evaluation campaigns -focused on speech processing and started in 1987.…”

Section: Shared Tasks In Nlpmentioning

confidence: 99%

Ethical Considerations in NLP Shared Tasks

Escartín¹,

Reijers²,

Lynn³

et al. 2017

Proceedings of the First ACL Workshop on Ethics in Natural Language Processing

View full text Add to dashboard Cite

Shared tasks are increasingly common in our field, and new challenges are suggested at almost every conference and workshop. However, as this has become an established way of pushing research forward, it is important to discuss how we researchers organise and participate in shared tasks, and make that information available to the community to allow further research improvements. In this paper, we present a number of ethical issues along with other areas of concern that are related to the competitive nature of shared tasks. As such issues could potentially impact on research ethics in the Natural Language Processing community, we also propose the development of a framework for the organisation of and participation in shared tasks that can help mitigate against these issues arising.

show abstract

“…Mariani et al, 2013), ELRA-LREC (J. Mariani et al, 2014), L&TC (J. , that they now enlarge to the Speech and Natural Language Processing (SNLP) field in general. They also investigated various aspects of scholar contributions and their evolution over time, such as the production of papers, the collaborations between authors, the citations of papers and authors, the trends in research topics, paper plagiarism and reuse.…”

Section: Text Analytics Of Scientific Papersmentioning

confidence: 99%

Reuse and plagiarism in Speech and Natural Language Processing publications

2017

Self Cite

View full text Add to dashboard Cite

The goal of this paper is to propose measures of innovation through the study of publications in the field of speech and language processing. It is based on the NLP4NLP corpus, which contains the articles published in major conferences and journals related to speech and language processing over 50 years . It represents 65,003 documents from 34 different sources, conferences and journals, published by 48,894 different authors in 558 events, for a total of more than 270 million words and 324,422 bibliographical references. The data was obtained in textual form or as an image that had to be converted into text. This resulted in a lower quality for the most ancient papers, that we measured through the computation of an unknown word ratio. The multi-word technical terms were automatically extracted after parsing, using a set of general language text corpora. The occurrences, frequencies, existences and presences of the terms were then computed overall, for each year and for each document. It resulted in a list of 3.5 million different terms and 24 million term occurrences. The evolution of the research topics over the year, as reflected by the terms presence, was then computed and we propose a measure of the topic popularity based on this computation. The author(s) who introduced the terms were searched for, together with the year when the term was first introduced and the publication where it was introduced. We then studied the global and evolutional contributions of authors to a given topic. We also studied the global and evolutional contributions of the various publications to a given topic. We finally propose a measure of innovativeness for authors and publications.

show abstract

Rediscovering 15 + 2 years of discoveries in language resources and evaluation

Cited by 6 publications

References 13 publications

The NLP4NLP Corpus (I): 50 Years of Publication, Collaboration and Citation in Speech and Language Processing

The NLP4NLP Corpus (I): 50 Years of Publication, Collaboration and Citation in Speech and Language Processing

Ethical Considerations in NLP Shared Tasks

Reuse and plagiarism in Speech and Natural Language Processing publications

Contact Info

Product

Resources

About