Improvements that don't add up

Evaluation-as-a-Service for the Computational Sciences

“…• access to annotated data sets and a clear evaluation scenario, making it quick and easy to publish if the results are good; • access to data sets that would be too big to be shared and that a single research group or a small group could not assemble and treat; • access to sensitive data such as medical data but also in other domains (copyrighted music, enterprise search data). Without EaaS the companies would likely not share the data but maybe work on it in-house only, such as large search engine companies currently do with their log files; • get a comparison to strong baselines, so other techniques and algorithms do not need to be reimplemented and then optimized; this has the reverse risk that it can make one's own results look less positive than comparing to a low baseline [Armstrong et al 2009]; • get impact via publications, mainly by reusing the data after the end of competitions for further publications; • if sharing of components is done, then this could also give more visibility, citations and reputation but this is currently not very often the case; • advertisement via demos that are dissemination channels of own techniques; • workshops to discuss with people working on the same data to get ideas on new approaches and avoid mistakes others have done but not published, as publications of negative results are rare; • access to a broader range of challenges and testing own tools on the data best adapted for them; • potentially better contacts to business partners if the challenges are proposed by a company for example, this could also lead to job offers for graduate students.…”

“…The availability of these resources should contribute to addressing an observed practice in computer science of comparing new algorithms to weak baselines [Armstrong et al 2009]. It should also ensure access to results using a large palette of metrics, so that all aspects of the performance of an approach can be examined.…”

Bibliometric-Enhanced Information Retrieval

“…Retrieval evaluations have shown that simple text-based retrieval methods scale up well but do not progress (Armstrong et al, 2009). Traditional retrieval has reached a high level in terms of measures like precision and recall, but scientists and scholars still face challenges present since the early days of digital libraries: mismatches between search terms and indexing terms, overload from result sets that are too large and complex, and the drawbacks of text-based relevance rankings.…”

Living Labs for Academic Search at CLEF 2020

“…The need for innovation in this field of academic search and IR, in general, is shown by the stagnating system performance in controlled evaluation campaigns, as demonstrated in TREC and CLEF meta-evaluation studies [1,11], as well as user studies in real systems of scientific information and digital libraries. Even though large amounts of data are available in highly specialized subject databases (such as ArXiV or PubMed), digital libraries (like the ACM Digital Library), or web search engines (Google Scholar or SemanticScholar), many user needs and requirements remain unsatisfied.…”