A PROMISE for Experimental Evaluation

Abstract

Abstract. Participative Research labOratory for Multimedia and Multilingual Information Systems Evaluation (PROMISE) is a Network ofExcellence, starting in conjunction with this first independent CLEF 2010 conference, and designed to support and develop the evaluation of multilingual and multimedia information access systems, largely through the activities taking place in Cross-Language Evaluation Forum (CLEF) today, and taking it forward in important new ways.PROMISE is coordinated by the University of Padua, and comprises 10 partners: the Swedish Institute for Computer Science, the University of Amsterdam, Sapienza University of Rome, University of Applied Sciences of Western Switzerland, the Information Retrieval Facility, the Zurich University of Applied Sciences, the Humboldt University of Berlin, the Evaluation and Language Resources Distribution Agency, and the Centre for the Evaluation of Language Communication Technologies.The single most important step forward for multilingual and multimedia information access which PROMISE will work towards is to provide an open evaluation infrastructure in order to support automation and collaboration in the evaluation process.

PROMISE retreat report prospects and opportunities for information access evaluation

“…We envision a methodology which allows for continuous evaluation [19] and assessment of retrieval results also during the retrieval process, in a trial-take decision-trial cyclic setting. This cyclic setting presents three main challenges.…”