Design factors affecting relevance judgment behaviour in the context of metadata surrogates

Abstract

For the full text of this licence, please go to: http://creativecommons.org/licenses/by-nc-nd/2.5/ Design factors affecting user interaction and relevance judgment behaviour with metadata surrogates: the case of structure, highlighting and quantity of information AbstractThis paper reports on the findings of a user study which investigated the effects of structure, highlighting and quantity of information on university students' interaction with metadata surrogates of learning object repositories (LORs) during the relevance judgment process. The literature review revealed a lack of studies investigating how metadata surrogates should be designed to meet users' needs and improve the effectiveness of relevance judgment. In order to address this issue, different versions of a prototype system, called META-LOR 2, were developed and evaluated in terms of the time needed for users to find relevant information, the accuracy of their relevance judgments and their satisfaction. To evaluate the prototype, a user study was conducted where test subjects were asked to complete a set of tasks and fill in a satisfaction questionnaire. The findings showed that participants performed better and were more satisfied with those versions of the prototype which highlighted metadata elements that included the query terms, used clusters or categories to organize metadata elements in the surrogates and included metadata elements which were relevant to the query or task at hand. Participants performed less efficiently and were less satisfied with metadata surrogates which lacked the use of a clear structure, did not highlight the query terms and contained a large number of metadata elements. It is anticipated that the findings of this research will enhance understanding of users' interaction with metadata surrogates during the relevance judgment process and improve the design of the search result interfaces of educational information retrieval systems, such as LORs.