“…The effects of incomplete relevance assessments, imperfect judgements, potential biases in the relevance pool and the effects of assessor domain expertise in relation to the topic have been investigated in various studies (Cuadra, 1967;Zobel, 1998;Buckley and Voorhees, 2004;Yilmaz and Aslam, 2006;Büttcher et al, 2007;Bailey et al, 2008;Kinney et al, 2008). Approaches to ensure completeness of relevance assessments include using the results from searches conducted manually to generate the pools and supplementing pools with relevant documents found by manually searching the document collection with an IR system, known as Interactive Search and Judge or ISJ (Cormack et al, 1998) Generating relevance assessment is often highly timeconsuming and labour intensive. This often leads to a bottleneck in the creation of test collections.…”