Finding better active learners for faster literature reviews

Abstract

Literature reviews can be time-consuming and tedious to complete. By cataloging and refactoring three state-of-the-art active learning techniques from evidence-based medicine and legal electronic discovery, this paper finds and implements FASTREAD, a faster technique for studying a large corpus of documents, combining and parametrizing the most efficient active learning algorithms. This paper assesses FASTREAD using datasets generated from existing SE literature reviews (Hall, Wahono, Radjenović, Kitchenham et al.). Compared to manual methods, FASTREAD lets researchers find 95% relevant studies after reviewing an order of magnitude fewer papers. Compared to other state-of-the-art automatic methods, FASTREAD reviews 20-50% fewer studies while finding same number of relevant primary studies in a systematic literature review.

Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction)

“…All those aforementioned tactics are built into EMBLEM [114], [116], the active learner used for this work. When reading commits, EMBLEM initially uses uncertainty sampling to fast build a classification model (for bug-fixing or non bug-fixing commit message), then switches to certainty sampling to greedily find bug-fixing commits.…”

“…values {N 1 = 4000, N 2 = 1, N 3 = 30, N 4 = 95%}). Those decisions where made by Yu et al [114] after exploring 32 different kinds of active learners. They report that, using the above requirements, EMBLEM found more relevant items faster than the previously reported state-of-the-art in incremental text mining retrieval [22], [102].…”

“…Our pre-experimental belief was that EMBLEM would require extensive tuning before it could be used for labelling Github commits. However, the effectiveness of EMBLEM was obtained using Yu et al's original decisions [114], [116] without extensive tuning. Future improvements can be achieved by tuning different settings.…”

“…Also, we could explore other control parameters for EMBLEM. All the above results were obtained using Yu et al's [114], [116] original requirements (e.g. the values of {N 1 = 4000, N 2 = 1, N 3 = 30, N 4 = 95%}) within the EMBLEM method.…”

TERMINATOR: better automated UI test case prioritization

“…Given the information available in automated UI testing, we extract three types of features: • Text feature: the same text mining feature extraction used in the total recall approaches [56,57] Using the foregoing types of features, the proposed framework is described in Algorithm 1 with engineering choices of N 1 , N 2 . N 1 is the batch size of the process.…”