Using Visual Text Mining to Support the Study Selection Activity in Systematic Literature Reviews

Felizardo, Katia Romero; Salleh, Norsaremah; Martins, Rafael Messias; Mendes, Emilia; MacDonell, Stephen G.; Maldonado, José Carlos

doi:10.1109/esem.2011.16

Cited by 47 publications

(54 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…[23,25]. However, more work needs to be done given the fact that SR is now cutting across disciplines from medicine to social science, software engineering and computer science.…”

Section: Discussionmentioning

confidence: 99%

“…Data source (DS): Out of the 35 papers reviewed, only [21][22][23][24][25] did not mention the source of their raw data but they all provided information on the structure (usually title, abstract and optional keywords or metadata) of their data; none of them provided the link to the data except [5,11,30,48]. The datasets are no more available at the links provided in [30,48].…”

Section: Study Replicationmentioning

confidence: 99%

See 1 more Smart Citation

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

Olorisade

Quincey

Brereton

et al. 2016

Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

View full text Add to dashboard Cite

Background: Since the introduction of the systematic review process to Software Engineering in 2004, researchers have investigated a number of ways to mitigate the amount of effort and time taken to filter through large volumes of literature.Aim: This study aims to provide a critical analysis of text mining techniques used to support the citation screening stage of the systematic review process. Method:We critically re-reviewed papers included in a previous systematic review which addressed the use of text mining methods to support the screening of papers for inclusion in a review. The previous review did not provide a detailed analysis of the text mining methods used. We focus on the availability in the papers of information about the text mining methods employed, including the description and explanation of the methods, parameter settings, assessment of the appropriateness of their application given the size and dimensionality of the data used, performance on training, testing and validation data sets, and further information that may support the reproducibility of the included studies.Results: Support Vector Machines (SVM), Naïve Bayes (NB) and Committee of classifiers (Ensemble) are the most used classification algorithms. In all of the studies, features were represented with Bag-of-Words (BOW) using both binary features (28%) and term frequency (66%). Five studies experimented with n-grams with n between 2 and 4, but mostly the unigram was used. χ 2 , information gain and tf-idf were the most commonly used feature selection techniques. Feature extraction was rarely used although LDA and topic modelling were used. Recall, precision, F and AUC were the most used metrics and cross validation was also well used. More than half of the studies used a corpus size of below 1,000 documents for their experiments while corpus size for around 80% of the studies was 3,000 or fewer documents. The major common ground we found for comparing performance assessment based on independent replication of studies was the use of the same dataset but a sound performance comparison could not be established because the studies had little else in common. In most of the studies, insufficient information was reported to enable independent replication. The studies analysed generally did not include any discussion of the statistical appropriateness of the text mining method that they applied. In the case of applications of SVM, none of the studies report the number of support vectors that they found to indicate the complexity of the prediction engine that they use, making it impossible to judge the extent to which over-fitting might account for the good performance results. Conclusions:There is yet to be concrete evidence about the effectiveness of text mining algorithms regarding their use in the automation of citation screening in systematic reviews. The studies indicate that options are still being explored, but there is a need for better reporting as well as more explicit process details and access to datasets to facilitate st...

show abstract

“…[23,25]. However, more work needs to be done given the fact that SR is now cutting across disciplines from medicine to social science, software engineering and computer science.…”

Section: Discussionmentioning

confidence: 99%

Section: Study Replicationmentioning

confidence: 99%

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

Olorisade

Quincey

Brereton

et al. 2016

Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering

View full text Add to dashboard Cite

show abstract

“…In this context, Marshall and Brereton (2013) report some tools available in the literature to support researchers conducting SRs. Two of them -StArt (State of the Art through Systematic Review) (Fabbri et al 2012) and Revis (Systematic Literature Review based on Visual Analytics) (Felizardo et al 2011) -present different and complementary features to support the decision-making process associated with the selection activity. StArt classifies the relevance of each primary study based on content relationships, and Revis uses citation relationships.…”

Section: Introductionmentioning

confidence: 99%

Semi-automatic selection of primary studies in systematic literature reviews: is it reasonable?

et al. 2014

Self Cite

View full text Add to dashboard Cite

The systematic review (SR) is a methodology used to find and aggregate all relevant existing evidence about a specific research question of interest. One of the activities associated with the SR process is the selection of primary studies, which is a time consuming manual task. The quality of primary study selection impacts the overall quality of SR. The goal of this paper is to propose a strategy named "Score Citation Automatic Selection" (SCAS), to automate part of the primary study selection activity. The SCAS strategy combines two different features, content and citation relationships between the studies, to make the selection activity as automated as possible. Aiming to evaluate the feasibility of our strategy, we conducted an exploratory case study to compare the accuracy of selecting primary studies manually and using the SCAS strategy. The case study shows that for three SRs published in the literature and previously conducted in a manual implementation, the average effort reduction was 58.2 % when applying the SCAS strategy to automate part of the initial selection of primary studies, and the percentage error was 12.98 %. Our case study provided confidence in our strategy, and suggested that it can reduce the effort required to select the primary studies without adversely affecting the overall results of SR.Empir Software Eng

show abstract

“…Details about the stages to create a content map (i.e. pre-processing; similarity calculation; and projection) can be found in [12,13].…”

Section: Vtm Techniquesmentioning

confidence: 99%

Visual Text Mining: Ensuring the Presence of Relevant Studies in Systematic Literature Reviews

Felizardo

Barbosa

Martins

et al. 2015

Int. J. Soft. Eng. Knowl. Eng.

View full text Add to dashboard Cite

One of the activities associated with the Systematic Literature Review (SLR) process is the selection review of primary studies. When the researcher faces large volumes of primary studies to be analyzed, the process used to select studies can be arduous. In a previous experiment, we conducted a pilot test to compare the performance and accuracy of PhD students in conducting the selection review activity manually and using Visual Text Mining (VTM) techniques. The goal of this paper is to describe a replication study involving PhD and Master students. The replication study uses the same experimental design and materials of the original experiment. This study also aims to investigate whether the researcher's level of experience with conducting SLRs and research in general impacts the outcome of the primary study selection step of the SLR process. The replication results have con¯rmed the outcomes of the original experiment, i.e., VTM is promising and can improve the performance of the selection review of primary studies. We also observed that both accuracy and performance increase in function of the researcher's experience level in conducting SLRs. The use of VTM can indeed be bene¯cial during the selection review activity.

show abstract

Using Visual Text Mining to Support the Study Selection Activity in Systematic Literature Reviews

Cited by 47 publications

References 14 publications

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

A critical analysis of studies that address the use of text mining for citation screening in systematic reviews

Semi-automatic selection of primary studies in systematic literature reviews: is it reasonable?

Visual Text Mining: Ensuring the Presence of Relevant Studies in Systematic Literature Reviews

Contact Info

Product

Resources

About