Analysis of a token density metric for concern detection in Matlab sources using UbiSOM

Abstract

Matrix and data manipulation programming languages are an essential tool for data analysts. However, these languages are often unstructured and lack modularity mechanisms. This article presents a knowledge discovery approach for studying manifestations of the lack of modularity support in that sort of languages. The study is focused on Matlab, as a well‐established representative of those languages. We present a technique for the automatic detection and quantification of concerns in Matlab and their exploration in a code base. The Ubiquitous Self Organizing Map (UbiSOM) is used to perform exploratory data analysis over concerns detected in a, possibly changing, repository of Matlab files. The UbiSOM is quite effective in detecting patterns of co‐occurrence of multiple concerns. To illustrate the technique, a repository comprising over 35,000 Matlab files is analysed. The results show that the use of Token Density metrics in conjunction with UbiSOM enables the detection of patterns of co‐occurrence of multiple concerns in m‐files.

Fourth special issue on knowledge discovery and business intelligence

“…The first paper, entitled "Analysis of a Token Density Metric for Concern Detection in Matlab Sources using UbiSOM" is related with the MSR field. Marques et al (2018) perform an exploratory data analysis, based on the Ubiquitous Self Organising Map (UbiSOM), to detect software concerns (i.e., cohesive set of functionalities enclosed in a module) in the Matlab language. The UbiSOM technique was illustrated using a repository with 35,000 Matlab files and the visual analytics results revealed interesting concern co-occurrence patterns.…”

An annotated repository for MATLAB code (S)

“…The latter often correspond to standalone programs or libraries made available for the user community. Lack of modularity makes it hard to obtain well-organised code that is easy to read and reuse [1] [3]- [5]. These shortcomings motivated ongoing research to study the symptoms induced by the lack of modularity [3] and use of that knowledge for the detection of unmodularised concerns in existing systems [6] [5].…”

“…The above research on concern detection techniques is centred on the idea of decomposing m-files into its low-level elements [3] and derive a number of metrics based on that information [6] [5]. A number of analysis components can subsequently be plugged into the system to derive higher-level information.…”

“…Developing and maturing the concern detection techniques entailed the assembling of repositories of many MATLAB systems in order to exercise and test functionalities. A number of metrics were derived, which were extracted from the code base by means of a tool that includes a lexical analyser for MATLAB [3] [6] [5]. Activities on the code repository included the performing of many kinds of search, e.g., to find new patterns, perform studies, check results.…”

“…Past work illustrated one potential use of the proposed repository through the implementation of a high level knowledge from a Self Organizing Map (SOM) data mining model that derives patterns from the code to enable the semi-automatic detection of (possibly unmodularised) concerns [6] [5]. By semi-automatic, we mean a process that starts with a machine learning classification phase, whose output can be corrected by a human or enriched by new annotations.…”