Relief-based feature selection: Introduction and review

Abstract

Feature selection plays a critical role in biomedical data mining, driven by increasing feature dimensionality in target problems and growing interest in advanced but computationally expensive methodologies able to model complex associations. Specifically, there is a need for feature selection methods that are computationally efficient, yet sensitive to complex patterns of association, e.g. interactions, so that informative features are not mistakenly eliminated prior to downstream modeling. This paper focuses on Relief-based algorithms (RBAs), a unique family of filter-style feature selection algorithms that have gained appeal by striking an effective balance between these objectives while flexibly adapting to various data characteristics, e.g. classification vs. regression. First, this work broadly examines types of feature selection and defines RBAs within that context. Next, we introduce the original Relief algorithm and associated concepts, emphasizing the intuition behind how it works, how feature weights generated by the algorithm can be interpreted, and why it is sensitive to feature interactions without evaluating combinations of features. Lastly, we include an expansive review of RBA methodological research beyond Relief and its popular descendant, ReliefF. In particular, we characterize branches of RBA research, and provide comparative summaries of RBA algorithms including contributions, strategies, functionality, time complexity, adaptation to key data characteristics, and software availability.

Unsuspected involvement of spinal cord in Alzheimer Disease

“…Ranking was implemented with the ReliefF algorithm 34 All results are reported in Table 2 and Table 3. AD patients compared to HC showed atrophy in all 223 brain structures.…”

Theoretical properties of nearest-neighbor distance distributions and novel metrics for high dimensional bioinformatics data

“…1 is asymptotically normal. Specifically, when q = 2, the distribution of D (2) ij asymptotically approaches N √ µ z 2 a p,…”

“…Based on the iid assumption for X ia and X ja , it follows from Thm. 2.1 that the joint density function g (2) of X ia and Z a is given by…”

“…Let F Z q a denote the distribution function of the random variable Z q a . Furthermore, we define the events E (1) and E (2) as…”

“…For random normal data (N (0, 1)), for example, the variance of the pairwise distances of a Manhattan metric is proportional to the number of attributes (p) whereas the variance is constant for a Euclidean metric. Relief methods [1][2][3] and nearest-neighbor projected distance regression (NDPR) [4] use nearest neighbors to compute attribute importance scores and often use adaptive neighborhoods that rely on the mean and variance of the distance distribution. The ability of this class of methods to identify association effects, like main effects or interaction effects, depends on parameters such as neighborhood radii or number of neighbors k [12,13].…”