Three-way decision and conformal prediction: Isomorphisms, differences and theoretical properties of cautious learning approaches

Abstract

The aim of this article is to study the relationship between two popular Cautious Learning approaches, namely: Three-way decision (TWD) and conformal prediction (CP). Based on the novel proposal of a technique to transform threeway decision classifiers into conformal predictors, and vice-versa, we provide conditions for the equivalence between TWD and CP. These theoretical results provide error-bound guarantees for TWD, together with a formal construction to define cost-sensitive cautious classifiers based on CP. The proposed techniques are then applied and evaluated on a collection of benchmark and real-world datasets. The results of the experiments show that the proposed techniques can be used to obtain cautious learning classifiers that are competitive with, and often out-perform, state-of-the-art approaches. Further, through a qualitative medical case study we discuss the usefulness of cautious learning in the development of robust Machine Learning.

Learning from fuzzy labels: Theoretical issues and algorithmic solutions

“…4 In the literature, the term Rademacher complexity can be used to denote either the empirical Rademacher complexity, as defined in Eq. (17), or its expectation R(H) = E S∼D m [R(H, S)]. The two versions of the Rademacher complexity are, obviously, related to each other: in particular, the expectation version can be bounded, with high probability, by the empirical one.…”

“…-Sagittal misalignment assessment [17], as an example of a superset learning task. In this case, two medical specialists annotated all the instances in the datasets, and the sets of labels were obtained by simply selecting, for each instance x, all the labels associated with x.…”

Exploring the State of Machine Learning and Deep Learning in Medicine: A Survey of the Italian Research Community

“…In the period analyzed, the top 3 journals for the number of publications are IEEE Access, Scientific Reports, and Applied Science. Notably, not all the 224 papers are analyzed with our methodology, since 10 of these (i.e., [25][26][27][28][29][30][31][32][33][34]) proposed new ML/DL methodologies, metrics, or approaches that may be strongly related to the medical area, but they are not focused on a specific disease or on a particular case study. However, we believe that it is very important that the Italian community not only provides a bridge between the ML/DL area and the medical area but also proposes solutions to the general issues, which arise from the peculiarity of the medical field.…”

Three-way Learnability: A Learning Theoretic Perspective on Three-way Decision

“…This approach attracted a large interest, also justified by promising empirical results in different ML tasks such as active learning [8], [9], cost-sensitive classification [10], clustering [11], [12], [9]. Despite these promising empirical results, the theoretical foundations of TWD-based ML received so far little attention [13], [14]. Indeed, even though, in the recent years, there has been an increasing interest toward generalizing computation learning theory (CLT) to cautious inference methods such as selective prediction [15] or the KWIK (Knows what it Knows) framework [16], such results cannot be easily applied to the TWD setting: While in the TWD setting abstention is a property of single classifiers; in the latter two frameworks abstention is usually achieved by consensus voting.…”