Binary Relevance Multi-label Conformal Predictor

Well-calibrated confidence measures for multi-label text classification with a large number of labels

“…Γ 𝜖 𝑥 𝑛+𝑚 ⊆ ({Ψ 1 , … , Ψ 𝑑 }), with at most 𝜖 chance of not containing the true label-set 𝑦 𝑛+𝑚 . This is the guarantee provided by the approaches proposed in [3] and [41]. The other variations on the other hand provide somewhat weaker guarantees.…”

“…Different variations of CP have been proposed for handling multi-label classification problems through problem reformulation. These include decomposing each multi-label instance into a number of single-label examples (Instance Reproduction (IR)) [39]; or decomposing the problem into several independent binary classification problems -one for each unique class (Binary-Relevance (BR)) [3,40]; or transforming the problem into a multi-class one by treating each label-set as a unique class (Label Powerset (LP)) [41].…”

“…Firstly, we want our approach to provide equally strong guarantees as those provided by CP in other settings. Secondly, the BR approach proposed in [3] applies CP independently for each label and then combines the resulting prediction sets. In order to produce valid prediction sets for a significance level 𝜖 without making any extra assumptions, the significance level of the individual CPs should be set to 𝜖∕𝑑 (where 𝑑 is the number of labels) as derived from the Bonferroni general inequality.…”

“…Approaches are grouped into two broad categories: problem (or pattern) transformation methods and algorithm adaptation methods, as outlined in [2]. In problem transformation methods the multi-label problem is reformulated, usually as a binary classification problem, as in [3], or as a multi-class classification problem using all relevant category combinations, i.e. power-set approach, as in [4].…”