“…Most research focuses on issues of the black box from a technological perspective, with limited attention given to the need for interpretability from a medical perspective. In reviewing the literature, the needs of clinicians include eight categories: (1) visualization representation of a process or clinical variable proxies for clinician decision-making [2,7,22,29,30,51,52,54], (2) accessibility and reliability of patients' data [4,17,19,[49][50][51]55], (3) interface of doctors-patients or human-computer interaction for interpreting outcomes [4,22,50], (4) transparent structure for users to validate outputs of the model with domain knowledge [2,7,20], (5) identifcation of biomarkers for supporting decision-making [3,29,51], (6) feature selection distilling information overload [9,19,20,52], (7) rule of representation for knowledge [2,19,20,53], and (8) clinicians' needs incorporated into the clinical workfow [7]. Te needs of patients for interpretability include (1) collecting patients' data of symptoms, physical exams, treatment, and reports of procedures and laboratory tests [4,17,19,50], (2) interface of doctors-patients interaction for interpreting outcomes [50], (...…”