Since the onset of the COVID-19 pandemic in 2019, many clinical prognostic scoring tools have been proposed or developed to aid clinicians in the disposition and severity assessment of pneumonia. However, there is limited work that focuses on explaining techniques that are best suiteds for clinicians in their decision making. In this paper, we present a new image explainability method named Ensemble XAI, which is based on the SHAP and Grad-CAM++ methods. It provides a visual explanation for a deep learning prognostic model that predicts the mortality risk of community-acquired pneumonia and COVID-19 respiratory infected patients. In addition, we surveyed the existing literature and compiled prevailing quantitative and qualitative metrics to systematically review the efficacy of Ensemble XAI, and to make comparisons with several state-of-the-art explainability methods (LIME, SHAP, Saliency Map, Grad-CAM, Grad-CAM++). Our quantitative experimental results have shown that Ensemble XAI has an comparable absence impact (decision impact: 0.72, confident impact: 0.24). Our qualitative experiment, in which a panel of 3 radiologists were involved to evaluate the degree of concordance and trust in the algorithms, has showed that Ensemble XAI has localization effectiveness (mean set accordance precision: 0.52, mean set accordance recall: 0.57, mean set 𝑭 𝟏 : 0.50, mean set IOU: 0.36) and is the most trusted method by the panel of radiologists (mean vote: 70.2%). Finally, the deep learning interpretation dashboard used for the radiologist panel voting will be made available to the community. Our code is available at https://github.com/IHIS-HealthInsights/Interpretation-Methods-Voting-dashboard. Impact Statement -Compared to other sectors that have deployed artificial intelligent (AI), the use of AI in healthcare understandably requires closer scrutiny due to the potential risks to patient safety, especially for clinical AI. As such, AI Explainability (XAI) is a key focus area in regard to the adoption of AI in healthcare. However, most of the current XAI methods for medical imaging revolve around quantitative assessment and there is a lack of systematic qualitative studies that seek to gain trust and concordance with clinicians. In this paper, we worked with a panel of clinicians to devise a comprehensive XAI evaluation framework combining quantitative and qualitative metrics to systematically review the efficacy of XAI techniques on deep learning models for pneumonia medical imaging. More importantly, we developed a new image explainability algorithm named Ensemble XAI, which gained the most trust by the panel of radiologists with a mean vote of 70.2%. It is envisioned that with the proposed XAI evaluation