Understanding the decisions of CNNs: An in-model approach

Rio-Torto, Isabel; Fernandes, Kelwin; Teixeira, Luís

doi:10.1016/j.patrec.2020.04.004

Cited by 19 publications

(7 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These activation maps can be visualized to see what characteristics are being picked up by the network to work toward final prediction results. An understanding of how deep learning models decide is very important for the deployment of robust, transparent, and trustworthy systems in real-world situations (Rio-Torto et al, 2020). Toward this end, most methods adopt a gradient-based approach and produce explanations called heatmaps by propagating pixel-wise relevance backward to the input of the network to highlight .…”

Section: Methodsmentioning

confidence: 99%

Predictability of sea surface temperature anomalies at the eastern pole of the Indian Ocean Dipole—using a convolutional neural network model

et al. 2022

View full text Add to dashboard Cite

In this study, we train a convolutional neural network (CNN) model using a selection of Coupled Model Intercomparison Project (CMIP) phase 5 and 6 models to investigate the predictability of the sea surface temperature (SST) variability off the Sumatra-Java coast in the tropical southeast Indian Ocean, the eastern pole of the Indian Ocean Dipole (IOD). Results show that the CNN model can beat the persistence of the interannual SST variability, such that the eastern IOD (EIOD) SST variability can be forecast up to 6 months in advance. Visualizing the CNN model using a gradient weighted class activation map shows that the strong positive IOD events (cold EIOD SST anomalies) can stem from different processes: internal Indian Ocean dynamics were associated with the 1994 positive IOD, teleconnection from the equatorial Pacific was important in 1997, and cooling off the Australian coast in the southeast Indian Ocean contributed to the 2019 positive IOD. The CNN model overcomes the winter prediction barrier of the IOD, to a large extent due to the frequent transition from a warm state of the Indian Ocean to a negative IOD condition (warm EIOD SST anomalies) over the boreal winter to the following spring period. The forecasting skills of the CNN model are on par with predictions from a coupled seasonal forecasting model (ACCESS-S2), even outperforming this dynamic model in seasons leading to the IOD peaks. The ability of the CNN model to identify key dynamic drivers of the EIOD SST variability suggests that the CMIP models can capture the internal Indian Ocean variability and its teleconnection with the Pacific climate variability.

show abstract

Section: Methodsmentioning

confidence: 99%

Predictability of sea surface temperature anomalies at the eastern pole of the Indian Ocean Dipole—using a convolutional neural network model

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Model explainability is essential for gaining trust and acceptance of AI systems in high-stakes areas, such as healthcare, where reliability and safety are critical [43], [44]. Medical anomaly detection [45], healthcare risk prediction system [46], [47], [48], [49], genetics [50], [51], and healthcare image processing [52], [53], [54] are some of the areas that are moving towards adoption of XAI. Another area is finance, such as AI-based credit score decisions [55], [56] and counterfeit banknotes detection [57].…”

Section: Explainable Artificial Intelligence (Xai)mentioning

confidence: 99%

Explainable Intrusion Detection Systems (X-IDS): A Survey of Current Methods, Challenges, and Opportunities

et al. 2022

View full text Add to dashboard Cite

The application of Artificial Intelligence (AI) and Machine Learning (ML) to cybersecurity challenges has gained traction in industry and academia, partially as a result of widespread malware attacks on critical systems such as cloud infrastructures and government institutions. Intrusion Detection Systems (IDS), using some forms of AI, have received widespread adoption due to their ability to handle vast amounts of data with a high prediction accuracy. These systems are hosted in the organizational Cyber Security Operation Center (CSoC) as a defense tool to monitor and detect malicious network flow that would otherwise impact the Confidentiality, Integrity, and Availability (CIA). CSoC analysts rely on these systems to make decisions about the detected threats. However, IDSs designed using Deep Learning (DL) techniques are often treated as black box models and do not provide a justification for their predictions. This creates a barrier for CSoC analysts, as they are unable to improve their decisions based on the model's predictions. One solution to this problem is to design explainable IDS (X-IDS). This survey reviews the state-of-the-art in explainable AI (XAI) for IDS, its current challenges, and discusses how these challenges span to the design of an X-IDS. In particular, we discuss black box and white box approaches comprehensively. We also present the tradeoff between these approaches in terms of their performance and ability to produce explanations. Furthermore, we propose a generic architecture that considers human-in-the-loop which can be used as a guideline when designing an X-IDS. Research recommendations are given from three critical viewpoints: the need to define explainability for IDS, the need to create explanations tailored to various stakeholders, and the need to design metrics to evaluate explanations.INDEX TERMS Explainable intrusion detection systems, explainable artificial intelligence, machine learning, deep learning, white box, black box, explainability, cybersecurity.

show abstract

“…In contrast to the previous approaches, Rio-Torto et al [102] proposed an in-model joint architecture composed of an explainer and a classifier to produce visual explanations for the predicted class labels. The explainer consists of an encoder-decoder network based on U-Net, and the classifier is based on VGG-16.…”

Section: Saliencymentioning

confidence: 99%

“…Recently, Rio-Torto et al [102] proposed the POMPOM (Percentage of Meaningful Pixels Outside the Mask) metric, which report the number of meaningful pixels outside the region of interest in relation to the total number of pixels, to evaluate the quality of a given explanation. Similarly, Barnett et al [13] introduced the Activation Precision evaluation metric, to quantify the proportion of relevant information from the "relevant region" used to classify the mass margin regarding the radiologist annotations.…”

Section: Evaluating the Quality Of Visual Explanationsmentioning

confidence: 99%

“…However, considering variability in experts opinions [130], this strategy is particularly biased and subjective. On the other hand, the existing strategies for objectively measuring the quality of visual explanations are still dependent on manual annotations of relevant regions [102], or iterative model retraining (ROAR [45]). For these reasons, we believe that the design of objective metrics for assessing the quality of the model explanations will be one of the important research rends on the topic of XAI.…”

Section: Challenges and Future Research Trendsmentioning

confidence: 99%

See 1 more Smart Citation

Explainable Deep Learning Methods in Medical Imaging Diagnosis: A Survey

Patrício¹,

Neves²,

Teixeira³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The remarkable success of deep learning has prompted interest in its application to medical diagnosis. Even tough state-ofthe-art deep learning models have achieved human-level accuracy on the classification of different types of medical data, these models are hardly adopted in clinical workflows, mainly due to their lack of interpretability. The black-box-ness of deep learning models has raised the need for devising strategies to explain the decision process of these models, leading to the creation of the topic of eXplainable Artificial Intelligence (XAI). In this context, we provide a thorough survey of XAI applied to medical diagnosis, including visual, textual, and example-based explanation methods. Moreover, this work reviews the existing medical imaging datasets and the existing metrics for evaluating the quality of the explanations . Complementary to most existing surveys, we include a performance comparison among a set of report generation-based methods. Finally, the major challenges in applying XAI to medical imaging are also discussed.CCS Concepts: • Applied computing → Health care information systems.

show abstract

Understanding the decisions of CNNs: An in-model approach

Cited by 19 publications

References 8 publications

Predictability of sea surface temperature anomalies at the eastern pole of the Indian Ocean Dipole—using a convolutional neural network model

Predictability of sea surface temperature anomalies at the eastern pole of the Indian Ocean Dipole—using a convolutional neural network model

Explainable Intrusion Detection Systems (X-IDS): A Survey of Current Methods, Challenges, and Opportunities

Explainable Deep Learning Methods in Medical Imaging Diagnosis: A Survey

Contact Info

Product

Resources

About