How to find your Artificial Intelligence explainer
Our easy-to-use DIANNA library helps you figure out which method works for your research.
by Elena Ranguelova and Yang Liu
Why Explainable Artificial Intelligence?
Explainable Artificial Intelligence (or XAI) is AI that provides interpretable outputs, that can be easily understood by humans. Unfortunately, often there is a trade-off between interpretability and performance — for example, traditional methods like decision trees are very interpretable, but have lower performance compared to the “black-box” (Deep) Neural Networks (DNN). In addition, what is needed to make something “understandable” depends on several factors, like context and the method’s target audience.
Since we would like to have both high-performing and interpretable models, we need XAI. There are many XAI methods to choose from, but which is the “best”? One of the goals of the Deep Insight and Neural Network Analysis (DIANNA) project is to help answer this question for researchers in any scientific domain, especially the non-AI experts. DIANNA provides a standard API for several XAI methods to make them easier for researchers to use. This blog post describes our process of selecting which XAI methods to support.
An overview of explainable AI approaches
There are many ways to approach the explainability of complex opaque models. We have considered the large group of post-hoc (after training) XAI methods. They provide insight into an AI model’s reasoning on a given data item of interest by quantifying the relevance of individual data features (image pixels, words in a text, etc.) with respect to a selected prediction. The resulting relevance scores, overlaid on the inputs, indicate how the decision has been made. The input, in this case, is the (scientific) data which can be of different modalities (image, text, speech, etc.). The XAI visualizes the knowledge captured by the trained AI model, which therefore can become a source of scientific insights.
Many post-hoc XAI methods exist and some apply to many data modalities. Some examples are LIME, SHAP, Layer-wise Relevance Propagation (LRP), Deep Taylor Decomposition (DTD), DeepLIFT, Pattern Attribution and PatternNet and RISE.
Here, we will use the LRP method to illustrate the principle of determining which features in an input contribute most strongly to the output of a trained DNN model. In this example, the input to the network is an image. We are interested in figuring out the relevance of each image pixel to a given image class. This could be the final DNN classification decision for that image (“cat”) or any other class known to the model (e.g. “dog”, “cow”, “bee”, etc. if the model is trained to classify animals). The output relevance is considered a weighted sum of the relevancies of all the pixels of the input image. The relevance at the output node in the network graph is decomposed by iterative propagation backward through the network layers until the input layer. The trained model weights are used in this decomposition. Using this relevance propagation process, the final relevance scores map (or heatmap) is obtained. The final relevance map gives us an indication for the DNN’s hidden logic.
Besides LRP, there are many other XAI methods, which use different ways of determining relevance. In order to determine which of the plethora of post-hoc XAI methods to include in DIANNA, we started by compiling an initial list of 18 XAI post-hoc feature-relevance methods following Arrieta et al. (2020):
- Deep Taylor decomposition (DTD): Paper| Presentation| Code
- Layer-wise Relevance Propagation (LRP): Paper|Blog| Code
- Integrated Gradients (IG): Paper| Tutorial| Code
- Local Interpretable Model-agnostic Explanations (LIME): Paper| Blog| Code
- SHapley Additive exPlanations (SHAP): Paper| Blog|Code
- PatternNet and PatternAttribution: Paper| Code
- Deep Learning Important FeaTures (DeepLIFT): Paper| Tutorial| Presentation| Code
- SmoothGrad: Paper| Demo| Code
- CLass-Enhanced Attentive Response(CLEAR): Paper
- (Adaptive) deconvolutional network (Deconvnet): Paper
- Gradient-weighted Class Activation Mapping (Grad-CAM): Paper| Blog|Code
- Randomized input sampling for explanation of black-box models (RISE): Paper| Code
- Extremal Perturbations (EP)| Paper| Code
- Saliency Analysis (SA): Paper| Code
- Anchors: Paper| Code
- Deep visualization: Paper| Demo| Code
- AutomaticSTRucture IDentification (ASTRID): Paper| Code
We excluded all XAI methods without a working and well-maintained open-source implementation or proven user community. That resulted in a subset of the 13 methods highlighted above.
Evaluation criteria
The large variety of XAI approaches benefits the AI community by providing freedom in the search for explainability. However, it also increases the difficulty of choosing the most appropriate method for each use case. Every XAI approach has pros and cons. In order to select the best methods for our task, we needed a framework to evaluate these methods systematically. Fortunately, this topic has already been visited by the XAI researchers and there is an extensive list of objective criteria defined in the Explainability fact sheet paper [Sokol, K., & Flach, P. (2020)]. The authors look at an abundant set of XAI aspects and cluster the requirements towards an explainer into 5 groups (or dimensions):
- Functional (F1-F9)
- Operational (O1-O10)
- Usability (U1-U11)
- Safety (S1-S4)
- Validation (V1-V2)
For each dimension, there are specific criteria, e.g. the “Functional” group has 9 and in total there are 36 criteria. We will describe our choices next.
The selection
In our DIANNA project, we built a Python package that brings XAI to non (X)AI experts and various scientific domain researchers. The XAI methods to be included in the package had to have the capacity to work with different types of data, various AI architectures and we chose to focus on the model explanation. Most importantly, the resulting explanations had to be easy to interpret. Based on our goals, we have compiled a shortlist of 18 requirements from the 36 requirements in the XAI fact sheet. Using these relevant for our target users' requirements, we evaluated the selected 13 methods listed in the XAI overview section.
Apart from the requirements on the list, we have also considered the similarity and the “popularity” of these methods, the quality of the existing implementations, and the difficulty of integrating them in DIANNA. The evaluation procedure can be summarized with the table shown below: (Deconvnet and GradCAM are filtered out since they can only work with Convolutional NNs and are therefore not included in the table.)
Finally, we narrowed down our list to 5 methods:
- Layer-wise Relevance Propagation (LRP)
- Local Interpretable Model-agnostic Explanations (LIME)
- SHapley Additive exPlanations (SHAP)
- PatternAttribution (PA)
- Randomized input sampling for explanation of black-box models (RISE)
Within limited project time, we were able to include three of them in our DIANNA library: RISE, LIME & SHAP. A preview of the explanations that we obtained using our DIANNA library is shown below:
We could observe that the methods are complimentary to each other in respect to their output, which can be particularly useful for researchers seeking new insights to their data and problem. We also believe that simpler datasets, e.g. a binary (only 0s and 1s) MNIST should be used as “Hello world” example for the XAI (user) community!
Our library is open-source and it is ready to help you decrypt your neural network models. Wait, you want to ask which format do we support? TensorFlow? PyTorch? Don’t worry. We can work with all of them, as we support ONNX — the de-facto DNN format standard! Any contributions to our repository are very welcome (talk is cheap, show me your pull request! Want to give it a try? Check out the link below our logo!
Acknowledgments
The work described in this blog has been performed by the members of the DIANNA team: Christiaan Meijer, Leon Oostrum, Yang Liu and Elena Ranguelova with the big help of Patrick Bos and Giulia Crocioni. Special thanks to Christiaan Meijer, Giulia Crocioni, Matthieu Laneuville, Pablo Rodríguez-Sánchez and Lieke de Boer for their valuable contributions. The DIANNA project is supported by Netherlands eScience Center and SURF.
References
[Arrieta et al. (2020)] Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., … & Herrera, F. (2020). Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information fusion, 58, pp. 82–115.
[Sokol, K., & Flach, P. (2020)] Sokol, K., & Flach, P. (2020, January). Explainability fact sheets: a framework for systematic assessment of explainable approaches. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 56–67.