Towards Deep Explanation in Machine Learning Supported by Visual Methods


Boris Kovalerchuk1, Muhammad Aurangzeb Ahmad2,3, Ankur Teredesai2,3


1. Dept. of Computer Science, Central Washington University

2. Dept. of Computer Science and Systems, University of Washington Tacoma

3. KenSci Inc.



Interpretability of Machine Learning (ML) models is a major area of current research, applications and debates in AI. The debates include a statement that most of the interpretation methods are not interpretable [ICCV 2019, https://arxiv.org/abs/1909.07082] and a call to stop explaining black box ML models for high stakes decisions and use interpretable models Instead [KDD 2019, https://arxiv.org/abs/1811.10154].

          This tutorial covers the state-of-the-art research, development, and applications in the area of Interpretable Knowledge Discovery and ML boosted by Visual Methods. The topic is interdisciplinary, bridging efforts of research and applied communities in AI, Machine Learning, Visual Analytics, Information Visualization, and HCI. This is a novel and fast-growing area with significant applications, and potential due to its importance in applications and prominence of visual ways of human cognition and perception. The recent progress in this area is very evident with a major deep learning explanation approach, based on visualization of salient areas and methods to visualize similarity of high-dimensional data in deep learning and other ML studies.  Multiple techniques are emerging including lossless and reversible methods to visualize high-dimensional data, which will be presented in this tutorial to stimulate studies beyond heatmaps, t-SNE, and black-box ML models in general.  


 Tutorial outline

The following topics and issues will be covered:

1.      Foundations of Interpretability

·       How interpretable are current interpretation methods?  The mentioned position expressed at ICCV 2019 is that most of them are not interpretable [https://arxiv.org/abs/1909.07082] and rely on visual interpretability to evaluate and prove explanations

·       What are the reasons that those explanation models are not explainable?

·       When black-box methods can explain black-box models?

·       How to make interpretation methods interpretable?


2.      Limits of Visual Interpretability in Deep Learning

·        To what extend the visual explanations of deep learning models are complete, and interpretable explanations, e.g., dominant heatmap visualization of salient areas, found by activation analysis.

·        Where are there disconnects between how humans impose semantics on deep learning explanations and what they may actually mean?

·        How interpretable is the process of tracing Deep Neural Network (DNN) back to find salient areas, when intermediate network layers have no direct meaning, in the user domain, and how to make this process interpretable? 


3.      User-centric Interpretability

·        How to make an explanation from the salient areas without referencing the internal DNN properties? Why are salient areas important in the user domain terms?

·        What is the role of the human interpreter in justifying and confirming such visual explanations? What is the value of the visual explanation, when an interpreter cannot make a sense of it?

·        What needs to be interpretable when we interpret ML model? Interpretability is a system wide phenomenon, features, parameters, and even insight delivery must be interpretable.


4.      Discovering visual interpretable models

·        How to make visual interpretable learning models with interpretable features vs. non-interpretable features?  

·        What are the ways to provide visual understanding of the learning models for non-image data, including data based on images?

·        What are strengths and weaknesses of current methods to discover the visual interpretable models?


5.      Open Problems and Current Research Frontiers

·        What is the Fidelity of the explanation? Many, if not most, explanations are wrong, while some explanations are useful. Requiring absolute fidelity in interpretable ML is unwarranted, given the complexity of models involved. What are the “good enough” models that allow debugging?

·        How to overcome human cognitive limitations to understand the complex explanation?  A picture is worth a thousand words. Humans have cognitive limitations with respect to the amount of information, which they can process at a time. Cognitive studies show that visual explanations allow processing of information in visual examples much faster, as compared to the other forms of explanations. Here the focus will be on examples from the real world, where visual explanations outperform.

·        How to benefit from Cross Domain Pollination?  What and how visual methods have been applied from non-image related ML tasks, which have been successful in the image domain, and what promise do they hold for the future?

·        How to overcome the model limitations in the search for an explanation? Explanations are only as good as the underlying model.

         The main motivation for the last part of the tutorial is stimulating the research and consensus, on the open problems of the deep interpretability, of ML models to be solved. The tutorial will be structured to present these topics in the listed order, with about 30 min for the discussion with the audience, at the end of the tutorial on open the problems and current research frontiers.

          We will use multiple relevant sources listed in the references including the recent book by Dr. Kovalerchuk “Visual Knowledge Discovery and Machine Learning” (Springer, 2018) and his upcoming chapter in “Handbook of Machine Learning for Data Science” (Springer, 2020).  Drs. Kovalerchuk, Ahmad and Teredesai delivered relevant tutorials at ACM KDD Conferences in 2018 and 2019. Dr. Kovalerchuk delivered relevant tutorials at WSDM 2020, ODSC West 2019, HCII 2018, IJCNN 2017, and at several universities in the US, Europe, and China. Additionally, Dr. Ahmad and Dr. Teredesai have given relevant tutorials at BCB 2018, ICHI 2019 and an ACM seminar.


  Tutorial and IJCAI Audience interests 

The target audience for the tutorial are AI/ML researchers, students and practitioners with basic knowledge of Machine Learning. 


Several tutorials at the prior conferences were devoted to understanding and interpretability of the Deep Learning ML models. While they presented many aspects of interpretability, a number of recent developments in deployment and usage of such systems, and new issues of fairness, accountability, transparency and ethics (FATE) in machine learning have come to the fore. 

       In this tutorial, we seek to address the recent issues in constructing explainable ML models, boosted by visual methods, not limited by explanation of deep learning models. We will present emerging methodologies, which employ the visual methods for moving, from the current quite shallow explanations, to deep explanations to provide the more structured, interpretable and causal models, with better explainable features and relations. The goal of the tutorial is to present the state of the art in this area, including methods, successful experimental studies, and open problems. The tutorial will contain the comparative analysis of explicit vs. implicit explanations, model-agnostic vs. model-specific explanations, visual human-friendly vs. unfriendly explanation, and others.

        Tutorial slides will be available to attendees. We will also create and maintain a website with additional resources related to the tutorial.



Boris Kovalerchuk,

Professor, Dept. of Computer Science at Central Washington University, USA, borisk@cwu.edu


Bio: Dr. Boris Kovalerchuk is a professor of Computer Science at Central Washington University, USA. His publications include three books "Data Mining in Finance" (Springer, 2000), "Visual and Spatial Analysis" (Springer, 2005), and "Visual Knowledge Discovery and Machine Learning" (Springer, 2018), chapters in the Data Mining/Machine learning Handbooks, and over 170 other publications. His research and teaching interests are in machine learning, visual analytics, visualization, uncertainty modeling, image and signal processing, and data fusion. Dr. Kovalerchuk has been a principal investigator of research projects in these areas, supported by the US Government agencies. He served as a senior visiting scientist at the US Air Force Research Laboratory, and as a member of expert panels at the international conferences, and panels organized by the US Government bodies. Prof. Kovalerchuk regularly teach classes on AI, Data Mining, Machine Learning, Information and Data Visualization, Visual Knowledge Discovery at Central Washington University using his books and other sources. He also have been teaching these topics at several other Universities in the US and abroad. Dr. Kovalerchuk delivered relevant tutorials at IJCNN 2017, HCII 2018, KDD 2019, ODSC West 2019; WSDM 2020.



Muhammad Aurangzeb Ahmad

Affiliate Assistant Professor, Dept. of Computer Science and Systems, University of Washington Tacoma, USA maahmad@uw.edu


Bio: Muhammad Aurangzeb Ahmad is an Affiliate Assistant Professor in the Department of Computer Science at University of Washington Tacoma and the Principal Research Data Scientist at KenSci, an Artificial Intelligence in healthcare focused startup in Seattle. He has had academic appointments at University of Washington, Center for Cognitive Science at University of Minnesota, Minnesota Population Center, and the Indian Institute of Technology at Kanpur.  Muhammad Aurangzeb has published over 50 research papers in the field of machine learning and artificial intelligence. His current research is focused on the responsible AI in healthcare via explainable, fair, unbiased, robust systems. Prof. Ahmad regularly teaches classes on AI, Data Mining, Machine Learning at University of Washington Tacoma, University of Washington Bothell and UW Professional & Continuing Education. He also been teaching these topics at several other Universities in the US and abroad. Dr. Ahmad delivered relevant tutorials at ACM Seminar 2019, AMIA 2019, BCB 2018 and ICHI 2018. http://www.aurumahmad.com.


Ankur Teredesai

Professor, Department of Computer Science & Systems, University of Washington Tacoma, USA, ankurt@uw.edu


Bio: Ankur Teredesai

Ankur M. Teredesai is a Professor of Computer Science & Systems at University of Washington Tacoma, and the founding director of the Center for Data Science. His research interests focus on data science applications for healthcare, and its societal impact. Apart from his academic appointments at RIT, and the University of Washington Tacoma, Teredesai has significant industry experience, having held various positions at C-DAC Pune, Microsoft, IBM T.J. Watson Labs, and a variety of technology startups. Prof. Teredesai has published over 75 papers on machine learning in leading venues, has managed large teams of data scientists and engineers, and deployed numerous machine learning applications across various industries: from web advertising, social recommendations, to handwriting recognition. Since 2009, for over a decade, his research focus has been making AI assistive for healthcare. His research contributions have led to advancing our understanding of risk, and utilization prediction for chronic conditions, such as diabetes and heart failure.  Prof. Teredesai regularly teaches classes on AI, Data Mining, Machine Learning, Informatics, Knowledge Discovery at University of Washington Tacoma. He also been teaching these topics at several other Universities and industry in the. Dr. Teredesai delivered relevant tutorials at ACM Seminar 2019, AMIA 2019, BCB 2018 and ICHI 2018. http://faculty.washington.edu/ankurt/. 


 Relevant related work

[1]    Cashman D., Humayoun SR., SR. Heimerl F, Park K.  et al. Visual Analytics for Automated Model Discovery. arXiv preprint arXiv:1809.10782. 2018

[2]    Endert A., Ribarsky W., Turkay C., Wong BW, .et al, The state of the art in integrating machine learning into visual analytics. In: Computer Graphics Forum 2017, Vol. 36, No. 8, 458-486.

[3]   Guo Y., Liu Y., Oerlemans A., Lao S., Wu S.., S. Lew S.,Deep learning for visual understanding: A review. Neurocomputing. 2016, 187:27-48.  

[4]    Kovalerchuk, B. Explainable Machine Learning and Visual Knowledge Discovery, Handbook of Machine Learning for Data Science, 3rd ed. Springer, 2020 (in print)

[5].   Kovalerchuk B., Intelligible Machine Learning and Knowledge Discovery Boosted by Visual Means, WSDM '20: Proceedings of the 13th ACM International Conference on Web Search and Data Mining, January 2020, pp. 881–883, https://doi.org/10.1145/3336191.3371872.

[6]   Kovalerchuk B. Enhancement of Cross Validation Using Hybrid Visual and Analytical Means with Shannon Function, In: Beyond Traditional Probabilistic Data Processing Techniques: Interval, Fuzzy etc. Methods and Their Applications, Springer, 2020. 30p, DOI: 10.1007/978-3-030-31041-7

[7]   Kovalerchuk B., Grishin V. Adjustable General Line Coordinates for Visual Knowledge Discovery in n-D data, Information Visualization, 18(1),2019, 3-32.

[8]   Kovalerchuk B., Visual Knowledge Discovery and Machine Learning, Springer, 2018.

[9]   Kovalerchuk B., Gharawi A., Decreasing Occlusion and Increasing Explanation in Interactive Visual Knowledge Discovery, In: S. Yamamoto and H. Mori (Eds.) Human Interface and the Management of Information. Interaction, Visualization, and Analytics, LNCS 10904, Springer, pp. 505–526, 2018.

[10] Kovalerchuk B., Neuhaus N., Toward Efficient Automation of Interpretable Machine Learning. In: 2018 IEEE International Conference on Big Data, pp. 4933 4940, Seattle, Dec. 10-13, 2018 IEEE.

[11]  Kovalerchuk B., Schwing J. (eds), Visual and Spatial Analysis: Advances in Visual Data Mining, Reasoning, and Problem Solving, Springer, 2005,

[12]  Kovalerchuk B., Agarwal B., Solving Non-image Learning Problems by Collocated Visualization and Deep Learning, CWU technical report, 2019.

[13] Koutra D., Di Jin, Y. Ning, C. Faloutsos. Perseus: An Interactive Large-Scale Graph Mining and Visualization Tool. Proc..of the VLDB

[14]. Neuhaus N, Kovalerchuk B., Interpretable Machine Learning with Boosting by Boolean Algorithm, IEEE Joint 2019 8th Intern. Conf. on Informatics, Electronics & Vision, Spokane, WA, 2019, 307-311.

[15]  Sacha D., Kraus M., Keim DA., Chen M.. VIS4ML: An Ontology for Visual Analytics Assisted Machine Learning. IEEE TVCG 2019 Jan;25(1):385-95.

[16] C. Seifert, A. Aamir, A. Balagopalan, D. Jain, A. Sharma, S. Grottel, S. Gumhold, Visualizations of deep neural networks in computer vision: A survey. In: Transparent Data Mining for Big and Small Data 2017, 123-144. Springer,

[17] Turkay C., Laramee R, Holzinger A., On the challenges and opportunities in visualization for machine learning and knowledge extraction: A research agenda. In: International Cross-Domain Conference for Machine Learning and Knowledge Extraction 2017, 191-198, Springer.

[18] Samek W., Montavon G., Vedaldi A., et al., (Eds), Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Springer, 2019

[19] Sietzen S., Waldner M., Interactive Feature Visualization in the Browser, 2nd Workshop on Visualization for AI Explainability, IEEE VIS., 2019  Canadahttp://visxai.stefansietzen.at/

[20] Parra D., Valdivieso H., Carvallo A., Rada G., Verbert K., Schreck T., Analyzing the Design Space for Visualizing Neural Attention in Text Classification, 2nd Workshop on Visualization for AI Explainability, IEEE VIS., 2019, https://observablehq.com/@clpuc/analyzing-the-design-space-for-visualizing-neural-attenti

[21] Vig J., Deconstructing BERT: Visualizing the Inner Workings of Attention, Analyzing the Design Space for Visualizing Neural Attention in Text Classification, 2nd Workshop on Visualization for AI Explainability, IEEE VIS., 2019, https://towardsdatascience.com/deconstructing-bert-part-2-visualizing-the-inner-workings-of-attention-60a16d86b5c1

[22] Burkhardt D., Gigante S., Krishnaswamy S., Selecting the right tool for the job: a comparison of visualization algorithms, 2nd Workshop on Visualization for AI Explainability, IEEE VIS., 2019, https://hub.gke.mybinder.org/user/krishnaswamylab-ation_selection-47z17vuu/notebooks/Selecting_the_right_tool_for the_


[23] Britton M., Visual Explanations for Gradient Boosting Models,   2nd Workshop on Visualization for AI Explainability, IEEE VIS., 2019, https://github.com/MattJBritton/ForestfortheTrees.

[24] Lundberg S., A unified approach to explain the output of any machine learning model, https://github.com/slundberg/shap

[25] He S, Borji A, Mi Y, Pugeault N. What catches the eye? Visualizing and understanding deep saliency models. arXiv preprint arXiv:1803.05753. 2018

[26] Cornia M, Baraldi L, Serra G, Cucchiara R. SAM: Pushing the Limits of Saliency Prediction Models. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops 2018, 890-1892.

[27] Li Y, Mou X. Saliency detection based on structural dissimilarity induced by image quality assessment model. Journal of Electronic Imaging. 2019 Apr;28(2):023025. https://arxiv.org/pdf/1905.10150

[28] Vergari A, Di Mauro N, Esposito F. Visualizing and understanding sum-product networks. Machine Learning. 2019 Apr 15;108(4):551-73. https://arxiv.org/pdf/1608.08266

[29] Schlegel U, Arnout H, El-Assady M, Oelke D, Keim DA. Towards a rigorous evaluation of XAI Methods On Time Series., 2019 ICCV Workshop on Interpreting and Explaining Visual Artificial Intelligence Models, https://arxiv.org/abs/1909.07082

[30] Schlegel U, Arnout H, El-Assady M, Oelke D, Keim DA. Towards A Rigorous Evaluation Of XAI Methods On Time Series. 2019, arXiv:1909.07082.  

[27] Gunning D. Explainable Artificial Intelligence (XAI). DARPA, 2017, https://www.darpa.mil/attachments/XAIProgramUpdate.pdf

[28] Teredesai, A., Eckert, C., Ahmad, MA., Kumar, V. Explainable Machine Learning Models for Healthcare AI September 26, 2018 ACM Seminar

[29] Ahmad, MA., Eckert, C., Teredesai, A., Kumar, V., Explainable Models for Healthcare AI KDD London, United Kingdom August 19-23, 2018

[30] Ahmad, MA., Eckert, C., Teredesai, A., Kumar, V., Machine Learning in Healthcare International Conference on Bioinformatics, Computational Biology, and Health Informatics, BCB 2018, August 29 - September 01, 2018 Washington, DC, USA.

[31] Ahmad, MA., Eckert, C., Teredesai, A., Kumar, V., Interpretable Machine Learning in Healthcare 2018 IEEE International Conference on Healthcare Informatics 4-7 June, New York, NY, USA.

[32] Ahmad, MA., Eckert, C., Teredesai, A., The Challenge of Imputation in Explainable Artificial Intelligence Models. Macau, China AISafety Workshop at IJCAI 2019

[33] Lu, J. and Ester, M., 2019. An Active Approach for Model Interpretation. arXiv preprint arXiv:1910.12207.

[34] Lakkaraju, H. and Bastani, O., 2019. " How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations. arXiv preprint arXiv:1911.06473

[35] Rudin, C. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead, ACM KDD 2019, https://arxiv.org/abs/1811.10154