Explainable AI in Healthcare applications

Author

K. Santosh Babu, Dr. Narendra

Abstract

 AI algorithms now outperform clinicians on a handful of diagnostic tasks, but most top-rated algorithms are 'black boxes' and cannot be probed for explanation by clinicians, patients and regulators who need, and are increasingly demanding, them. Explainable AI (XAI) is an effort to close this explanatory gap in healthcare and this paper is the first structured narrative review of healthcare XAI so far: (i) classifies XAI techniques (LIME, SHAP, saliency and Grad-CAM, integrated gradients, attention, counterfactuals, and inherently interpretable models), (ii) maps techniques to clinical tasks (medical imaging, risk prediction, intensive care monitoring, genomics, drug discovery), (iii) summarizes criteria to evaluate explanations, (iv) identifies major issues faced in practice (fidelity and faithfulness, shortcut learning, bias, regulation) and (v) provides recommendations for XAI researchers, doctors, healthcare managers and regulators, with special reference to India. The review finds that explanation techniques are most developed for tabular patient data and medical images; unfaithfulness in post-hoc explanations could lead to false reassurance; and clinical validation of explanations with real users is infrequent. As a conclusion, XAI should be viewed as one element in a broader assurance framework that should encompass validation, continuous monitoring and governance.

Keywords

Explainable AI, Interpretable Machine Learning, Healthcare, Medical Imaging, Clinical Decision Support, SHAP, LIME, Grad-CAM, Trust, India

Full Text:

Download Paper PDF

References

Foundational methods
1.1.. Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). "Why should I trust you?": Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1135-1144.
2.Lundberg, S. M., & Lee, S.-I. (2017). A unified approach to interpreting model predictions. Advances in Neural Information Processing Systems, 30.
3.Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual explanations from deep networks via gradient-based localization. Proceedings of the IEEE International Conference on Computer Vision.
4.Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribution for deep networks. Proceedings of the 34th International Conference on Machine Learning, 3319-3328.
5.Simonyan, K., Vedaldi, A., & Zisserman, A. (2014). Deep inside convolutional networks: Visualising image classification models and saliency maps. Workshop at the International Conference on Learning Representations.
6.Doshi-Velez, F., & Kim, B. (2017). Towards a rigorous science of interpretable machine learning. ArXiv:1702.08608.
Healthcare XAI studies, reviews and critiques
7.Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015). Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. Proceedings of the 21st ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1721-1730.
8.Lundberg, S. M., Nair, B., Vavilala, M. S., et al. (2018). Explainable machine-learning predictions for the prevention of hypoxaemia during surgery. Nature Biomedical Engineering, 2, 749-760.
9.Lundberg, S. M., Erion, G., Chen, H., et al. (2020). From local explanations to global understanding with explainable AI for trees. Nature Machine Intelligence, 2, 56-67.
10.10. Rudin, C. (2019). Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1, 206-215.
11.11. Ghassemi, M., Oakden-Rayner, L., & Beam, A. L. (2021). The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health, 3(11), e745-e750.
12.DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021). AI for radiographic COVID-19 detection takes a shortcut. Nature Machine Intelligence, 3, 610–619.
13.Holzinger, A., Langs, G., Denk, H., Zatloukal, K., & Müller, H. (2019). Causability and explainability of artificial intelligence in medicine. WIREs Data Mining and Knowledge Discovery, 9(4), e1312.
14.Amann, J., Blasimme, A., Vayena, E., Frey, D., & Madai, V. I. (2020). Explainability for artificial intelligence in healthcare: A multidisciplinary perspective. BMC Medical Informatics and Decision Making, 20, 310.
15.Tjoa, E., & Guan, C. (2021). A survey on explainable artificial intelligence (XAI): Toward medical XAI. IEEE Transactions on Neural Networks and Learning Systems, 32(11), 4793-4813.
16.Adadi, A., & Berrada, M. (2018). Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access, 6, 52138-52160.
17.Arrieta, A. B., et al. (2020). Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion, 58, 82-115.
Clinical AI context
18.18. Gulshan, V., et al. (2016). Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA, 316(22), 2402-2410.
19.19. A, et al. (2017). A dermatologist-level classification of skin cancer with deep neural networks Nature 542 115-118.
20.20. E. J. Topol, Topol, E. J. (2019). High-performance medicine: The convergence of human and artificial intelligence. Nature Medicine, 25, 44-56.
21.21. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447-453.
22.Policy and regulatory documents
23.22. Ethical guidelines for application of artificial intelligence in biomedical research and healthcare (2023).
24.23. Government of India. Digital Personal Data Protection Act, 2023.
25.24. European Union. Regulation (EU) 2016/679 (General Data Protection Regulation) Regulation (EU) 2024/1689 (Artificial Intelligence Act).

Share your valuable work from Social Media Buttons