Transparency tinkering: Working with opacities in the development of medical artificial intelligence

Abstract

Artificial Intelligence (AI) technologies are increasingly researched and applied in medicine and healthcare. The high stakes of these applications have raised a particular demand for them to be, to some degree, transparent. Yet, how AI experts consider transparency and turn it into practice in the development of medical AI remains underexplored. As the notion of AI technologies as black boxes persists, more empirical and conceptual understanding is needed of how AI experts consider and practically navigate this opacity and what work goes into trying to make AI transparent, explainable, knowable and trustworthy. This article aims to increase the knowledge of how AI experts working on medical AI deal with opacity and make sense of their models. Interviews and observations with AI experts show how transparency is tinkered with across different approaches and modalities; in the trying out of models, the use of visual clues, and scoring of importance and uncertainty. Transparency practices are deployed for the experts’ own sensemaking of models and medical conditions, as well as for other actors’ understanding. This study shows that explainable AI (XAI) is met with both hope and scepticism as a solution to AI opacity, how opacity-mitigating practices are continuously adapted, and how sensemaking of medical AI is a contingent, relational and situated process. By introducing a conceptual framework of transparency tinkering , this article suggests entry points of study to make these complexities visible, move away from the algorithmic drama and gain insights into the tinkering through which transparency is enacted in practice.