Amann et al.

BMC Med Inform Decis Mak (2020) 20:310

https://doi.org/10.1186/s12911-020-01332-6

RESEARCH ARTICLE Open Access

Explainability for artificial intelligence

in healthcare: a multidisciplinary perspective
Julia Amann1* , Alessandro Blasimme1, Effy Vayena1, Dietmar Frey2 and Vince I. Madai2,3 on behalf of the
Precise4Q consortium

Abstract
Background: Explainability is one of the most heavily debated topics when it comes to the application of artificial
intelligence (AI) in healthcare. Even though AI-driven systems have been shown to outperform humans in certain
analytical tasks, the lack of explainability continues to spark criticism. Yet, explainability is not a purely technological
issue, instead it invokes a host of medical, legal, ethical, and societal questions that require thorough exploration. This
paper provides a comprehensive assessment of the role of explainability in medical AI and makes an ethical evalua-
tion of what explainability means for the adoption of AI-driven tools into clinical practice.
Methods: Taking AI-based clinical decision support systems as a case in point, we adopted a multidisciplinary
approach to analyze the relevance of explainability for medical AI from the technological, legal, medical, and patient
perspectives. Drawing on the findings of this conceptual analysis, we then conducted an ethical assessment using the
“Principles of Biomedical Ethics” by Beauchamp and Childress (autonomy, beneficence, nonmaleficence, and justice)
as an analytical framework to determine the need for explainability in medical AI.
Results: Each of the domains highlights a different set of core considerations and values that are relevant for
understanding the role of explainability in clinical practice. From the technological point of view, explainability has to
be considered both in terms how it can be achieved and what is beneficial from a development perspective. When
looking at the legal perspective we identified informed consent, certification and approval as medical devices, and
liability as core touchpoints for explainability. Both the medical and patient perspectives emphasize the importance of
considering the interplay between human actors and medical AI. We conclude that omitting explainability in clinical
decision support systems poses a threat to core ethical values in medicine and may have detrimental consequences
for individual and public health.
Conclusions: To ensure that medical AI lives up to its promises, there is a need to sensitize developers, healthcare
professionals, and legislators to the challenges and limitations of opaque algorithms in medical AI and to foster multi-
disciplinary collaboration moving forward.
Keywords: Artificial intelligence, Machine learning, Explainability, Interpretability, Clinical decision support

Background
All over the world, healthcare costs are skyrocketing.
Increasing life expectancy, soaring rates of chronic dis-
eases, and the continuous development of costly new
therapies contribute to this trend. Thus, it comes as
*Correspondence: [email protected]
1
no surprise that scholars predict a grim future for the
Health Ethics and Policy Lab, Department of Health Sciences
and Technology, ETH Zurich, Hottingerstrasse 10, 8092 Zurich, Switzerland
sustainability of healthcare systems throughout the
Full list of author information is available at the end of the article world. Artificial intelligence (AI) promises to alleviate

© The Author(s) 2020. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which
permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the
original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or
other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line
to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory
regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this
licence, visit http://creat​iveco​mmons​.org/licen​ses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creat​iveco​
mmons​.org/publi​cdoma​in/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
Amann et al. BMC Med Inform Decis Mak (2020) 20:310 Page 2 of 9

the impact of these developments by improving health- evaluation of what explainability means for the adoption
care and making it more cost-effective [1]. In clinical of AI-driven tools into clinical practice.
practice, AI often comes in the form of clinical deci-
sion support systems (CDSS), assisting clinicians in Methods
diagnosis of disease and treatment decisions. Where Taking AI-based CDSS as a case in point, we discuss
conventional CDSS match the characteristics of indi- the relevance of explainability for medical AI from the
vidual patients to an existing knowledge base, AI-based technological, legal, medical, and patient perspective.
CDSSs apply AI models trained on data from patients To this end, we performed a conceptual analysis of the
matching the use-case at hand. Yet, despite its unde- pertinent literature on explainable AI in these domains.
niable potential, AI is not a universal solution. As his- In our analysis, we aimed to identify aspects relevant to
tory has shown, technological progress always goes determining the necessity and role of explainability for
hand in hand with novel questions and significant chal- each domain, respectively. Drawing on these different
lenges. Some of these challenges are tied to the techni- perspectives, we then conclude by distilling the ethical
cal properties of AI, others relate to the legal, medical, implications of explainability for the future use of AI in
and patient perspectives, making it necessary to adopt the healthcare setting. We do the latter by examining
a multidisciplinary perspective. explainability against the four ethical principles of auton-
In this paper, we take such a multidisciplinary view on a omy, beneficence, non-maleficence, and justice.
major medical AI challenge: explainability. In its essence,
explainability can be understood as a characteristic of an Results
AI-driven system allowing a person to reconstruct why The technological perspective
a certain AI came up with the presented predictions. From the technological perspective, we will explore two
An important point to note here is that explainability issues. First, what explainability methods are and, second,
has many facets and, unfortunately, the terminology of where they are applied in medical AI development.
explainability is not well defined. Other terms such as With regards to methodology, explainability can either
interpretability and/or transparency are often used syn- be an inherent characteristic of an algorithm or can be
onymously [2, 3]. We thus simply refer to explainability approximated by other methods [2]. The latter is highly
or explainable AI throughout the manuscript and add the important for methods that have until recently been
necessary context for understanding. labeled as “black-box models” such as artificial neural
Explainability is a heavily debated topic with far- network (ANN) models. To explain their predictions,
reaching implications that extend beyond the technical however, numerous methods exist today [6]. Importantly,
properties of AI. Even though research indicates that AI however, inherent explainability will, in general, be more
algorithms can outperform humans in certain analyti- accurate than methods that only approximate explain-
cal tasks (e.g. pattern recognition in imaging), the lack of ability [2]. This can be attributed to the complex char-
explainability has been criticized in the medical domain acteristics of many modern machine learning methods.
[4]. Legal and ethical uncertainties surrounding this In ANNs, for example, the inner workings of sometimes
issue may impede progress and prevent novel technolo- millions of weights between artificial neurons need to be
gies from fulfilling their potential to improve patient and interpreted in a way that humans can understand. Thus,
population health. Yet, without thorough consideration contrasting methods with inherent explainability have a
of the role of explainability in medical AI, these technolo- crucial advantage. However, these methods are usually
gies may forgo core ethical and professional principles, also traditional methods, such as linear or logistic regres-
disregard regulatory issues, and cause considerable harm sion. For many use cases, there is an inferiority of these
[5]. traditional methods in performance compared to mod-
To contribute to the discourse on explainable AI in ern state-of-the-art methods such as ANNs [7]. Thus,
medicine, this paper seeks to draw attention to the inter- there is a trade-off between performance and explainabil-
disciplinary nature of explainability and its implica- ity, and this trade-off is a big challenge for the developers
tions for the future of healthcare. In particular, our work of clinical decision support systems. It should be noted
focuses on the relevance of explainability for CDSS. The that some assume that this trade-off does not exist in
originality of our work lies in the fact that we look at reality, but it is a mere artifact of suboptimal modelling
explainability from multiple perspectives that are often approaches, as pointed out by Rudin et al. [2]. While the
regarded as independent and separable from each other. work of Rudin et al. is important to raise attention to the
This paper has two central aims: (1) to provide a compre- shortcomings of approximating explainability methods, it
hensive assessment of the role of explainability in CDSS is likely that some approximating methods, in contrast to
for use in clinical practice and; (2) to make an ethical the notion of [2], have value given the complex nature of
Amann et al. BMC Med Inform Decis Mak (2020) 20:310 Page 3 of 9

explaining machine learning models. Additionally, while The legal perspective

we can make the qualitative assessment that inherent From the legal perspective, the question arises if and, if
explainability is likely better than approximated explaina- yes, to what extent explainability in AI is legally required.
bility, there exist only exploratory initial attempts to rank Taking the cue from other fields such as public adminis-
explainability methods quantitatively [8]. Notwithstand- tration, transparency and traceability have to meet even
ing, for many applications—and generally in AI product higher standards when it comes to health care and the
development—there is a de facto preference for modern individual patient [12]. As shown above, artificial intel-
algorithms such as ANNs. Additionally, it cannot be ruled ligence approaches such as machine learning and deep
out that for some applications such modern methods do learning have the potential to significantly advance the
exhibit actual higher performance. This necessitates to quality of health care. Identifying patterns in diagnos-
critically assess explainability methods further, both with tics, anomaly detection and, in the end, providing deci-
regards to technical development, e.g. for methods rank- sion support are already changing standards of care and
ing and optimization of methods for certain inputs, and clinical practice. To fully exploit these opportunities for
with regards to the role of explainability from a multiple improving patients’ outcomes and saving lives by advanc-
stakeholder view as done in the current work. ing detection, prevention, and treatment of diseases,
From the development point-of-view, explainability the sensitive issues of data privacy and security, patient
will regularly be helpful for developers to sanity check consent, and autonomy have to be fully considered. This
their AI models beyond mere performance. For exam- means that from a legal perspective, data—its acquisi-
ple, it is highly beneficial to rule out that the prediction tion, storage, transfer, processing, and analysis—will
performance is based on meta-data rather than the data have to comply with all laws, regulations and further
itself. A famous non-medical example was the classifica- legal requirements. In addition, the law and its inter-
tion task to discern between huskies and wolves, where pretation and implementation have to constantly adapt
the prediction was solely driven by the identification of a to the evolving state-of-the-art in technology [13]. Even
snowy background rather than real differences between when fulfilling all of these rather obvious requirements
huskies and wolves [9]. This phenomenon is also called the question remains if the application of AI-driven solu-
a “Clever Hans” phenomenon [10]. Clever Hans phe- tions and tools demand explainability. In other words, do
nomena are also found in medicine. An example is the doctors and patients need information not only about the
model developed by researchers from Mount Sinai hos- results that are provided but also about the characteris-
pital which performed very well in distinguishing high- tics and features these results are based upon, and the
risk patients from non-high-risk patients based on x-ray respective underlying assumptions. And, might the nec-
imaging. However, when the tool was applied outside of essary inclusion of other stakeholders require an under-
Mount Sinai, the performance plummeted. As it turned standing and explainability of algorithms and models.
out the AI model did not learn clinically relevant infor- From a Western legal point-of-view, we identified three
mation from the images. In analogy to the snowy back- core fields for explainability: (1) Informed consent, (2)
ground in the example introduced above, the prediction Certification and approval as medical devices (acc. to
was based on hardware related meta-data tied to the Food and Drug Administration/FDA and Medical Device
specific x-ray machine that was used to image the high- Regulation/MDR) and (3) Liability.
risk ICU patients exclusively at Mount Sinai [11]. Thus, Personal health data may be only processed by law
the system was able to distinguish only which machine after the individual consents to its use. In the absence
was used for imaging and not the risk of the patients. of general laws facilitating the use of personal data and
Explainability methods allow developers to identify these information, this informed consent is the standard for
types of errors before AI tools go into clinical validation today’s use of patient data in AI applications [14]. This
and the certification process, as the Clever Hans predic- is particularly challenging since the consent has to be
tors (snowy background, hardware information) would specified in advance, i.e. the purpose of the given project
be identified as prediction relevant by the explainability and its aims have to be outlined. The natural advantage
methods rather than meaningful features from a domain of AI is that it does not necessitate pre-selection of fea-
perspective. This saves time and development costs. It tures and can identify novel patterns or find new bio-
should be noted that explainability methods aimed at markers. If restricted to specific purposes—as required
developers to provide insight into their models have for informed consent—this unique advantage might not
different prerequisites than systems aimed at techno- be fully exploitable. For obtaining informed consent for
logically unsavvy end-users such as clinical doctors and diagnostic procedures or interventions the law requires
patients. For developers, these methods can be more individual and comprehensive information about and
complex in their approach and visualization. understanding of these processes. In the case of AI-based
Amann et al. BMC Med Inform Decis Mak (2020) 20:310 Page 4 of 9

decision support, the underlying processes and algo- Data Protection Regulation (GDPR) in the European
rithms have therefore to be explained to the individual Union requires the use of explainable AI in tools work-
patient. Just like in the case of obtaining consent for ing with patient data [17, 18]. Also here, it cannot be
undergoing an MR imaging procedure, the patient might ruled out that the currently ambiguous phrasings will be
not necessarily need to know every detail but certainly amended in favor of one that promotes explainability in
has to be informed about core principles, and especially the future.
the risks. Yet, contrary to an MR imaging procedure, Finally, the question arises, to what extent the patient
physicians are unable to provide this type of information has to be made aware that treatment decisions such as
for an opaque CDSS. What physicians should at least be those derived by a clinical decision support system might
able to provide are explanations around two principles: rely on AI and the legal and litigation question if the phy-
(1) the agent view of AI, i.e. what it takes as input; what it sician adhered to the recommendation or overruled the
does with the environment; and what it produces as out- machine. For the US, as Cohen laid out, there is currently
put, and (2) explaining the training of the mapping which no clear-cut answer to what extent the integration of ML/
produces the output by letting it learn from examples— DL into clinical decision-making has to be disclosed with
which encompasses unsupervised, supervised, and rein- regard to liability [14]. Hacker et al. argue that legally it
forcement learning. Yet, it is important to note that for is likely that explainability will be a prerequisite from a
AI-based CDSS the extent of the information is a priori contract and tort law perspective where doctors may
highly difficult to define, has to be adjusted to the respec- have to use a certain tool to avoid the threat of a medical
tive use case, and will certainly need clarification from malpractice lawsuit [17]. The final answer to this lies with
the legislative bodies. For this, a framework for defining the courts, however, and will be given rather sooner than
the "right" level of explainability, as Beaudouin et al. put it later as an increasing number of AI-based systems will be
[15], should be developed. Clearly, this also raises impor- in use.
tant questions about the role and tasks of physicians, Taken together, the legal implications of introducing AI
underscoring the need for tailored training and profes- technologies into health care are significant and the con-
sional development in the area of medical AI. stant conflict between innovation and regulation needs
With regard to certification and approval as medical careful orchestration. Potentially life-saving just as new
devices, the respective bodies have been slow to intro- cancer medication or antibiotics, AI-based decision sup-
duce requirements for explainable AI and its implica- port needs guidelines and legal crash barriers to avoid
tions on the development and marketing of products. In existential infringement on patients’ rights and auton-
a recent discussion paper, the FDA facilitates in its total omy. Explainability is an essential quality in this context
product lifecycle approach (TPLC) the constant develop- and we would argue that performance is only sufficient
ment and improvement of AI-based medical products. in cases, where it is not possible to provide explainability.
Explainability is not mentioned but an “Appropriate level Overall, there is a strong need for explainability in legal
of transparency (clarity) of the output and the algorithm aspects and opening the black box has become essential
aimed at users” is required [16]. This is mainly aimed at and will prove to be the watershed moment for the appli-
the functions of the software and its modifications over cation of AI in medicine.
time. The MDR does not specifically regulate the need
for explainability with regard to medical devices that use The medical perspective
artificial intelligence and machine learning in particu- From the medical perspective, the first consideration is
lar. However, also here, the need for accountability and what distinguishes AI-based clinical decision support
transparency are set and the evolution of xAI might lead from established diagnostic tools, such as advanced labo-
the legislative and the notified bodies to change the regu- ratory testing for example? Especially as they do exhibit
lations and their interpretation accordingly. considerable overlaps: Both can provide results used for
In conclusion, both FDA and MDR are currently rather CDSSs, for both performance is a key issue, and their
vaguely requiring explainability, i.e. information for results are documentable. We also understand the inner
traceability, transparency, and explainability of develop- working of laboratory testing, as it is often the case with
ment of ML/DL models that inform medical treatment. other diagnostic tests, such as imaging, so they would not
Most certainly, these requirements will be defined more be regarded as black box methods. On the other hand,
precisely in the future mandating producers of AI-based for these methods we cannot explain the result of any
medical devices/software to provide insight into the individual test. This makes it evident that from a medical
training and testing of the models, the data, and the over- perspective, we need to distinguish two levels of explain-
all development processes. We would also like to men- ability. First level explainability allows us to under-
tion that there is a current debate on whether the General stand how the system arrives at conclusions in general.
Amann et al. BMC Med Inform Decis Mak (2020) 20:310 Page 5 of 9

In analogy to laboratory testing, where we know which [1]. While this ensures that AI bias can be reduced to a
biological and biochemical reactions lead to the results, minimum, it will still be almost impossible to generate
we can provide feature importance rankings that explain AI tools without any trace of bias. If bias is present, then
which inputs are important for the AI-based CDSSs. there will be prediction errors in patients not represent-
Second level explainability allows us to identify which ing the training sample. Taken together, both random
features were important for an individual prediction. and systematic sum up to the total number of errors that
Individual predictions can be safe-checked for patterns physicians and patients will encounter in the clinical set-
that might indicate a false prediction, e.g. in case of unu- ting, even when a fully validated high-performing AI sys-
sual feature distribution in an out-of-sample case. This tem is used.
second level explainability will regularly be available for This is why, from a medical point-of-view, not only
AI-based CDSS but not for other diagnostic tests. This clinical validation but also explainability plays an instru-
also has implications for the presentation of explainabil- mental role in the clinical setting. Explainability enables
ity results to doctors (and patients). Depending on the the resolution of disagreement between an AI system
clinical use case and the risk attributed to that particu- and human experts, no matter on whose side the error
lar use case, first level explanations might be sufficient, in judgment is situated. It should be noted that this will
whereas other use cases will regularly require second succeed mostly in cases of systematic error, of AI bias,
level explanations to safe-guard patients. rather than in cases of random error. Random errors are
To date, clinical validation is currently the first widely much harder to identify and will likely go unnoticed in
discussed requirement for a medical AI system. Explain- case of agreement between the tool and the physician
ability is often only considered on second thought. The or will lead to situations of disagreement between the
reason for this seems obvious: Medical AI systems and tool and the physician. This situation is discussed in the
especially CDSSs, whether AI-powered or not, have to ethical considerations section. Explainability results are
undergo a rigorous validation process to meet regulatory usually represented visually or through natural language
standards and achieve medical certification [1]. Once explanations. Both show the clinicians how different fac-
this process is completed successfully, there is proof tors contributed to the final recommendation. In other
that the system can perform in the highly heterogene- words, explainability can assist clinicians in evaluating
ous real-world clinical setting. Here, it is important to the recommendations provided by a system based on
understand how clinical validation is measured. A com- their experience and clinical judgment. This allows them
mon performance indicator is prediction performance, to make an informed decision whether or not to rely on
often referred to as prediction accuracy. Different meas- the system’s recommendations and can, consequently,
ures exist for prediction accuracy, tailored to certain use- strengthen their trust in the system. Particularly in
cases, but their common characteristic is that they reflect cases where the CDSS produces recommendations that
the prediction quality and thus general clinical usefulness are strongly out of line with a clinician’s expectations,
of a model. Thus, one of the main goals of model devel- explainability allows verification whether the param-
opment is to increase prediction performance and pro- eters taken into account by the system make sense from
vide low error rates. And, indeed, AI-powered systems a clinical point-of-view. By laying open the inner work-
have been shown to produce overall lower error rates ings of the CDSS, explainability can, thus, assist clinicians
than traditional methods [19–21]. in identifying false positives and false negatives more
Despite all efforts, however, AI systems cannot provide easily. As clinicians identify instances in which the sys-
perfect accuracy owing to different sources of error. For tem performs poorly, they can report these cases back
one, because of naturally imperfect datasets in medi- to developers to foster quality assurance and product
cine (e.g. due to noise or recording errors), it is basically improvement. Given these considerations, explainability
impossible to develop a model without any errors. These may be a key driver for the uptake of AI-driven CDSS in
errors are random errors. Thus, there will always be cer- clinical practice, as trust in these systems is not yet estab-
tain cases of false positive and false negative predictions. lished [22, 23]. Here, it is important to note that any use
For another, a particularly important source of error is of AI-based CDSS may influence a physician in reaching
AI bias. AI bias leads to systematic errors, a systematic a decision. It will, therefore, be of critical importance to
deviation from the expected prediction behavior of the establish transparent documentation on how recommen-
AI tool. Ideally, the data used for training fully repre- dations were derived.
sent the population in which the AI tool is later applied.
A major goal of AI in healthcare product development is The patient perspective
to approximate this ideal state via thorough clinical vali- Looking at the issue of explainability from the patient
dation and development on heterogeneous data sources perspective raises the question of whether the use of
Amann et al. BMC Med Inform Decis Mak (2020) 20:310 Page 6 of 9

AI-powered decision aids is compatible with the inher- more accurate risk perceptions [34, 35]. This may, in turn,
ent values of patient-centered care. Patient-centered boost patients’ motivation to engage in shared decision-
care aims to be responsive to and respectful of individ- making and to act upon risk-relevant information [35].
ual patients’ values and needs [24]. It considers patients
as active partners in the care process, emphasizing their Ethical implications
right to choice and control over medical decisions. A key With the increasing penetration of AI-powered systems
component of patient-centered care is shared decision- in healthcare, there is a necessity to explore the ethi-
making aimed at identifying the treatment best suited to cal issues accompanying this imminent paradigm shift.
the individual patients’ situation [25, 26]. It involves an A commonly applied and well-fitting ethical frame-
open conversation between the patient and the clinician, work when assessing biomedical ethical challenges are
where the clinician informs the patient about the poten- the “Principles of Biomedical Ethics” by Beauchamp
tial risks and benefits of available courses of action and and Childress [36, 37] introducing four key principles:
the patient discusses their values and priorities [27, 28]. autonomy, beneficence, nonmaleficence, and justice [36].
Several evidence-based tools have been developed While principlism is not the only available bioethical
to facilitate shared decision-making, among them, so- framework, it is a very useful basic practical framework
called conversation aids [29]. Unlike patient decision aids with high acceptance both in research and medical set-
(which are used by the patient in preparation prior to the tings [36–38]. Thus, in the following, we assess explaina-
clinical encounter), conversation aids are designed for bility with regards to the aforementioned four principles.
use within the clinical encounter to guide the patient and Concerning autonomy, explainability has implica-
clinician through the shared decision-making process tions for patients and physicians alike [31]. One of the
[28, 30]. They incorporate established medical facts about major safeguards of patients’ autonomy is represented by
their conditions and, by synthesizing available informa- informed consent, that is an autonomous, generally writ-
tion, they can help patients to understand their individual ten authorization with which the patient grants a doctor
risks and outcomes, to explore the available options, and his or her permission to perform a given medical act [39].
to determine which course of action best fits their goals Proper informed consent is premised upon exhaustive
and priorities [30–32]. So, what if individual risk was not and understandable information regarding the nature
calculated using established risk prediction models but and risks of a medical procedure, and lack of undue inter-
instead relied on a validated, yet not explainable, data- ference with the patient’s voluntary decision to undergo
driven approach? Would it make a difference from the the procedure. At the moment, an ethical consensus has
patient’s perspective? Seeking to address these questions, not yet emerged as to whether disclosing the use of an
it was recently argued that so-called ‘black-box medicine’ opaque medical AI algorithm should be a mandatory
conflicts with core ideals of patient-centered medicine requirement of informed consent. A failure to disclose
[33]. Since clinicians are no longer able to fully compre- the use of an opaque AI system may undermine patients’
hend the inner workings and calculations of the decision autonomy and negatively impact the doctor-patient rela-
aid they are not able to explain to the patient how certain tionship, jeopardizing patients’ trust, and might violate
outcomes or recommendations were derived [33]. the compliance with clinical recommendations. If the
Explainability can address this issue by providing cli- patient were to find out in hindsight that a clinician’s rec-
nicians and patients with a personalized conversation ommendation was derived from an opaque AI system,
aid that is based on the patient’s individual characteris- this may lead the patient to not only challenge the recom-
tics and risk factors. By simulating the impact of differ- mendation but might also lead to a justified request for
ent treatment or lifestyle interventions, an explainable AI explanation—which in the case of an opaque system, the
decision aid could help to raise patients’ choice aware- clinician would not be able to provide. Opaque medical
ness and support clinicians in eliciting patient values and AI can, therefore, represent an obstacle to the provision
preferences [34]. As described previously, explainabil- of accurate information and thus potentially jeopardize
ity provides a visual representation or natural language informed consent. Appropriate ethical and explainabil-
explanation of how different factors contributed to the ity standards are therefore important to safeguard the
final risk assessment. Yet, to interpret system-derived autonomy-preserving function of informed consent.
explanations and probabilities, patients rely on the cli- Attention should be paid to the risk that the introduc-
nician’s ability to understand and convey these explana- tion of opaque AI into medical decision making may
tions in a way that is accurate and understandable. If used foster paternalism by limiting opportunities for patients
appropriately, explainable AI decision support systems to express their expectations and preferences regarding
may not only contribute to patients feeling more knowl- medical procedures [39]. A necessary prerequisite for
edgeable and better informed but could also promote shared decision making is full autonomy of the patient,
Amann et al. BMC Med Inform Decis Mak (2020) 20:310 Page 7 of 9

but full autonomy can only be achieved if the patient is a question of epistemic authority, and it is unclear how
presented with a range of meaningful options to choose physicians should decide whether they can trust the epis-
from [40]. In this respect, patients’ opportunities to temic authority of a black box model enough to defer to
exert their autonomy regarding medical procedures get its decision [42]. Grote et al. [42] argue that in the case
reduced as opaque AI becomes more central to medical of opaque AI there is not enough epistemic support for
decision making. In particular, the challenge that arises deference. Moreover, they further argue that confronted
with opaque CDSS is that it remains unclear whether with a black-box system, clinical decision support might
and how patient values and preferences are accounted not enhance the capabilities of physicians, but rather
for by the model. This state of affairs could be addressed limit them. Here, physicians might be forced into “defen-
by means of “value-flexible” AI that provides differ- sive medicine” where they dogmatically follow the out-
ent options for the patient [41]. We further argue that put of the machine to avoid being questioned or held
explainability is a necessary step towards value-flexible accountable [42]. Such a situation would cause a serious
AI. The patient needs to be able to understand which threat to physician autonomy. Additionally, physicians
variables play an important role in the inner workings of will rarely have the time to perform an in-depth analysis
the AI system to determine—with the aid of the doctor— of why their clinical judgement is in disagreement with
whether the goals and weighting of the AI system align the AI system. Thus, looking merely at a performance
with their values or not. For example, AI systems primed output is not sufficient in the clinical context. The opti-
for “survival” as the outcome might not be aligned with mal outcome for all patients can only be expected with
the value of patients for whom a “reduction of suffering” healthcare staff that can make informed decisions when
is more important [41]. Lastly, when a choice is made, to apply an AI-powered CDSS and how to interpret its
patients need to be able to trust an AI system to decide results. It is thus hard to imagine how beneficence in the
with confidence and autonomy to follow its guidance context of medical AI can be fulfilled with any “black
[42]. This is not possible when the AI model is opaque. box” application.
Therefore, explainability is—both from the physician’s The need for explainability is also evident when assess-
and patient’s point-of-view—an ethical prerequisite for ing the principle of non-maleficence in the context of
systems supporting critical medical decision making. medical AI. Non-maleficence states that physicians have
While the principles of beneficence and non-malefi- a fundamental duty not to harm their patients either
cence are related, they nonetheless shed light on different intentionally or through excessive or inappropriate use of
aspects, also with regards to explainability. Beneficence medical means. Why is performance not enough? It has
urges physicians to maximize patient benefits. When been argued that a black box medical AI-based only on
applying AI-based systems, physicians are thus expected validated maximized performance is ethically justifiable
to use the tools in a manner that promotes the opti- even if the causal mechanisms behind a given AI-pre-
mal outcome for the respective patient. Yet, to provide scribed intervention remain opaque to the clinician [44].
patients with the most appropriate options to promote Reliance on anecdotal or purely experiential evidence
their health and wellbeing, physicians need to be able about the efficacy of a given treatment is indeed still quite
to use the full capabilities of the system. This implies common in medicine. Yet this is no excuse to forego
that physicians have knowledge of the system beyond a explanations as a major requirement of sound clinical
robotic application in a certain clinical use case, allow- judgment when such an explanation is indeed possible.
ing them to reflect on the system’s output. For physicians, Recent progress in elucidating at least the principal fea-
explainability in the form of visualizations or natural lan- tures of AI models, while not providing full mechanistic
guage explanations enables confident clinical decisions explanations of AI-decisions, create a prima facie ethical
instead of having to simply trust an automated output. obligation to reduce opacity and increase the interpret-
They can critically assess the system-derived outcomes ability of medical AI. Failure to do so would mean inten-
and make their own judgments whether the results seem tionally undermining a physician’s capacity to control
trustworthy or not. This allows them to adapt predic- for possible misclassifications of individual clinical cases
tions and recommendations to individual circumstances due, for instance, to excessive bias or variance in train-
where necessary. As such, clinicians can not only reduce ing datasets. We thus conclude that also with regards to
the risk of eliciting false hope or creating false despair beneficence and non-maleficence, explainability is a nec-
but can also flag potentially inappropriate interven- essary characteristic of clinically applied AI systems.
tions using their clinical judgment [43]. This is especially The principle of justice postulates that people should
important when we imagine a situation where a physi- have equal access to the benefits of medical progress
cian and an AI system are in disagreement, a situation without ethically unjustified discrimination of any par-
that is not easily resolved [42]. Fundamentally, this is ticular individuals or social group [36]. Some AI systems,
Amann et al. BMC Med Inform Decis Mak (2020) 20:310 Page 8 of 9

however, violate this principle. Recently, for example, professionals, and legislators to the challenges and
Obermeyer et al. reported on a medical AI system dis- limitations of opaque algorithms in medical AI and to
criminating against people of color [5]. Explainability can foster multidisciplinary collaboration to tackle these
support developers and clinicians to detect and correct challenges with joined forces.
such biases—a major potential source for injustice—ide-
ally at the early stage of AI development and validation,
Abbreviations
e.g. by identification of important features indicating AI: Artificial intelligence; ANN: Artificial neural network; CDSS: Clinical decision
a bias in the model. However, for explainability to ful- support system; FDA: Food and Drug Administration; GDPR: General Data Pro-
fill this purpose, the relevant stakeholder groups must tection Regulation; ICU: Intensive care unit; MDR: Medical Device Regulation;
TPLC: Total product lifecycle approach.
be sensitized to the risk of bias and its potential conse-
quences for individuals’ health and wellbeing. At times, Acknowledgements
it might be tempting to prioritize accuracy and simply The authors would like to thank Dr. Nora A. Tahy for review of the manuscript.
refrain from investing resources into developing explain- Authors’ contributions
able AI. Yet to ensure that AI-powered decision support JA: Conceptualization; analysis, writing—original draft; writing—review and
systems realize their potential, developers, and clinicians editing. AB: analysis; writing—original draft; writing—review and editing. EV:
analysis; writing—original draft; writing—review and editing. DF: analysis;
need to be attentive to the potential flaws and limitations writing—original draft; writing—review and editing. VIM: conceptualization;
of these new tools. Thus, also from the justice perspec- analysis, writing—original draft; Writing—review and editing. All authors read
tive, explainability becomes an ethical prerequisite for and approved the final manuscript.
the development and application of AI-based clinical Funding
decision support. This research has received funding from the European Union’s Horizon 2020
research and innovation programme under grant agreement No. 777107
(PRECISE4Q). The funding body had no role in the study design, the collection,
Conclusion analysis, and interpretation of the data nor the preparation of the manuscript.
In this paper, we explored the role of explainable AI in Availability of data and materials
clinical decision support systems from the technological, Not applicable.
legal, medical, and patient perspectives. In doing so, we
Ethics approval and consent to participate
have shown that explainability is a multifaceted concept Not applicable.
that has far-reaching implications for the various stake-
holder groups involved. Medical AI poses challenges to Consent for publication
Not applicable.
developers, medical professionals, and legislators as it
requires a reconsideration of roles and responsibilities. Competing interests
Based on our analysis, we consider explainability a neces- The authors declare no competing interests.
sary requirement to address these challenges in a sustain- Author details
able manner that is compatible with professional norms 1
Health Ethics and Policy Lab, Department of Health Sciences and Technol-
and values. ogy, ETH Zurich, Hottingerstrasse 10, 8092 Zurich, Switzerland. 2 Charité Lab
for Artificial Intelligence in Medicine—CLAIM, Charité - Universitätsmedizin
Notably, a move towards opaque algorithms in CDSS Berlin, Berlin, Germany. 3 School of Computing and Digital Technology, Faculty
may inadvertently lead to a revival of paternalistic con- of Computing, Engineering and the Built Environment, Birmingham City
cepts of care that relegate patients to passive spectators University, Birmingham, UK.
in the medical decision-making process. It might also Received: 22 July 2020 Accepted: 15 November 2020
bring forward a new type of medicine where physicians
become slaves to the tool’s output to avoid legal and
medical repercussions. And, last but not least, opaque
systems might provoke a faulty allocation of resources
violating their just distribution. In this paper, we have References
argued that explainability can help to ensure that patients 1. Higgins D, Madai VI. From bit to bedside: a practical framework for
remain at the center of care and that together with clini- artificial intelligence product development in healthcare. Adv Intell
Syst. 2020;2:2000052.
cians they can make informed and autonomous decisions 2. Rudin C. Stop explaining black box machine learning models for high
about their health. Moreover, explainability can promote stakes decisions and use interpretable models instead. Nat Mach Intell.
the just distribution of available resources. 2019;1:206–15.
3. Doran D, Schulz S, Besold TR. What does explainable AI really mean?
We conclude that omitting explainability in clinical A new conceptualization of perspectives. ArXiv171000794 Cs. 2017.
decision support systems poses a threat to core ethi- http://arxiv​.org/abs/1710.00794​. Accessed 3 Sept 2019.
cal values in medicine and may have detrimental con- 4. Shortliffe EH, Sepúlveda MJ. Clinical decision support in the era of
artificial intelligence. JAMA. 2018;320:2199–200.
sequences for individual and public health. Further
work is needed to sensitize developers, healthcare
Amann et al. BMC Med Inform Decis Mak (2020) 20:310 Page 9 of 9

5. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias 25. Barry MJ, Edgman-Levitan S. Shared decision making—the pinnacle
in an algorithm used to manage the health of populations. Science. patient-centered care. N Engl J Med. 2012;366:780–1.
2019;366:447–53. 26. Kunneman M, Montori VM, Castaneda-Guarderas A, Hess EP. What
6. Samek W, Montavon G, Vedaldi A, Hansen LK, Müller K-R, editors. Explain- is shared decision making? (and What it is not). Acad Emerg Med.
able AI: interpreting, explaining and visualizing deep learning. Berlin: 2016;23:1320–4.
Springer; 2019. https​://doi.org/10.1007/978-3-030-28954​-6. 27. O’Neill ES, Grande SW, Sherman A, Elwyn G, Coylewright M. Availability of
7. Esteva A, Robicquet A, Ramsundar B, Kuleshov V, DePristo M, Chou K, patient decision aids for stroke prevention in atrial fibrillation: a system-
et al. A guide to deep learning in healthcare. Nat Med. 2019;25:24–9. atic review. Am Heart J. 2017;191:1–11.
8. Islam SR, Eberle W, Ghafoor SK. Towards quantification of explainability 28. Noseworthy PA, Brito JP, Kunneman M, Hargraves IG, Zeballos-Palacios C,
in explainable artificial intelligence methods. ArXiv191110104 Cs Q-Fin. Montori VM, et al. Shared decision-making in atrial fibrillation: navigating
2019. http://arxiv​.org/abs/1911.10104​. Accessed 2 Oct 2020. complex issues in partnership with the patient. J Interv Card Electro-
9. Samek W, Montavon G, Lapuschkin S, Anders CJ, Müller K-R. Toward physiol. 2019;56:159–63.
interpretable machine learning: transparent deep neural networks and 29. Dobler CC, Sanchez M, Gionfriddo MR, Alvarez-Villalobos NA, Ospina
beyond. ArXiv200307631 Cs Stat. 2020. http://arxiv​.org/abs/2003.07631​. NS, Spencer-Bonilla G, et al. Impact of decision aids used during clinical
Accessed 2 Oct 2020. encounters on clinician outcomes and consultation length: a systematic
10. Lapuschkin S, Wäldchen S, Binder A, Montavon G, Samek W, Müller K-R. review. BMJ Qual Saf. 2019;28:499–510.
Unmasking Clever Hans predictors and assessing what machines really 30. Noseworthy PA, Kaufman ES, Chen LY, Chung MK, Elkind Mitchell SV,
learn. Nat Commun. 2019;10:1096. Joglar JA, et al. Subclinical and device-detected atrial fibrillation: ponder-
11. Zech JR, Badgeley MA, Liu M, Costa AB, Titano JJ, Oermann EK. Vari- ing the knowledge gap: a scientific statement from the American Heart
able generalization performance of a deep learning model to detect Association. Circulation. 2019;140:e944–63.
pneumonia in chest radiographs: a cross-sectional study. PLOS Med. 31. Spencer-Bonilla G, Thota A, Organick P, Ponce OJ, Kunneman M, Giblon
2018;15:e1002683. R, et al. Normalization of a conversation tool to promote shared decision
12. Olsen HP, Slosser JL, Hildebrandt TT, Wiesener C. What’s in the box? The making about anticoagulation in patients with atrial fibrillation within
legal requirement of explainability in computationally aided decision- a practical randomized trial of its effectiveness: a cross-sectional study.
making in public administration. SSRN Scholarly Paper. Rochester: Social Trials. 2020;21:395.
Science Research Network; 2019. https​://doi.org/10.2139/ssrn.34029​74. 32. Bonner C, Bell K, Jansen J, Glasziou P, Irwig L, Doust J, et al. Should heart
13. Schönberger D. Artificial intelligence in healthcare: a critical analysis of age calculators be used alongside absolute cardiovascular disease risk
the legal and ethical implications. Int J Law Inf Technol. 2019;27:171–203. assessment? BMC Cardiovasc Disord. 2018;18:19.
14. Cohen IG. Informed consent and medical artificial intelligence: what 33. Bjerring JC, Busch J. Artificial intelligence and patient-centered decision-
to tell the patient? SSRN Scholarly Paper. Rochester, NY: Social Science making. Philos Technol. 2020. https​://doi.org/10.1007/s1334​7-019-00391​
Research Network; 2020. https​://doi.org/10.2139/ssrn.35295​76. -6.
15. Beaudouin V, Bloch I, Bounie D, Clémençon S, d’Alché-Buc F, Eagan J, 34. Politi MC, Dizon DS, Frosch DL, Kuzemchak MD, Stiggelbout AM. Impor-
et al. Identifying the “right” level of explanation in a given situation. SSRN tance of clarifying patients’ desired role in shared decision making to
Electron J. 2020. https​://doi.org/10.2139/ssrn.36049​24. match their level of engagement with their preferences. BMJ. 2013. https​
16. FDA. Proposed regulatory framework for modifications to artificial intel- ://doi.org/10.1136/bmj.f7066​.
ligence/machine learning (AI/ML)-based Software as a Medical Device 35. Stacey D, Légaré F, Lewis K, Barry MJ, Bennett CL, Eden KB, et al. Decision
(SaMD). 2020. https​://www.fda.gov/files​/medic​al%20dev​ices/publi​shed/ aids for people facing health treatment or screening decisions. Cochrane
US-FDA-Artif​i cial​-Intel​ligen​ce-and-Machi​ne-Learn​ing-Discu​ssion​-Paper​ Database Syst Rev. 2017. https​://doi.org/10.1002/14651​858.CD001​431.
.pdf. Accessed 5 July 2020. pub5.
17. Hacker P, Krestel R, Grundmann S, Naumann F. Explainable AI under 36. Beauchamp TL. Principles of biomedical ethics. Paperback May-2008.
contract and tort law: legal incentives and technical challenges. SSRN New York: Oxford University Press; 2008.
Scholarly Paper. Rochester, NY: Social Science Research Network; 2020. 37. Gillon R. Defending the four principles approach as a good basis for good
https​://paper​s.ssrn.com/abstr​act=35134​33. Accessed 13 Feb 2020. medical practice and therefore for good medical ethics. J Med Ethics.
18. Ferretti A, Schneider M, Blasimme A. Machine learning in medicine: 2015;41:111–6.
opening the new data protection black box. Eur Data Prot Law Rev EDPL. 38. Mittelstadt B. Principles alone cannot guarantee ethical AI. Nat Mach
2018;4:320. Intell. 2019;1:501–7.
19. Weng SF, Reps J, Kai J, Garibaldi JM, Qureshi N. Can machine-learning 39. Faden RR, Beauchamp TL. A history and theory of informed consent.
improve cardiovascular risk prediction using routine clinical data? PLoS Oxford: Oxford University Press; 1986.
ONE. 2017;12:e0174944. 40. Raz J. The Morality of Freedom. Oxford: Oxford University Press; 2020.
20. Kakadiaris IA, Vrigkas M, Yen AA, Kuznetsova T, Budoff M, Naghavi M. https​://doi.org/10.1093/01982​48075​.001.0001/acpro​f-97801​98248​071.
Machine learning outperforms ACC/AHA CVD risk calculator in MESA. J 41. McDougall RJ. Computer knows best? The need for value-flexibility in
Am Heart Assoc. 2018;7:e009476. medical AI. J Med Ethics. 2019;45:156–60.
21. Liu T, Fan W, Wu C. A hybrid machine learning approach to cerebral 42. Grote T, Berens P. On the ethics of algorithmic decision-making in health-
stroke prediction based on imbalanced medical dataset. Artif Intell Med. care. J Med Ethics. 2019. https​://doi.org/10.1136/medet​hics-2019-10558​6.
2019;101:101723–101723. 43. Beil M, Proft I, van Heerden D, Sviri S, van Heerden PV. Ethical consid-
22. Cutillo CM, Sharma KR, Foschini L, Kundu S, Mackintosh M, Mandl KD. erations about artificial intelligence for prognostication in intensive
Machine intelligence in healthcare—perspectives on trustworthiness, care. Intensive Care Med Exp. 2019. https​://doi.org/10.1186/s4063​
explainability, usability, and transparency. NPJ Digit Med. 2020;3:1–5. 5-019-0286-6.
23. Tonekaboni S, Joshi S, McCradden MD, Goldenberg A. What clinicians 44. London AJ. Artificial intelligence and black-box medical decisions: accu-
want: contextualizing explainable machine learning for clinical end use. racy versus explainability. Hastings Cent Rep. 2019;49:15–21.
ArXiv190505134 Cs Stat. 2019. http://arxiv​.org/abs/1905.05134​. Accessed
3 Sept 2019.
24. Institute of Medicine (US) Committee on Quality of Health Care in Publisher’s Note
America. Crossing the quality chasm: a new health system for the 21st Springer Nature remains neutral with regard to jurisdictional claims in pub-
century. Washington, DC: National Academies Press (US); 2001. http:// lished maps and institutional affiliations.
www.ncbi.nlm.nih.gov/books​/NBK22​2274/. Accessed 21 May 2020.