Science and Engineering Ethics

https://doi.org/10.1007/s11948-019-00084-5

ORIGINAL RESEARCH/SCHOLARSHIP

Building Moral Robots: Ethical Pitfalls and Challenges

John‑Stewart Gordon1,2 

Received: 9 July 2018 / Accepted: 10 January 2019

© Springer Nature B.V. 2019

Abstract
This paper examines the ethical pitfalls and challenges that non-ethicists, such as
researchers and programmers in the fields of computer science, artificial intelligence
and robotics, face when building moral machines. Whether ethics is “computable”
depends on how programmers understand ethics in the first place and on the ade-
quacy of their understanding of the ethical problems and methodological challenges
in these fields. Researchers and programmers face at least two types of problems
due to their general lack of ethical knowledge or expertise. The first type is so-called
rookie mistakes, which could be addressed by providing these people with the nec-
essary ethical knowledge. The second, more difficult methodological issue concerns
areas of peer disagreement in ethics, where no easy solutions are currently avail-
able. This paper examines several existing approaches to highlight the ethical pit-
falls and challenges involved. Familiarity with these and similar problems can help
programmers to avoid pitfalls and build better moral machines. The paper concludes
that ethical decisions regarding moral robots should be based on avoiding what is
immoral (i.e. prohibiting certain immoral actions) in combination with a pluralistic
ethical method of solving moral problems, rather than relying on a particular ethical
approach, so as to avoid a normative bias.

Keywords  Moral machines · Full ethical agents · Ethical expertise · Programming

ethics · Moral pluralism

* John‑Stewart Gordon
[email protected]; [email protected]
http://www.johnstgordon.com
1
Department of Philosophy and Social Critique, Faculty of Political Science and Diplomacy,
Vytautas Magnus University, V. Putvinskio g. 23 (R 403), 44243 Kaunas, Lithuania
2
Research Cluster for Applied Ethics, Faculty of Law, Vytautas Magnus University, V. Putvinskio
g. 23 (R 403), 44243 Kaunas, Lithuania

13
Vol.:(0123456789)
 J.-S. Gordon

Introduction

In recent years, the interdisciplinary field of machine ethics—that is, how to pro-
gram robots with ethical rules so that they become either implicit or explicit moral
agents (Moor 2006)—has become of utmost importance because of current and
anticipated technological developments in the fields of computer science, artificial
intelligence (AI) and robotics (Lin et al. 2014; Wallach and Allen 2010; Anderson
and Anderson 2011a; Gunkel and Bryson 2014). One of the most important issues
in machine ethics concerns the goal of making artificially intelligent robots “ethi-
cal”. Susan Anderson, a pioneer in the field, defines this goal as “to create a machine
that follows an ideal ethical principle or set of principles in guiding its behaviour;
in other words, it is guided by this principle, or these principles, in the decisions it
makes about possible courses of action it could take. We can say, more simply, that
this involves ‘adding an ethical dimension’ to the machine” (2011c: 22).
Based on the great technological advances in AI, the advent of fully autonomous,
human-like, intelligent robots capable of ethical reasoning and decision making
seems unavoidable. There exist various ethical programs with numerous approaches
that could also—sooner or later—be used for robots. This paper is not concerned
with the moral and legal status of such intelligent artificial beings. Rather, it focuses
on the ethical pitfalls and challenges that researchers and programmers in the fields
of computer science, AI and robotics face when they attempt to build such moral
machines. Whether ethics is “computable”1 depends in part on how those research-
ers understand ethics and attempt to implement that understanding in programs,
but also more generally on whether they have some expertise in ethics and moral
philosophy in the first place. Based on a review of current programs in the field, it
appears that at this time, researchers and programmers have neither a good enough
understanding nor sufficient ethical expertise to build moral machines that would be
comparable to human beings with respect to ethical reasoning and decision making.
In general, one can observe two main types of problems that non-ethicists commit
when they attempt to add an “ethical dimension” to their machines. The first type
could be called rookie mistakes; they involve a misunderstanding of moral issues
or starting from wrong ethical assumptions (such as overestimating the capacity of
a given ethical approach to solve complex moral cases). These problems show that
the researchers and programmers committing the errors are somewhat unaware of
the moral complexity of the issues with which they are grappling or the knowledge
already achieved in the field of ethics. The second category contains methodological
issues that pose a challenge even to ethical experts because they disagree on how to
solve those issues, as will be discussed below.
Building moral machines is quite conceivable and could be a valuable advance
for humanity. However, achieving this goal will require the builders to have suitable
ethical expertise. This paper highlights some of the most significant ethical pitfalls

1
  In this paper, the concept of computability is used in a rather broad sense. It concerns the way in which
a machine is built to make ethical decisions that generally seek to resemble the moral reasoning and deci-
sion making of human beings.

13
Building Moral Robots: Ethical Pitfalls and Challenges

and challenges that stand in the way of building moral machines. It argues that ethi-
cal decisions of moral robots should rather be based on (1) avoiding the immoral, or
prohibiting certain immoral actions, and (2) a pluralistic ethical method for solving
moral problems, as opposed to reliance on a particular ethical approach.
The next section of this paper provides some brief preliminary remarks on
the meaning of moral agency in the context of AI. The third part examines four
recent ethical programs that are supposed to mirror human-like ethical reasoning
and decision making: (1) the casuistic approach (Guarini 2006), (2) the top-down
approach of the MoralDM (Dehghani et al. 2011), (3) the hybrid approach (Wallach
et  al. 2010), and (4) the web-based approach (Rzepka and Araki 2005). Based on
this review, the fourth part explains why researchers and programmers have failed
to achieve ethical reasoning and decision making in their programs. Four issues
are presented—lack of ethical expertise, pluralism in ethics, dilemma cases, and
machine bias—that indicate that the arrival of moral machines is a long way off. The
paper then deals with two vital methodological issues—the problem of justification
and avoiding the immoral—that should be taken into account in the construction of
moral machines. The final part offers some conclusions.

Preliminary Remarks

Autonomy and AI: The Preconditions of Moral Agency

According to Bekey (2014: 18), the notion of autonomy can be defined as “the
capacity to operate in the real-world environment without any form of external
control, once the machine is activated and at least in some areas of operation, for
extended periods of time”. That means that if we maximally increase the capacity
to operate in the real-world environment, we could create intelligent robots (IRs)
that are able to perform various tasks of utmost complexity, most probably even
unmatched by human beings. The way to increase this capacity is to perfect the use
of AI in robots, i.e. “making machines do things that would require intelligence if
done by men” (Marvin Minsky, cited by Copeland 1993: 1).
Technically speaking, IRs need proper algorithms to solve any problem or issue
based on the input perceived. It has been suggested, however, that IRs cannot
become moral agents because the algorithms are finite or deterministic by nature
and hence the robots lack free will, which is commonly understood as a prerequisite
for moral agency. If IRs must act according to a fixed set of rules, then they could
not be seen as moral agents (Johnson and Axinn 2014). This objection has already
been addressed at length (Gordon 2018), but two important points should be briefly
reviewed here. First, it is possible that IRs will be able to learn from their past expe-
riences in such a way as to re-program themselves and hence alter their algorithms
by changing their pre-fixed set of rules to adapt to new situations. Second, many
influential philosophers and neurobiologists working on the interdisciplinary and
complex issue of free will have forcefully questioned (1) whether human beings
have free will and (2) whether free will is a necessary precondition of moral agency.
To show that moral machines will never be able to make human-like autonomous

13
 J.-S. Gordon

decisions, opponents must counter these two objections. Such a decisive refutation
has not been provided yet, and it is very unlikely that it will be provided any time
soon. Therefore, the free will objection—i.e. to deny IRs moral agency because
of the nature of their algorithms—is unconvincing, because it suggests that robots
should meet a supposedly necessary precondition for moral agency (free will) the
fulfilment of which even by human beings remains in dispute.

Implicit, Explicit and Full Ethical Agents

In his influential paper “The Nature, Importance, and Difficulty of Machine Eth-
ics” (2006), James H. Moor made an important distinction between various forms of
ethical agents—ethical-impact agents, implicit ethical agents, explicit ethical agents
and full ethical agents—which has been frequently quoted by other authors. Accord-
ing to Moor, an ethical-impact agent is a machine that helps to avoid an otherwise
immoral situation; as an example, he cites the development of robotic camel jockeys
to replace small boys who are kept as slaves to ride the camels in races. “Computing
technology often has important ethical impact. The young boys replaced by robotic
camel jockeys are freed from slavery” (Moor 2006: 19). An implicit ethical agent is
a machine programmed to act ethically or to avoid unethical behaviour by following
the programming of a designer who in turn follows ethical principles (for example,
ATMs, web banking software or autopilot). An explicit ethical agent, in contrast,
actually uses ethical principles to solve ethical issues. Finally, “A full ethical agent
can make explicit ethical judgments and generally is competent to reasonably justify
them” (Moor 2006: 20). Explicit ethical agents and full ethical agents both use ethi-
cal principles to make ethical decisions, but only the latter can provide a justification
for those decisions.
Most authors refer to only two of these four categories (implicit and explicit) and
thereby leave the impression that explicit ethical agents are the prime example of
moral agents. This is unfortunate, since it confines the discussion to “limited explicit
ethical agents” instead of full ethical agents (Moor 2006: 21).2 Moor himself states:
A full ethical agent can make explicit ethical judgements and generally is
competent to reasonably justify them. An average adult human is a full ethi-
cal agent. We typically regard humans as having consciousness, intentionality,
and free will. Can a machine be a full ethical agent? It’s here that the debate
about machine ethics becomes most heated. Many believe a bright line exists
between the senses of machine ethics discussed so far and a full ethical agent.
For them, a machine can’t cross this line. The bright line marks a crucial onto-
logical difference between humans and whatever machines might be in the
future. (Moor 2006: 20)

2
  For example, Anderson and Anderson (2007: 15) observe, “Using Moor’s terminology, most of those
working on machine ethics would say that the ultimate goal is to create a machine that is an explicit
ethical agent.” This might be true, but we should be prepared for the advent of IRs who are full ethical
agents, regardless of how long it may take for such IRs to become a reality.

13
Building Moral Robots: Ethical Pitfalls and Challenges

Moor believes that it is not possible to answer the question of whether machines
will become full ethical agents by adhering to “philosophical argument or empirical
research” and therefore suggests focusing on “developing limited explicit agents”
even though they will “fall short of being full ethical agents”. They could, according
to Moor, still “help prevent unethical outcomes”, but one should not be “too opti-
mistic about our ability to develop” even such machines (Moor 2006: 21).
This perspective faces two problems. First, it is unclear whether Moor believes
that it is, in general, impossible to answer by philosophical or empirical means the
question of whether machines could become full ethical agents, or only that humans
are currently unable to do so. A more optimistic approach will assume only the latter
claim. Second, his cautionary statement that it might be impossible to develop even
machines that are explicit ethical agents is rather unconvincing. His reasons are as
follows: (1) even human beings have a limited understanding of the complex topic
of ethics, (2) humans’ lack of understanding of the phenomenon of learning, particu-
larly concerning the child machine envisioned by Turing,3 and (3) the computer’s
lack of common sense and world knowledge (Moor 2006: 21). The main reason why
one should reject these objections is that the trajectory of technological development
seems quite promising; moreover, many would argue that moral philosophy itself
has, despite the continued and undisputed existence of unresolved complex moral
issues, nonetheless made significant progress during the last centuries as well.
However, one should not overlook two vital points concerning the contested ques-
tion of moral progress. First, it seems easier to formulate moral rules that classify
certain actions as immoral than to prescribe correct moral behaviour for all people
(see Gordon 2016b). This is because there exist a plurality of lifestyles from which
different normative actions may emerge that can be seen as equally good or, at least,
not morally reprehensible, whereas we can identify particular actions which are gen-
erally acknowledged as immoral, such as rape, torture, theft and child molestation.
Second, there have been important, undeniable moral developments in the history
of humanity that can be counted as moral progress, such as the abolition of slavery
(even though many people are still kept as slaves around the world), the recogni-
tion of universal human rights (and their transition into international legislation), the
idea of a fair trial and that the punishment should always match the crime, gender
equality and the recognition of animal rights. The notion of moral progress does not
claim that all these points are fully accepted and acknowledged by all people, but
that humanity overall has made great moral progress on these issues during the last
few centuries.
Hence, there is no reason not to consider seriously the possibility of machines
that will become full ethical agents. Should IRs become more complex, more pow-
erful, highly autonomous and endowed with an unmatched artificial intelligence,
moral standards that match the great freedom of such machines will be essential to
avoid causing harm (see Picard 1997).

3
  See Allen et al. (2000: 251–261) for the interesting idea of the so-called Moral Turing Test (MTT) that
robots would pass if they (1) give correct answers to moral dilemmas and (2) offer the appropriate justifi-
cation for their claims. This would amount to understanding ethics.

13
 J.-S. Gordon

The Idea of Moral Machines

This section offers a brief overview of some recent attempts to show how intelligent
machines should actually make moral decisions.4 The different approaches can be
categorized as follows:

1. Bottom-up approaches: casuistry (Guarini 2006); virtue ethics (Abney 2014).

2. Top-down approaches: the MoralDM (Dehghani et al. 2011); prima facie duties
(Anderson and Anderson 2011b: 476–492); deontology (Powers 2011); divine
command ethics (Bringsjord and Taylor 2014).
3. Mixed approaches: the moral reasoner (Pontier and Hoorn 2012); the hybrid
approach (Wallach et al. 2010; Wallach and Allen 2010).
4. Other approaches: the web-based approach (Rzepka and Araki 2005); the Bud-
dhist approach (Hughes 2014).

Table 1 summarizes and comments briefly on four of these approaches. A more

detailed response from an ethical perspective follows.

Comments: How Not to Build Ethical Programs

Whereas Guarini’s system faces problems with regard to the reclassification of cases
due to the lack of proper reflection and exact representation (Guarini admits that
casuistry alone is insufficient for machine ethics), Rzepka and Araki’s web-based
GENTA approach offers a kind of majority-rule ethics based on the average behav-
iour of human beings (including immoral actions). From a moral point of view, the
idea of relying on a majority-based decision procedure in ethics is not only mislead-
ing, but can never be considered an appropriate means of ethical reasoning and deci-
sion making. The MoralDM approach of Dehghani et al. is of utmost significance in
that it pays proper respect to the two main ethical theories (deontology and utilitari-
anism) and combines them in a way that is fruitful and quite promising. However,
their additional strategy of using empirical studies to mirror human moral decisions
by considering as correct only those decisions that align with the majority of human
subjects is misleading and seriously flawed. In this regard, Susan Anderson (2011a:
525) rightly claims that their system can be regarded only as a model for descrip-
tive and not for normative ethics. The same objection could be raised against the
hybrid model of moral reasoning developed by Wallach, Franklin and Allen, who
attempt to re-create human decision making by appealing to a complex combination
of top-down and bottom-up approaches. In addition, their somewhat idiosyncratic

4
  It is not claimed that the approaches analysed in the table are the only important ones, but they are a
representative sample of the existing range of accounts that have been extensively discussed in recent
years.

13
 Table 1  Machine ethics: four approaches
Bottom-up approach Top-down approach Mixed approach Other

Guarini (2006): The casuistic approach Dehghani et al. (2011): The MoralDM Wallach et al. (2010), Wallach and Allen Rzepka and Araki (2005): The web-based
(2010): The hybrid approach approach
The system (using a neural network) Their system combines two main ethical Their hybrid model of human cognition Their system, GENTA (General Belief
bases its ethical decisions on a learn- theories, utilitarianism and deontol- combines a so-called top-down (the- Retrieving Agent), makes ethical deci-
ing process in which it is presented ogy, in combination with analogi- ory-driven reasoning) and a bottom-up sions based on statistical analysis using
Building Moral Robots: Ethical Pitfalls and Challenges

with known correct answers to ethical cal reasoning. Utilitarian reasoning approach (shaped by evolution and the full spectrum of web-based knowl-
dilemmas. After that, the system is applies until “sacred values” are learning) that is both considered the edge with respect to human behaviour,
able to solve new ethical dilemmas on involved, at which point the system foundation for moral reasoning and opinions, common consequences,
its own operates in a deontological mode and decision making. The result thus far is exceptions, etc
becomes less sensitive to the utility LIDA, an artificial general intelligence
of actions and consequences. To software offering a comprehensive,
align the system with human moral conceptual and computational model
decisions, they evaluate the system that covers a large portion of human
against psychological studies of how cognition
the majority of human beings decided
particular cases

13
 J.-S. Gordon

understanding of both approaches in moral philosophy does not really match how
moral philosophers understand and use them in normative ethics.5
The ethical pitfalls and challenges that undermine all four approaches—reliance
on casuistry alone (Guarini), the use of majority-based decision procedures in ethics
(MoralDM and GENTA), and the idea of combining two types of moral reasoning
and decision making that are commonly viewed as mutually exclusive (LIDA)—
could have been avoided through a more comprehensive understanding of ethics and
moral philosophy. Although debate continues over the ethical merit of casuistry (it
seems to be a matter of degree), relying on majority opinion as a basis for discerning
ethical truth goes beyond any reasonable and acceptable standards in ethics. This
objection is also substantiated by common sense. The only solution to such prob-
lems is to work closely with moral philosophers so as to avoid steps that the philoso-
phers will readily identify as rookie mistakes. There are other more complex ethical
issues that even moral philosophers cannot solve completely, but even in those cases
they can clearly determine the demarcation lines and describe the problem suffi-
ciently to understand the implications of a given issue.
The next section examines some of the main ethical pitfalls and challenges that
occur in efforts to build moral machines. Researchers and programmers should
become more aware of the deep methodological problems related to ethical reason-
ing and decision making that—without proper understanding—may hinder them in
their endeavours to construct intelligent moral machines.

Understanding Ethical Pitfalls and Challenges When Building Moral

Machines

The above-mentioned approaches that attempt to empower IRs to make sound moral
decisions are unconvincing for several reasons, which will be examined in this sec-
tion. This critique is not intended to imply that IRs will never be able to make moral
decisions and thereby become full ethical agents; rather, the analysis suggests that
ethical reasoning and decision making are, by virtue of their complex nature, more
difficult than AI researchers and programmers may assume at first glance (Anderson
2011b).6 In the long term, however, there is no principled reason why moral IRs
should not eventually become a reality (Anderson 2011b: 164).

5
  The model presented by Wallach et al. is not necessarily inadequate with respect to how moral deci-
sion making works in an empirical sense, but their approach is descriptive rather than normative by
nature. Therefore, their empirical model does not solve the normative problem of how moral machines
should actually act. Descriptive ethics and normative ethics are two different things. The first tells us how
human beings make moral decisions; the latter is concerned with how we should act.
6
  For a brief but good overview of the philosophical concerns, see Anderson (2011b: 162–167) and Gips
(2011: 244–253).

13
Building Moral Robots: Ethical Pitfalls and Challenges

Lack of Ethical Expertise

Because of the lack of sufficient ethical expertise among non-ethicists who work in
the field of machine ethics, there is a great danger that AI researchers will start from
wrong ethical assumptions and build their ethical approach to machines on shaky
grounds. For example, Anderson and Anderson (2007), who are among the pioneers
in the field of machine ethics, developed three different applications: (1) a medi-
cal advisor system for dilemmas (MedEthEx), (2) a medication reminder system for
elderly people (EthEl), and (3) the instantiation of EthEl in a Nao robot, which they
believe is the “first example of a robot that follows an ethical principle in determin-
ing which actions it will take” (2011b: 480). However, they base their famous medi-
cal advisor system (MedEthEx) on some contested claims that undermine their oth-
erwise interesting account. They state, “Because there is more agreement between
ethicists working on biomedical ethics than in other areas, and because there are
fewer duties, we decided to begin to develop our prima facie duty approach to com-
puting ethics using Beauchamp and Childress’s principles of biomedical ethics”
(Anderson and Anderson 2007: 22).
Experts in bioethics acknowledge that their field is one of the most complex
and challenging areas in all of applied ethics. There is no common ground or unity
among bioethicists. Furthermore, the suggested bioethical approach of principlism,
even though it has been widely used for many decades by practitioners and bioethi-
cists alike, is complex and not easy to apply, in particular with respect to the meth-
ods of specification and balancing (Gordon 2011a, b). As a result, Anderson and
Anderson start from misleading assumptions and use a theory that is quite demand-
ing and, according to Clouser and Gert (1990), highly controversial. They thus jeop-
ardise their own project because they misjudge the ethical complexity present in the
field of bioethics and overestimate the capacity of principlism to function as a role
model for machine ethics.
In another work, Susan Anderson (2011c: 21–27) takes a somewhat different
approach. Rightly pointing out that ethics is too complex for everyone to adhere to
a simple absolute ethical theory, she suggests instead starting from particular cases
and eventually using “a large number of cases in a fashion that permits automated
analysis. From this information, general ethical principles may emerge” (25). How-
ever, her own approach of multiple prima facie duties, based on Ross (1930), is
eventually faced with the general problem of making reasonable decisions between
conflicting duties. Furthermore, she interestingly suggests starting by working
on machines that will serve as “ethical advisors” before creating “moral robots”,
because the latter category requires that “a difficult judgment be made about the sta-
tus of the machine itself” (26). Granted, one can thereby avoid the difficult questions
of the moral status and moral rights of intelligent machines, but the earlier one starts
to think about these issues, the better equipped one will be when the time comes.
Furthermore, the idea of using machines as moral advisors is also supported by, for
example, Giubilini and Savulescu (2018), McLaren (2011: 310–312), and Torrance
(2008). For example, Torrance argues that, if we build such machines, we will also
learn something about human morality, and that such machines will, in general, be
useful and instructive to human beings.

13
 J.-S. Gordon

Pluralism in Ethics

Numerous ethical theories have been used to solve moral issues, among which the
most prominent are virtue ethics, deontology, utilitarianism, casuistry and contract
theory. There is currently no common ground among moral experts as to which ethi-
cal theory to use. Ethicists, like laypeople, choose among these varying approaches
when dealing with moral problems. This pluralism is equally visible in approaches
to machine ethics. The underlying idea that IRs will be better at making moral deci-
sions than human beings (Hall 2011: 522–5237; Dietrich 2011)8 faces at least two
problems. First, before one can claim that machines make morally better decisions,
one must have a proper moral approach that is, in fact, better than the other avail-
able theories.9 Second, the idea that many ethical issues could have equally good
but different decisions must be taken into account as well (for example, the same
moral problem could be adequately solved differently in different cultural settings).
Furthermore, when it comes to pluralism in ethics, it seems reasonable to avoid
extreme moral relativism and to accept a firm core of universal moral norms that all
human beings acknowledge (e.g. one must not commit murder or rape, insult other
people, or violate the human rights of others). Beyond this core of moral norms,
people should be free to act according to their particular moralities in their given
community, as long as those particular norms do not conflict with the core of uni-
versal norms or with the lingua franca of international human rights (Gordon 2015;
Macklin 1999). This view is further substantiated by Ruth Macklin’s pithy depiction
of the problem of moral relativism:
A long-standing debate surrounds the question whether ethics are relative to
time and place. One side argues that there is no obvious source of a universal
morality and that ethical rightness and wrongness are products of their cultural
and historical setting. Opponents claim that even if a universal set of ethical
norms has not yet been articulated or agreed upon, ethical relativism is a per-
nicious doctrine that must be rejected. The first group replies that the search
for universal ethical precepts is a quest for the Holy Grail. The second group
responds with the telling charge: If ethics were relative to time, place, and cul-
ture, then what the Nazis did was “right” for them, and there is no basis for
moral criticism by anyone outside the Nazi society. (Macklin 1999: 4)

7
  Hall (2011: 523) claims, “In the long run, AIs will run everything, and should, because they can do a
better job than humans. Not only will their intellectual prowess exceed ours, but their moral judgment as
well.”
8
  Dietrich argues that human beings, as biological entities, might have a genetic predisposition for selfish
behaviour as a survival mechanism with respect to competition with others. Machines lack this predis-
position and could be made in such a way that they both possess the good features of human beings and
follow ethical principles. They could then be seen as “Homo sapiens 2.0”—enhanced or improved human
beings.
9
  One might also claim here that the ability of moral machines to perform “better” than humans in the
realm of ethics relates to the fact that they do not suffer from moral imperfections—e.g. prejudice, self-
ishness, partiality, weakness of will, and being carried away by emotions—in their ethical reasoning and
decision making.

13
Building Moral Robots: Ethical Pitfalls and Challenges

Dilemma Cases

One fundamental decision involves whether to adopt, for example, a neo-Kantian

approach or a hedonistic utilitarian approach to solving moral issues. The moral
differences between these two rival approaches become apparent when one consid-
ers a famous moral dilemma, the so-called trolley10 case. A tram is out of control
and about to kill five people. The tram can be redirected by reversing the railway
points; however, doing so will cause one worker to be killed by the redirected tram.
Is it morally permissible to cause the death of one worker, by reversing the railway
points, to save the lives of five people (Foot 1967; Thomson 1976)?
According to a Kantian approach, one is not allowed to exploit the worker for the
purpose of saving the five people who would be killed if one does not redirect the
tram. From a utilitarian standpoint, on the other hand, one must consider the conse-
quences of one’s actions and the total amount of utility, i.e. the greatest good for the
greatest number of people, and therefore one must redirect the tram. The two ethi-
cal theories ultimately lead to different solutions, each of which is—from the stand-
point of the particular theory—well justified. That is, if the machine is using a neo-
Kantian approach, it will make a different ethical decision from a machine applying
utilitarian moral theory. Of course, human beings face the same moral dilemmas,
and there is currently no other foundation by which to determine conclusively that
one theory is wrong and the other theory is correct. In other words, moral machines
will face moral dilemmas and other complex cases where no “correct” solutions are
readily available. Human life is morally complex and certainly not computable by a
simple formula. Therefore, a moral machine must be capable of dealing with such
cases in a way that does justice to the depth of the particular case and to morality as
such.
One might object that, at this moment, there is no particular need for machines to
decide such moral dilemmas. But this objection is dubious, given the current devel-
opment of self-driving cars11 and driverless train systems (Allen et al. 2011: 51) that
may face precisely such complex moral decisions. Whether the machine will eventu-
ally be any better than humans in finding the best moral solution in such dilemma
cases remains to be seen.

10
  See Pereira and Saptawijaya (2011) for a detailed description of how their system, ACORDA, uses
prospective logic programming to solve various dilemmas like the trolley case.
11
  See, for example, the interesting programmable “moral machine” developed by Scalable Cooperation
at MIT Media Lab concerning the question “Between whom is the self-driving car deciding?” in a case
where the car’s brakes fail. It is a “platform for gathering a human perspective on moral decisions made
by machine intelligence, such as self-driving cars … where a driverless car must choose the lesser of
two evils, such as killing two passengers or five pedestrians. As an outside observer, you judge which
outcome you think is more acceptable” (www.moral​machi​ne.mit.edu, accessed 2 July 2018). However,
the MIT trial does not involve cars working in concert, but only in isolation or in a way that involves
human–robot interaction. Self-driving cars may eventually be part of a larger system in which machines
will communicate with each other and thereby solve upcoming problems more fully than can occur when
each machine functions in isolation, particularly in the context of automated driving (Borenstein et  al.
2017).

13
 J.-S. Gordon

Machine Bias

An additional complication concerns the application of smart technologies in the

realm of law. Many people have predicted the eventual advent of AI judges who will
judge human beings as well as other robots. Even with their advanced programming
and their potential access to all legal documents available in the world, including
legal commentaries, previous decisions, and sophisticated legal strategies, how such
programs might interact with each other amidst the moves and counter-moves of
legal defense and prosecution teams remains, at least to some extent, an open ques-
tion. Much will depend on how the basic data used by the AI program are prepared
so as to avoid the so-called machine bias in law (see Angwin et al. 2016).
At this time, AI judges remain a speculative concept, but the use of AI programs
in law is already happening. For example, there exist AI programs capable of deter-
mining court decisions by the European Court of Human Rights (Aletras et al. 2016)
as well as the US Supreme Court (Katz et al. 2017). Furthermore, some AI programs
are assisting judges in several US states by helping them to assess the likelihood that
a particular person will commit another offence. This risk assessment is of utmost
importance in decisions whether to either release an offender or suspect or keep the
person in prison. It turns out, however, that this AI risk assessment has a strong bias
against certain ethnicities and has therefore been shown to be unjust (Angwin et al.
2016). Unfortunately, judges in many US states are continuing to base decisions on
these programs. Machine bias is a serious moral issue to which programmers must
pay close attention so as to avoid building machines that disadvantage one group
over others.

Moral Machines: Two Brief Methodological Remarks

As previously mentioned, there exist several perennial methodological problems in

ethics that even moral philosophers have not yet solved. Among these, two persistent
issues, discussed in this section, are of utmost significance in efforts to build moral
machines. First, the problem of justification arises from the fact that there is cur-
rently no ultimate ethical theory widely considered the only correct moral approach.
Second, if pluralism does exist in the field of ethics, one should simply try to avoid
doing what is immoral instead of attempting to find the morally perfect solution in a
given case. Researchers and programmers in computer science, AI and robotics who
seek to build moral machines must have a clear understanding of these methodologi-
cal problems in ethics; otherwise, the project itself will be doomed to failure.

The Problem of Justification

Empirically speaking, one can regularly make an interesting but important observa-
tion, while sitting on ethics committees, with regard to the difference between the
commonly agreed-upon solution, on one hand, and the varying justifications given
by members on the other hand. Usually, the decisions made by ethics committees are

13
Building Moral Robots: Ethical Pitfalls and Challenges

quite uniform, yet the committee members often provide very different justifications
for their decisions (Bayertz 1991). This result suggests that different ethical theo-
ries or perspectives overlap in their view of many moral issues, even though they
apply completely different lines of argumentations. Of course, it could be objected
that ethics committees generally are seeking the best pragmatic decision, whereas
“normal” moral reasoning and decision making are concerned with the right moral
decision, independent of any external factors that might favour a more pragmatic
solution. The goal of ethics committees is hence twofold—namely, to find the best
available and most pragmatic solution in their decision making.
The key point here is that AI researchers must be aware, when programming
moral machines, of the nuance that different ethical theories often provide the same
decisions but for different reasons. To avoid bias in ethics, it would seem worthwhile
to create moral machines that examine moral problems from different ethical per-
spectives, using multiple ethical theories, and then—after having determined and
examined all possible issues—make a well-balanced moral decision. This type of
bias is different from the machine bias problem discussed above, where either (1)
improper data or (2) a defective algorithm guides the machine towards disadvan-
taging particular groups, such as an ethnic group or women. This theoretical bias
really concerns how ethical reasoning and decision making are done by moral phi-
losophers or laypeople. As noted above, the moral solution of some difficult cases,
such as the trolley case, may differ substantially depending on the ethical theory
used. For example, a neo-Kantian approach emphasizes the core values of rational-
ity and autonomy in its ethical reasoning and decision making, whereas a hedonistic,
utilitarian approach favours sentience, i.e. the ability to suffer and to feel pain. As a
result, the two ethical theories may arrive at different moral outcomes based on their
different core values. Therefore, some moral philosophers such as Brody (1988)
and Gordon (2016a) have suggested a more pluralistic approach based on several
key concepts in order to avoid a theoretical bias in decision making. This reasoning
suggests that, to be theoretically unbiased, machine ethics should use a method that
does justice to the complexity of the moral life.

Switching Gears: Avoiding the Immoral

Machine ethics should “focus on the similarities between the various ethical
approaches in order to gain a better understanding of what one should not do. In
other words, one should try to spell out in more detail not how moral agents should
act but, rather, what they should avoid doing” (Gordon 2016b: 290). This approach
has an analogy in the context of moral enhancement, where a similar question—
namely, which ethical approach should be used as a default position to make human
beings more moral—has been raised. One reasonable answer to this question is to
try to agree “on a binding list of things we want to avoid in our moral lives” (Gor-
don 2016b: 290) and to create machines that will act accordingly while, at the same
time, permitting them to make moral decisions within a framework that allows for
different but equally acceptable moral solutions. After all, that seems to be the best
that one can currently hope for in the area of machine ethics.

13
 J.-S. Gordon

In this regard, human morality is somewhat different from machine ethics,

because human beings commonly make moral decisions against the background of
the more foundational question of how to live a good life. Machines, on the other
hand, can be expected to solve the moral problem at hand independently of any
considerations of personal meaning, fulfilment or happiness. Whether the two per-
spectives will eventually yield different results or be virtually in accord with each
other is entirely another matter. One can quite reasonably view human morality as
being enriched and shaped by the assumption that the lives of human beings have a
deeper meaning by virtue of their existential nature (e.g. their desire for a fulfilling,
happy life) than intelligent machines can have. Hence, one may justifiably argue,
human morality goes beyond the motif of simply avoiding the immoral, because
human beings also want to live a good life. At least in the near future, machines will
not be able to raise such existential questions or to arrange their lives accordingly.
Therefore, their moral reasoning and decision making lack this additional existential
perspective.
The ideas discussed herein will not be easy to implement. However, it should
be clear that the quest for the holy grail of an ultimate machine ethics that would
enable machines to make moral decisions—or even the morally best decisions—is
grounded in more general efforts to find an ultimate moral theory that can then,
in turn, also be used by machines. Susan Anderson proposes that machine ethics
might contribute to human moral progress: “I believe that machine ethics research
has the potential to achieve breakthroughs in ethical theory that will lead to uni-
versally accepted ethical principles, and that interacting with ‘ethical’ machines
might inspire us to behave more ethically ourselves” (2011: 524; see also Hall 2011:
523).12 At this time, however, there exists no such ultimate moral theory that can
adequately capture the whole complexity of the moral life of human beings (let
alone non-human entities such as animals, nature, and artificial beings). There-
fore, one should take a more modest approach towards the development of moral
machines and focus on attempting to avoid what is immoral as a starting point, as
well as on using a pluralistic ethical method instead of a single moral theory to solve
moral problems, since many ethical mistakes are the result of either ignorance or the
inability to recognize certain actions as morally problematic.

Conclusions

Moral machines capable of autonomous ethical reasoning and decision making

without any human supervision whatsoever may well emerge in future. However,
recent approaches in machine ethics have demonstrated the need for researchers and
programmers to seek advice from ethicists so that they can avoid rookie mistakes
and better understand deep-seated methodological problems in ethics. If they do not
properly address these problems, their efforts to build adequate moral machines will

12
  The idea that machines might come up with new or unexpressed ethical principles has been advanced
by Anderson and Anderson (2011b: 476–492, esp. 479).

13
Building Moral Robots: Ethical Pitfalls and Challenges

be severely undermined. How to decide what actions are morally correct is one of
the most difficult questions in our lives. Understanding the ethical pitfalls and chal-
lenges involved in these decisions is absolutely necessary to build intelligent, unbi-
ased moral machines.

Acknowledgements  I would like to thank the anonymous reviewers for their valuable comments. This
research is funded by the European Social Fund according to the activity ‘Improvement of researchers’
qualification by implementing world-class R&D projects of Measure No. 09.3.3-LMT-K-712.

References
Abney, K. (2014). Robotics, ethical theory, and metaethics: A guide for the perplexed. In P. Lin, K.
Abney, & G. Bekey (Eds.), Robot ethics: The ethical and social implications of robotics (pp. 35–52).
Cambridge, MA: MIT Press.
Aletras, N., Tsarapatsanis, D., Preoţiuc-Pietro, D., & Lampos, V. (2016). Predicting judicial decisions of
the European Court of Human Rights: A natural language processing perspective. PeerJ Computer
Science, 2(e93), 1–19.
Allen, C., Varner, G., & Zinser, J. (2000). Prolegomena to any future artificial moral agent. Journal of
Experimental & Theoretical Artificial Intelligence, 12(3), 251–261.
Allen, C., Wallach, W., & Smit, I. (2011). Why machine ethics? In M. Anderson & S. Anderson (Eds.),
Machine ethics (pp. 51–61). Cambridge, MA: Cambridge University Press.
Anderson, S. L. (2011a). How machines might help us achieve breakthroughs in ethical theory and
inspire us to behave better. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 524–
5309). Cambridge, MA: Cambridge University Press.
Anderson, S. L. (2011b). Philosophical concerns with machine ethics. In M. Anderson & S. L. Anderson
(Eds.), Machine ethics (pp. 162–167). Cambridge, MA: Cambridge University Press.
Anderson, S. L. (2011c). Machine metaethics. In M. Anderson & S. L. Anderson (Eds.), Machine ethics
(pp. 21–27). Cambridge, MA: Cambridge University Press.
Anderson, M., & Anderson, S. L. (2007). Machine ethics: Creating an ethical intelligent agent. AI Maga-
zine, 28(4), 15–26.
Anderson, M., & Anderson, S. L. (2011a). Machine ethics. Cambridge, MA: Cambridge University Press.
Anderson, M., & Anderson, S. L. (2011b). A prima facie duty approach to machine ethics. In M. Ander-
son & S. L. Anderson (Eds.), Machine ethics (pp. 476–492). Cambridge, MA: Cambridge Univer-
sity Press.
Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine bias. ProPublica. https​://www.propu​
blica​.org/artic​le/machi​ne-bias-risk-asses​sment​s-in-crimi​nal-sente​ncing​. Accessed 8 March 2018.
Bayertz, K. (1991). Praktische Philosophie als angewandte Ethik”. In K. Bayertz (Ed.), Praktische Phi-
losophie: Grundorientierungen angewandter Ethik (pp. 7–47). Hamburg: Rowohlt Taschenbuch
Verlag.
Bekey, G. A. (2014). Current trends in robotics: Technology and ethics. In P. Lin, K. Abney, & G. Bekey
(Eds.), Robot ethics: The ethical and social implications of robotics (pp. 17–34). Cambridge, MA:
MIT Press.
Borenstein, J., Herkert, J. R., & Miller, K. W. (2017). Self-driving cars and engineering ethics: The
need for a system level analysis. Science and Engineering Ethics. https​://doi.org/10.1007/s1194​
8-017-0006-0.
Bringsjord, S., & Taylor, J. (2014). The divine-command approach to robot ethics. In P. Lin, K. Abney, &
G. Bekey (Eds.), Robot ethics: The ethical and social implications of robotics (pp. 85–108). Cam-
bridge, MA: MIT Press.
Brody, B. (1988). Life and death decision making. Oxford: Oxford University Press.
Clouser, K. D., & Gert, B. (1990). A critique of principlism. Journal of Medicine and Philosophy, 15(2),
219–236.
Coeckelbergh, M. (2014). The moral standing of machines: Towards a relational and non-Cartesian moral
hermeneutics. Philosophy & Technology, 27(1), 61–77.
Copeland, J. (1993). Artificial intelligence: A philosophical introduction. Oxford: Wiley.

13
 J.-S. Gordon

Dehghani, M., Forbus, K., Tomai, E., & Klenk, M. (2011). An integrated reasoning approach to moral
decision making. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 422–441). Cam-
bridge, MA: Cambridge University Press.
Dietrich, E. (2011). Homo sapiens 2.0: Building the better robots of our nature. In M. Anderson & S. L.
Anderson (Eds.), Machine ethics (pp. 531–538). Cambridge, MA: Cambridge University Press.
Floridi, L. (2011). On the morality of artificial agents. In M. Anderson & S. L. Anderson (Eds.), Machine
ethics (pp. 184–212). Cambridge: Cambridge University Press.
Floridi, L., & Sanders, J. W. (2004). On the morality of artificial agents. Minds and Machines, 14(3),
349–379.
Foot, P. (1967). The problem of abortion and the doctrine of double effect. Oxford Review, 5, 5–15.
Gips, J. (2011). Towards the ethical robot. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp.
244–253). Cambridge, MA: Cambridge University Press.
Giubilini, A., & Savulescu, J. (2018). The artificial moral advisor. The “Ideal Observer” meets artificial
intelligence. Philosophy & Technology, 31(2), 169–188.
Gordon, J.-S. (2011). Global ethics and principlism. Kennedy Institute of Ethics Journal, 21(3), 251–276.
Gordon, J.-S. (2015). Human rights and cultural identity. Baltic Journal of Law & Politics, 8(2), 112–135.
Gordon, J. S. (2016a). Ethics as a method. In J. Schildmann, J. S. Gordon, & J. Vollmann (Eds.), Clini-
cal ethics consultation: Theories and methods, implementation, evaluation (pp. 11–20). London:
Routledge.
Gordon, J.-S. (2016b). Should moral enhancement be compulsory? Ethical Perspectives, 23(2), 277–305.
Gordon, J.-S. (2018). What do we owe to intelligent robots? AI & Society. https​://doi.org/10.1007/s0014​
6-018-0844-6.
Gordon, J.-S., Rauprich, O., & Vollmann, J. (2011). Applying the four-principle approach. Bioethics,
25(6), 293–300.
Guarini, M. (2006). Particularism and the classification and reclassification of moral cases. IEEE Intel-
ligent Systems, 21(4), 22–28.
Gunkel, D. J., & Bryson, J. (2014). The machine as moral agent and patient. Philosophy & Technology,
27(1), 5–142.
Hall, J. S. (2011). Ethics for self-improving machines. In M. Anderson & S. L. Anderson (Eds.), Machine
ethics (pp. 512–523). Cambridge, MA: Cambridge University Press.
Hughes, J. (2014). Compassionate AI and selfless robots: A Buddhist approach. In P. Lin, K. Abney, & G.
Bekey (Eds.), Robot ethics: The ethical and social implications of robotics (pp. 69–84). Cambridge,
MA: MIT Press.
Johnson, A. M., & Axinn, S. (2014). Acting versus being moral: The limits of technological moral actors.
In Proceedings of the IEEE 2014 international symposium on ethics in engineering, science, and
technology (pp. 30(1)–30(4)). Piscataway, NJ: IEEE Press.
Kant, I., & Paton, H. J. (2009). Groundwork of the metaphysic of morals. New York: Harper Perennial
Modern Classics.
Katz, D. M., Bommarito, M. J., II, & Blackman, J. (2017). A general approach for predicting the behavior
of the Supreme Court of the United States. PLoS ONE, 12(4), 1–18.
Lin, P., Abney, K., & Bekey, G. A. (Eds.). (2014). Robot ethics: The ethical and social implications of
robotics. Intelligent robotics and autonomous agents. Cambridge, MA: MIT Press.
Macklin, R. (1999). Against relativism. Cultural diversity and the search for ethical universals in medi-
cine. New York: Oxford University Press.
McLaren, B. M. (2011). Computational models of ethical reasoning: Challenges, initial steps, and future
directions. In M. Anderson & S. L. Anderson (Eds.), Machine ethics (pp. 297–315). Cambridge,
MA: Cambridge University Press.
Moor, J. H. (2006). The nature, importance, and difficulty of machine ethics. Research Gate, 21(4),
18–21.
Nadeau, J. E. (2006). Only androids can be ethical. In K. Ford & C. Glymour (Eds.), Thinking about
android epistemology (pp. 241–248). Cambridge, MA: MIT Press.
Pereira, L. M., & Saptawijaya, A. (2011). Modeling morality with prospective logic. In M. Anderson & S.
L. Anderson (Eds.), Machine ethics (pp. 398–421). Cambridge, MA: Cambridge University Press.
Picard, R. (1997). Affective computing. Cambridge: The MIT Press.
Pontier, M. A., & Hoorn, J. F. (2012). Toward machines that behave ethically better than humans do. In
N. Miyake, B. Peebles, & R. P. Cooper (Eds.), Proceedings of the 34th international annual confer-
ence of the cognitive science society (pp. 2198–2203). CogSci’12.

13
Building Moral Robots: Ethical Pitfalls and Challenges

Powers, T. (2011). Prospectives for a Kantian machine. In M. Anderson & S. L. Anderson (Eds.),
Machine ethics (pp. 464–475). Cambridge, MA: Cambridge University Press.
Rodogno, R. (2016). Robots and the limits of morality. In M. Norskov (Ed.), Social robots: Boundaries,
potential, challenges. London: Routledge. http://pure.au.dk/porta​l/files​/90856​828/Robot​s_and_the_
Limit​s_of_Moral​ity.pdf. Accessed 12 March 2016.
Ross, W. D. (1930). The right and the good. Oxford: Clarendon Press.
Rzepka, R., & Araki, K. (2005). What statistics could do for ethics? The idea of common sense pro-
cessing based safety valve. In AAAI fall symposium on machine ethics, Technical Report FS-05-06:
85-87.
Sullins, J. P. (2011). When is a robot a moral agent? In M. Anderson & S. L. Anderson (Eds.), Machine
ethics (pp. 151–161). Cambridge, MA: Cambridge University Press.
Thomson, J. J. (1976). Killing, letting die, and the trolley problem. The Monist, 59(2), 204–217.
Torrance, S. (2008). Ethics and consciousness in artificial agents. AI & Society, 22(4), 495–521.
Wallach, W., & Allen, C. (2010). Moral machines: Teaching robots right from wrong. Oxford: Oxford
University Press.
Wallach, W., Franklin, S., & Allen, C. (2010). A conceptual and computational model of moral decision
making in human and artificial agents. Topics in Cognitive Science, 2(3), 454–485.

Publisher’s Note  Springer Nature remains neutral with regard to jurisdictional claims in published
maps and institutional affiliations.

13