PRIO POLICY BRIEF 02 2014

Defining the Scope of

Visiting Address: Hausmanns gate 7
PO Box 9229 Grønland, NO-0134 Oslo, Norway
Peace Research Institute Oslo (PRIO)

Autonomy
Issues for the Campaign to Stop Killer Robots
Twitter: PRIOUpdates
Facebook: PRIO.org
www.prio.org

The Campaign to Stop Killer Robots was launched in April 2013

with the objective of achieving a ban on the development, produc-
tion and deployment of lethal autonomous weapons. In May 2014,
the issue will be discussed by a UN expert meeting under the aus-
pices of the Convention on Conventional Weapons (CCW) in Ge-
neva. At this stage, it is inevitable that there will be much debate
and discussion over the scope and meaning of any future prohibi-
tion. The Campaign is still being shaped, and what will be neces-
sary for its success is that over the next few years a group of states
ISBN: 978-82-7288-540-2 (print)

and governments coalesce around a shared understanding of the

978-82-7288-541-9 (online)

problem and its solutions.

This policy brief1 provides a pragmatic theory of humanitarian dis-
armament and discusses a set of key challenges related to the ques-
tion of scope – that is, what exactly would be covered by a possible
international agreement on lethal autonomous robots. The Cam-
paign must tackle these issues if it is to move from a startup phase
and into activism with sustained political impact. Attention is given
to the Brimstone, a UK anti-tank weapon and one of the most ad-
vanced ‘fire and forget’ missiles in use, whose degree of autonomy
is also a matter of some contention.
Nicholas Marsh Peace Research Institute Oslo (PRIO)
 A Pragmatic Theory of Successful Hu- tial). lethal fully autonomous robot is the deci-
manitarian Disarmament sionmaking capability of the machine – that it
If it is not possible to persuade states of the
is able to navigate around its environment and
necessity of a prohibition, the Campaign will
A humanitarian disarmament treaty is de- discriminate between targets.
most likely not find support from the states
signed to reduce human suffering rather than
that will be required to form a coalition and The above definitions used by the Campaign,
to manage the affairs of great powers. In the
negotiate a successful treaty. Noel Sharkey and Christof Heyns coalesce
experience of this author, two things are re-
around a definition of machine which is capa-
quired for a successful campaign for a hu- Defining the Threshold of Autonomy ble of lethal force, and can autonomously
manitarian disarmament treaty.
move around the battlefield and select a tar-
One of the most contentious issues is likely to
 First, the campaign needs to stigmatize the get. However, this is not a universally shared
concern the threshold at which a weapon
type of weapon concerned by showing that it understanding of the essential characteristics
system is deemed to be ‘fully autonomous’.
causes an unacceptable level of harm. Doing of a lethal autonomous weapon. The UK
Discussions on this issue will necessarily be
so involves highlighting aspects such as the Ministry of Defence defines an ‘autonomous
both technical and intensely pragmatic. The
propensity of the weapon to cause death and system’ in the following manner:
threshold that is set would determine which
injury among civilians. A key element of the
systems are banned and which are allowed to An autonomous system is capable of understand-
campaigns to ban anti-personnel land mines
continue in operation. ing higher level intent and direction. From this
or cluster munitions was the message that
understanding and its perception of its environ-
these weapons remain lethal long after the Setting the threshold of autonomy is going to
ment, such a system is able to take appropriate
fighting has ended, and so pose a deadly risk involve significant debate because machine
action to bring about a desired state.... Autono-
to civilians. decisionmaking exists on a continuum. There
mous systems will, in effect, be self-aware and
is a grey area in which systems can be said to
 The second task is to persuade govern- their response to inputs indistinguishable from, or
be partly autonomous, and there is no consen-
ments that they do not need the weapon in even superior to, that of a manned aircraft. As
sus as to exactly what counts as a fully auton-
question. States are being asked to voluntarily such, they must be capable of achieving the same
omous machine. A key task for the Campaign
give up the use of a weapon, and usually level of situational understanding as a human.7
will be to create consensus on this issue
prohibitive treaties also cover development,
among both NGOs and the states that would Sharkey criticizes the UK Ministry of Defence
production, stockpiling and trade. For a gov-
have to negotiate and then implement a ban. definition8 on the grounds that not only does
ernment to abjure a weapon it needs to first
the technology to enable human-like cogni-
accept that the weapon is not militarily essen- Scholar Noel Sharkey has defined a lethal
tion not exist, but it is very unlikely to exist in
tial. fully autonomous robot as one that operates
the foreseeable future. The point for Sharkey
in an open and unstructured environment;
A key element of the campaigns to ban cluster is that full autonomy occurs when machines
receives information from sensors; and pro-
munitions and landmines involved spreading have sole responsibility for selecting targets
cesses the information in order to move,
the message that these weapons were not and using weapons – which could happen in
select targets and fire – all without human
essential – that alternatives existed, the role machines with decisionmaking capabilities far
supervision.3 The Human Rights Watch re-
envisioned for them no longer existed, or that below levels of human cognition. Using the
port Losing Humanity: The Case Against
their use is so abhorrent as to be counterpro- above definition, the UK Ministry of Defence
Killer Robots similarly defines a fully auton-
ductive. A counter example is the case of could employ machines with the ability to
omous robot as being one that is ‘capable of
nuclear weapons. No sensible person doubts move, locate and select targets, and then fire
selecting targets and delivering force without
their devastating effects upon civilians. None- at them, without defining them as ‘autono-
any human input or interaction’.4 Later, the
theless, nuclear weapons remain in state mous’ if they lacked a particular aspect of
report refers to a fully autonomous weapon
arsenals because the nuclear states still view human cognition. The next section examines
that would identify targets and trigger itself,
them as being necessary. Nuclear disarma- how these definitional disputes apply to spe-
and also to Sharkey’s definition noted above. 5
ment will therefore not occur until the gov- cific weapon systems.
ernments of those states are persuaded that Likewise, Christof Heyns, the UN Special
such weapons cannot be used under any Rapporteur on extrajudicial, summary or The Case of ‘Fire and Forget’ Weap-
circumstances; that they are considered mili- arbitrary executions, considered the technolo- ons: The Brimstone
tarily useless.2 gy and defined lethal autonomous robots as
This section highlights the ambiguous status
being ‘weapon systems that, once activated,
The Campaign to Stop Killer Robots needs to of a particular weapon, the Brimstone anti-
can select and engage targets without further
strike a balance between a prohibition with a tank missile, one of a class of ‘fire and forget’
intervention by a human operator. The im-
wide ranging scope and a pragmatic accom- weapons that includes missiles with autono-
portant element is that the robot has an au-
modation of the interests of likely State sup- mous capabilities.
tonomous “choice” regarding selection of a
porters of the Campaign (who need to be
target and the use of lethal force’.6 The Brimstone is a ‘fire and forget’ weapon
reassured that they won’t have to give up
system – one of a range of weapons that are
something they perceive to be militarily essen- The emphasis in all these definitions of a
designed to be used precisely as that term

PRIO POLICY BRIEF 02 2014 www.prio.org

suggests. Most are missiles that guide them- its ability to autonomously select targets was relevant to the current debate prompted by the
selves to their target once launched, allowing ill-suited to contemporary operations – espe- Campaign to Stop Killer Robots. Five areas of
the firer to concentrate on other tasks, such as cially Afghanistan. There, because of the particular concern are presented below:
escaping enemy retaliation. Some simply fly conflict’s complex nature, rules of engage-
 First, at the technical level, is the detailed
to preset coordinates, while others actively ment required that a human monitor the
consideration of exactly what counts as lethal
look for targets. engagement right up until impact of the mis-
and fully autonomous. The status of ‘fire and
sile. A ‘fire and forget’ missile was inappro-
The Brimstone, a UK-built anti-tank weapon, forget’ weapons has been included in media
priate in such an environment.12 In response,
is currently one of the most advanced ‘fire and articles and commentary on the issue of au-
the UK Ministry of Defence commissioned an
forget’ missiles. Development started in the tonomous lethal weapons.16 This policy brief
urgent update so that the missile could also be
1990s, and the system has been in service argues that the Brimstone and other weapons
guided with a laser seeker that would guide
since 2005. A description of it provided by the cross the definitional threshold provided by
the missile to a target directly selected by a
UK’s Royal Air Force states that: the Campaign and other commentators such
human operator. The ability of the Brimstone
as Noel Sharkey and Christof Heyns.
Brimstone is a fully autonomous, fire-and-forget, to autonomously select targets with its radar,
anti-armour weapon, effective against all known however, has been used. For example, the first  Second, a narrow definition of autonomy –
and projected armoured threats.... During the use of the autonomous radar seeker was re- such as the one used by the UK Ministry of
search phase of the engagement, Brimstone’s ported on 16 September 2011: over Libya, a Defence, which specifies ‘human like capabili-
[radar] seeker searches for targets in its path, salvo of 24 missiles using radar guidance was ties’ – would exclude not only all current
comparing them to a known target signature in used to destroy seven or eight vehicles travel- technology, but likely all machine deci-
its memory. The missile automatically rejects ling in a convoy.13 sionmaking in existence during the foreseea-
returns which do not match (such as cars, buses, ble future. One potential hurdle for the Cam-
The Brimstone is not unique. There are tens
buildings) and continues searching and compar- paign is likely to be states that support the
of different types of ‘fire and forget’ missiles
ing until it identifies a valid target. The missiles idea of a ban, but set the threshold for auton-
in existence. Many just have the capability to
can be programmed not to search for targets until omy so high that it will not affect any of the
fly to preset coordinates and thus do not select
they reach a given point, allowing them to safely robot systems they wish to deploy.
targets without human intervention. Never-
overfly friendly forces, or only to accept targets in
theless, the Brimstone is not the only missile  Third, a governmental preference for an
a designated box area, thus avoiding collateral
to have an autonomous target-selection capa- excessively high threshold of autonomy raises
damage.9
bility. In particular, some missiles used to a paradox. A key motivation for the Campaign
attack shipping actively search and identify is that robots cannot make the ethical and
These capabilities appear to place the Brim-
targets. The missiles frequently have ranges of contextual assessments that humans can.
stone within the above-mentioned definition
over 100 kilometres, and thus enemy ships However, a robot with a decisionmaking
of an autonomous lethal robot as defined by
will have moved during the period between capability as advanced as human cognition
the Campaign to Stop Killer Robots, as well as
launch and the missile reaching the target could presumably make such judgements.
by Noel Sharkey and Christof Heyns. It would
area.
not fall squarely within the UK Ministry of  Fourth, it is one thing for the Campaign to
Defence’s definition, however, as while the There is not sufficient space in this brief to attempt to preemptively stop the development
Brimstone can distinguish between potential examine in detail the exact nature of the tech- of lethal autonomous weapons that do not
targets, it is unable to make human-like as- nology. However, I will highlight two claims currently exist: it is hard to argue that a weap-
sessments of intent. The UK Ministry of made by the manufacturers of such weapons on is military essential when it has not yet
Defence position document on autonomous that indicate the development of autonomous even been developed. It will be much more
systems, however, does mention that the target selection. The Norwegian Joint Strike difficult, though not impossible, to get states
Brimstone is an example of a weapon that Missile is advertised by its manufacturer the to give up arms that are already deployed. At
‘probably’ achieves a degree of autonomous Kongsberg Group as having ‘Autonomous the very least, if the Brimstone and other ‘fire
operation in certain environments.10 The Target Recognition with identification of and forget’ missiles are assumed to fall under
specific question of whether the Brimstone is targets to ship class level’ and also employs a the definition used by the Campaign, then
a fully autonomous weapon is also considered library of potential targets.14 Similarly, the considerable effort may need to be expended
briefly in an article by the UK Royal Aeronau- company MBDA advertises its MM40 BLOCK on explaining why they are not to be consid-
tical Society’s Air Power Group. This reiter- 3 Exocet missile by stating ‘terminal guidance ered militarily essential. For pragmatic rea-
ates that the Brimstone does not meet the UK relies on a sophisticated J-band active seeker sons, it might be possible to negotiate that the
armed forces definition of an autonomous to discriminate and select targets at sea’.15 scope of an international ban should exclude
system, but does not comment on the wider certain systems even if they cross the thresh-
debate.11 Issues for Campaigners and Policy- old of full autonomy. But, even so, creating
makers such a loophole would be a difficult decision.
The operational use of the Brimstone actually
highlights the very issues raised by the Cam- The Brimstone and other ‘fire and forget’ Finally, the focus of the Campaign to Stop
paign. Originally designed for the Cold War, weapons with similar capabilities are clearly Killer Robots has been very much on weapons
platforms – air- or ground-based robots such
as unmanned aerial vehicles – that if fully 11
29 January 2014. The author thanks the other mem- Aerospace (2013) ‘The ‘drone’ debate the ethics of
autonomous could decide to fire weapons. armed unmanned aircraft’. November issue, pp. 14-19.
bers of the panel – Christian Holmboe Ruge, Alexan-
The reality of current autonomous lethal URL: http://aerosociety.com/Assets/Docs/ Publica-
der Harang and Simon O’Connor – and members of
systems is much more prosaic. The Brim- tions/The_Ethics_of_Armed_UAS.pdf (accessed 4
the audience for their insightful comments. Kristin
stone and other ‘fire and forget’ missiles do April 2014).
Bergtora Sandvik provided very astute feedback on a
not look like robots as popularly conceived, 12
draft. The author also appreciates the assistance See Markowitz, Mike and John D. Gresham (2012)
even if they meet the definition provided by
provided by Maral Mirshahi in editing the brief prior Dual-Mode Brimstone Missile Proves Itself in Com-
the Campaign, Noel Sharkey and Christof
to publication. bat. Defense Media Network, 26 April. URL:
Heyns. This mirrors the general development
http://www.defensemedianetwork.com/stories/dual-
of automation. In decades gone by, science- 2
Lewis, Jeffrey (2013) Poison Control – Why Can’t
mode-brimstone-missile-proves-itself-in-combat/
fiction authors conceived of intelligent hu- We Get Rid of Nukes the Same Way We Got Rid of
(accessed 2 February 2014).
manoid robots that would mimic the activities Chemical Weapons? Foreign Policy, 14 October. URL:
of humans. In fact, technology has developed http://www.foreignpolicy.com/articles/2013/ 13
UK Ministry of Defence (2011) RAF Conducts
very differently. For example, instead of hav- 10/14/poison_control_nuclear_chemical_weapons Precision Strikes over Libya. Press Release, 16 Sep-
ing one general-purpose robot that would take (accessed 27 March 2014). tember. URL: https://www.gov.uk/government/news/
over people’s everyday tasks, there are now 3
raf-conducts-precision-strikes-over-libya (accessed 2
numerous examples of machines that use Sharkey, Noel (2012) Automating Warfare: Lessons
February 2014).
Learned from the Drones. Journal of Law, Information
autonomous decisionmaking to fulfil specific 14
& Science 21(2): p. 2. Kongsberg Gruppen (2013) Kongsberg and Boeing
limited tasks: for example, a smartphone that
Complete Joint Strike Missile (JSM) Check on F/A-18
accepts spoken instructions and recommends 4
Human Rights Watch (2012) Losing Humanity: The
Super Hornet. Press Release, 29 November. URL:
music; robots that will clean a room without Case Against Killer Robots, p.2.
http://www.kongsberg.com/en/kog/news/2013/novem
direct human control; and cars equipped with
5
Ibid, p.6. ber/kongsberg-and-boeing-complete-joint-strike-
sensors that detect an incoming cyclist or
missile/ (accessed 2 February 2014).
pedestrian and automatically brake to prevent 6
Heyns, Christof (2013) Report of the Special Rappor-
a collision. The battlefield is following a simi- teur on Extrajudicial, Summary or Arbitrary Executions.
15
MBDA Missile Systems (2014) EXOCET MM40 –
lar trajectory. Instead of Terminator-like hu- A/HRC/23/47. New York: United Nations, pp. 6–7. New Generation Long Range Anti-Ship Missile Sys-
manoid machines, the lethal autonomous tem. URL: http://www.mbda-
7
robots of today, and most likely the future, are UK Ministry of Defence (2011) The UK Approach to systems.com/mediagallery /files/exocet_mm40_ds.pdf
those in which sensors and artificial deci- Unmanned Aircraft Systems, pp. 2-2 to 2-3. (accessed 5 March 2014).
sionmaking are applied to a specific and lim- 8
Sharkey, Noel (2012) Automating Warfare: Lessons 16
ited role. The challenge for the Campaign will Wired (2008) Robot Wars Continues. 31 March.
Learned from the Drones. Journal of Law, Information URL: http://www.wired.com/dangerroom/
be to capture the public imagination in order & Science 21(2): pp. 2-3.
to build a sufficiently large coalition able to 2008/03/the-robot-war-c/ (accessed 5 March 2014).
9
prohibit such a heterogeneous array of lethal Handy, Brian (ed.) (2007) Royal Air Force Aircraft &
autonomous weapons. Weapons. DCC(RAF) Publications, p. 87. URL:
http://www.raf.mod.uk/rafcms/mediafiles/0186cc2a_1
143_ec82_2ef2bffff37857da.pdf (accessed 5 March
2014).
Notes
10
UK Ministry of Defence (2011) Joint Doctrine 2/11:
1
This PRIO Policy Brief is based upon remarks made The UK Approach to Unmanned Aircraft Systems, p. 5-4.
by Nicholas Marsh at a seminar on autonomous lethal
URL: http://dronewarsuk.files.wordpress.com/
robots organized by the Norwegian Centre for 2011/04/uk-approach-to-uav.pdf (accessed 27 March
Humanitarian Studies, which took place at PRIO on 2014)

THE AUTHOR THE PROJECT PRIO

Nicholas Marsh is a Researcher at PRIO and This policy brief is a product of the Critical The Peace Research Institute Oslo (PRIO) is a
since 2001 has worked there on the trade, Humanitarian Technology Project. The pro- non-profit peace research institute (estab-
production, use and governance of weapons. ject examines the influence of new technolo- lished in 1959) whose overarching purpose is
His current research interests include the use gies upon humanitarian action, and aims to to conduct research on the conditions for
of weapons by non-state parties involved in inform the development of a Norwegian poli- peaceful relations between states, groups and
civil war, and the international trade in small cy on humanitarian technology. For more people. The institute is independent, interna-
arms and light weapons. Email: [email protected] information about the project, please visit: tional and interdisciplinary, and explores is-
www.humanitarianstudies.no. sues related to all facets of peace and conflict.

PRIO POLICY BRIEF 02 2014 www.prio.org