AI & SOCIETY (2021) 36:1263–1280

https://doi.org/10.1007/s00146-020-01097-6

ORIGINAL ARTICLE

The Nooscope manifested: AI as instrument of knowledge extractivism

Matteo Pasquinelli1 · Vladan Joler2

Received: 27 March 2020 / Accepted: 14 October 2020 / Published online: 21 November 2020
© The Author(s) 2020

Abstract
Some enlightenment regarding the project to mechanise reason. The assembly line of machine learning: data, algorithm,
model. The training dataset: the social origins of machine intelligence. The history of AI as the automation of perception.
The learning algorithm: compressing the world into a statistical model. All models are wrong, but some are useful. World
to vector: the society of classification and prediction bots. Faults of a statistical instrument: the undetection of the new.
Adversarial intelligence vs. statistical intelligence: labour in the age of AI.

Keywords Nooscope · Political economy · Mechanised knowledge · Information compression · Ethical machine learning

1 Some enlightenment The purpose of the Nooscope map is to secularize AI

regarding the project to mechanise from the ideological status of ‘intelligent machine’ to one of
reason knowledge instruments. Rather than evoking legends of alien
cognition, it is more reasonable to consider machine learn-
The Nooscope is a cartography of the limits of artificial ing as an instrument of knowledge magnification that helps
intelligence, intended as a provocation to both computer to perceive features, patterns, and correlations through vast
science and the humanities. Any map is a partial perspec- spaces of data beyond human reach. In the history of science
tive, a way to provoke debate. Similarly, this map is a mani- and technology, this is no news; it has already been pursued
festo—of AI dissidents. Its main purpose is to challenge the by optical instruments throughout the histories of astronomy
mystifications of artificial intelligence. First, as a technical and medicine.3 In the tradition of science, machine learning
definition of intelligence and, second, as a political form that is just a Nooscope, an instrument to see and navigate the
would be autonomous from society and the human.1 In the space of knowledge (from the Greek skopein ‘to examine,
expression ‘artificial intelligence’, the adjective ‘artificial’ look’ and noos ‘knowledge’).
carries the myth of the technology’s autonomy; it hints to Borrowing the idea from Gottfried Wilhelm Leibniz, the
caricatural ‘alien minds’ that self-reproduce in silico but, Nooscope diagram applies the analogy of optical media to
actually, mystifies two processes of proper alienation; the the structure of all machine learning apparatuses. Discuss-
growing geopolitical autonomy of hi-tech companies and ing the power of his calculus ratiocinator and ‘characteristic
the invisibilization of workers’ autonomy worldwide. The numbers’ (the idea to design a numerical universal language
modern project to mechanise human reason has clearly to codify and solve all the problems of human reasoning),
mutated, in the twenty first century, into a corporate regime Leibniz made an analogy with instruments of visual mag-
of knowledge extractivism and epistemic colonialism.2 This nification such as the microscope and telescope. He wrote:
is unsurprising, since machine learning algorithms are the ‘Once the characteristic numbers are established for most
most powerful algorithms for information compression. concepts, mankind will then possess a new instrument which
will enhance the capabilities of the mind to a far greater
1
* Matteo Pasquinelli On the autonomy of technology see: Winner (2001).
mpasquinelli@hfg‑karlsruhe.de 2
For the colonial extensions of the operations of logistics, algo-
1
rithms and finance see: Mezzadra and Neilson (2019). On the epis-
Media Philosophy Department, Karlsruhe University of Arts temic colonialism of AI see: Pasquinelli (2019b).
and Design, Karlsruhe, Germany 3
Digital humanities term a similar technique distant reading, which
2
New Media Department, Academy of Arts, University has gradually involved data analytics and machine learning in literary
of Novi Sad, Novi Sad, Serbia and art history. See: Moretti (2013).

13
Vol.:(0123456789)
1264 AI & SOCIETY (2021) 36:1263–1280

extent than optical instruments strengthen the eyes, and On the invention of metaphors as instrument of knowl-
will supersede the microscope and telescope to the same edge magnification.
extent that reason is superior to eyesight’ (Leibniz 1677, Emanuele Tesauro, Il canocchiale aristotelico [The Aris-
p. 23). Although the purpose of this text is not to reiterate totelian Telescope], frontispiece of the 1670 edition, Turin.
the opposition between quantitative and qualitative cultures,
Leibniz’s credo need not be followed. Controversies cannot
be conclusively computed. Machine learning is not the ulti- 2 The assembly line of machine learning:
mate form of intelligence. data, algorithm, model
Instruments of measurement and perception always come
with inbuilt aberrations. In the same way that the lenses of The history of AI is a history of experiments, machine fail-
microscopes and telescopes are never perfectly curvilinear ures, academic controversies, epic rivalries around military
and smooth, the logical lenses of machine learning embody funding, popularly known as ‘winters of AI.’4 Although
faults and biases. To understand machine learning and reg- corporate AI today describes its power with the language
ister its impact on society is to study the degree by which of ‘black magic’ and ‘superhuman cognition’, current tech-
social data are diffracted and distorted by these lenses. This niques are still at the experimental stage (Campolo and
is generally known as the debate on bias in AI, but the politi- Crawford 2020). AI is now at the same stage as when the
cal implications of the logical form of machine learning are steam engine was invented, before the laws of thermodynam-
deeper. Machine learning is not bringing a new dark age but ics necessary to explain and control its inner workings, had
one of diffracted rationality, in which, as it will be shown, been discovered. Similarly, today, there are efficient neural
an episteme of causation is replaced by one of automated networks for image recognition, but there is no theory of
correlations. More in general, AI is a new regime of truth, learning to explain why they work so well and how they
scientific proof, social normativity and rationality, which fail so badly. Like any invention, the paradigm of machine
often does take the shape of a statistical hallucination. This learning consolidated slowly, in this case through the last
diagram manifesto is another way to say that AI, the king of half-century. A master algorithm has not appeared overnight.
computation (patriarchal fantasy of mechanised knowledge, Rather, there has been a gradual construction of a method
‘master algorithm’ and alpha machine) is naked. Here, we
are peeping into its black box.

4
For a concise history of AI see: Cardon et al. (2018b).

13
AI & SOCIETY (2021) 36:1263–1280 1265

of computation that still has to find a common language. Paraphrasing Michelle Alexander, Ruha Benjamin has called
Manuals of machine learning for students, for instance, do it the New Jim Code: ‘the employment of new technologies
not yet share a common terminology. How to sketch, then, that reflect and reproduce existing inequalities but that are
a critical grammar of machine learning that may be concise promoted and perceived as more objective or progressive
and accessible, without playing into the paranoid game of than the discriminatory systems of a previous era’ (Ben-
defining General Intelligence? jamin 2019, p. 5). Dataset bias is introduced through the
As an instrument of knowledge, machine learning is com- preparation of training data by human operators. The most
posed of an object to be observed (training dataset), an instru- delicate part of the process is data labelling, in which old
ment of observation (learning algorithm) and a final repre- and conservative taxonomies can cause a distorted view of
sentation (statistical model). The assemblage of these three the world, misrepresenting social diversities and exacerbat-
elements is proposed here as a spurious and baroque diagram ing social hierarchies (see below the case of ImageNet).
of machine learning, extravagantly termed Nooscope.5 Stay- Algorithmic bias (also known as machine bias, statistical
ing with the analogy of optical media, the information flow bias or model bias, to which the Nooscope diagram gives par-
of machine learning is like a light beam that is projected by ticular attention) is the further amplification of historical bias
the training data, compressed by the algorithm and diffracted and dataset bias by machine learning algorithms. The problem
towards the world by the lens of the statistical model. of bias has mostly originated from the fact that machine learn-
The Nooscope diagram aims to illustrate two sides of ing algorithms are among the most efficient for information
machine learning at the same time: how it works and how compression, which engenders issues of information resolu-
it fails—enumerating its main components, as well as the tion, diffraction and loss.9 Since ancient times, algorithms have
broad spectrum of errors, limitations, approximations, biases, been procedures of an economic nature, designed to achieve
faults, fallacies and vulnerabilities that are native to its para- a result in the shortest number of steps consuming the least
digm.6 This double operation stresses that AI is not a mon- amount of resources: space, time, energy and labour (Pas-
olithic paradigm of rationality but a spurious architecture quinelli (forthcoming) The eye of the master. Verso, London).
made of adapting techniques and tricks. Besides, the limits The arms race of AI companies is, still today, concerned with
of AI are not simply technical but are imbricated with human finding the simplest and fastest algorithms with which to capi-
bias. In the Nooscope diagram, the essential components of talise data. If information compression produces the maximum
machine learning are represented at the centre, human biases rate of profit in corporate AI, from the societal point of view,
and interventions on the left, and technical biases and lim- it produces discrimination and the loss of cultural diversity.
itations on the right. Optical lenses symbolize biases and While the social consequences of AI are popularly under-
approximations representing the compression and distortion stood under the issue of bias, the common understanding of
of the information flow. The total bias of machine learning technical limitations is known as the black box problem. The
is represented by the central lens of the statistical model black box effect is an actual issue of deep neural networks
through which the perception of the world is diffracted. (which filter information so much that their chain of reason-
The limitations of AI are generally perceived today thanks ing cannot be reversed) but has become a generic pretext
to the discourse on bias—the amplification of gender, race, for the opinion that AI systems are not just inscrutable and
ability, and class discrimination by algorithms. In machine opaque, but even ‘alien’ and out of control.10 The black box
learning, it is necessary to distinguish between historical effect is part of the nature of any experimental machine at
bias, dataset bias, and algorithm bias, all of which occur at the early stage of development (it has already been noticed
different stages of the information flow.7 Historical bias (or that the functioning of the steam engine remained a mys-
world bias) is already apparent in society before technologi- tery for some time, even after having been successfully
cal intervention. Nonetheless, the naturalisation of such bias, tested). The actual problem is the black box rhetoric, which
that is the silent integration of inequality into an apparently is closely tied to conspiracy theory sentiments in which AI
neutral technology is by itself harmful (Eubanks 2018).8 is an occult power that cannot be studied, known, or politi-
cally controlled.

5
The use of the visual analogy is also intended to record the fad-
8
ing distinction between image and logic, representation and infer- See also: Crawford (2017).
ence, in the technical composition of AI. The statistical models of 9
Computer scientists argue that AI belongs to a subfield of signal
machine learning are operative representations (in the sense of Harun processing, that is data compression.
Farocki’s operative images). 10
6
Projects such as Explainable Artificial Intelligence, Interpretable
For a systematic study of the logical limitations of machine learn- Deep Learning and Heatmapping among others have demonstrated
ing see: Malik (2002). that breaking into the ‘black box’ of machine learning is possible.
7
For a more detailed list of AI biases see: Guttag and Suresh (2019) Nevertheless, the full interpretability and explicability of machine
and Galstyan et al. (2019). learning statistical models remains a myth. See: Lipton (2016).

13
1266 AI & SOCIETY (2021) 36:1263–1280

3 The training dataset: the social origins The training dataset is a cultural construct, not just a tech-
of machine intelligence nical one. It usually comprises input data that are associated
with ideal output data, such as pictures with their descrip-
Mass digitalisation, which expanded with the Internet in tions, also called labels or metadata.11 The canonical exam-
the 1990s and escalated with datacentres in the 2000s, has ple would be a museum collection and its archive, in which
made available vast resources of data that, for the first time artworks are organised by metadata such as author, year,
in history, are free and unregulated. A regime of knowledge medium, etc. The semiotic process of assigning a name or
extractivism (then known as Big Data) gradually employed a category to a picture is never impartial; this action leaves
efficient algorithms to extract ‘intelligence’ from these open another deep human imprint on the final result of machine
sources of data, mainly for the purpose of predicting con- cognition. A training dataset for machine learning is usu-
sumer behaviours and selling ads. The knowledge economy ally composed through the following steps: (1) production:
morphed into a novel form of capitalism, called cognitive labour or phenomena that produce information; (2) capture:
capitalism and then surveillance capitalism, by different encoding of information into a data format by an instrument;
authors (Corsani et al. 2004; Zuboff 2019). It was the Inter- (3) formatting: organisation of data into a dataset; (4) label-
net information overflow, vast datacentres, faster micropro- ling: in supervised learning, the classification of data into
cessors and algorithms for data compression that laid the categories (metadata).
groundwork for the rise of AI monopolies in the twenty first Machine intelligence is trained on vast datasets that are
century. accumulated in ways neither technically neutral nor socially
What kind of cultural and technical object is the dataset impartial. Raw data do not exist, as it is dependent on human
that constitutes the source of AI? The quality of training labour, personal data, and social behaviours that accrue over
data is the most important factor affecting the so-called long periods, through extended networks and controver-
‘intelligence’ that machine learning algorithms extract. sial taxonomies.12 The main training datasets for machine
There is an important perspective to take into account, to learning (NMIST, ImageNet, Labelled Faces in the Wild,
understand AI as a Nooscope. Data are the first source of etc.) originated in corporations, universities, and military
value and intelligence. Algorithms are second; they are the agencies of the Global North. But taking a more careful
machines that compute such value and intelligence into a look, one discovers a profound division of labour that inner-
model. However, training data are never raw, independent vates into the Global South via crowdsourcing platforms
and unbiased (they are already themselves ‘algorithmic’) that are used to edit and validate data.13 The parable of the
(Gitelman 2013). The carving, formatting and editing of ImageNet dataset exemplifies the troubles of many AI data-
training datasets are a laborious and delicate undertaking, sets. ImageNet is a training dataset for Deep Learning that
which is probably more significant for the final results than has become the de facto benchmark for image recognition
the technical parameters that control the learning algorithm. algorithms: indeed, the Deep Learning revolution started in
The act of selecting one data source rather than another is 2012 when Alex Krizhevsky, Ilya Sutskever and Geoffrey
the profound mark of human intervention into the domain Hinton won the annual ImageNet challenge with the convo-
of the ‘artificial’ minds. lutional neural network AlexNet.14 ImageNet was initiated
by computer scientist Fei-Fei Li back in 2006.15 Fei-Fei Li
had three intuitions to build a reliable dataset for image rec-
ognition. First, to download millions of free images from
web services such as Flickr and Google. Second, to adopt
the computational taxonomy WordNet for image labels.16

11
In supervised learning. Also self-supervised learning maintains
forms of human intervention.
12
On taxonomy as a form of knowledge and power see: Foucault
(2005).
13
Such as Amazon Mechanical Turk, cynically termed ‘artificial
artificial intelligence’ by Jeff Bezos. See: Pontin (2007).
14
Although the convolutional architecture dates back to Yann
LeCun’s work in the late 1980s, Deep Learning starts with this paper:
Krizhevsky et al. (2017).
15
For an accessible (yet not very critical) account of the ImageNet
development see: Mitchell (2019).
16
WordNet is ‘a lexical database of semantic relations between
words’ which was initiated by George Armitage at Princeton Univer-
sity in 1985. It provides a strict tree-like structure of definitions.

13
AI & SOCIETY (2021) 36:1263–1280 1267

Third, to outsource the work of labelling millions of images Combinatorial patterns and Kufic scripts, Topkapi scroll,
via the crowdsourcing platform Amazon Mechanical Turk. ca. 1500, Iran.
At the end of the day (and of the assembly line), anonymous
workers from all over the planet were paid few cents per task
to label hundreds of pictures per minute according to the 4 The history of AI as the automation
WordNet taxonomy: their labour resulted in the engineer- of perception
ing of a controversial cultural construct. AI scholars Kate
Crawford and artist Trevor Paglen have investigated and The need to demystify AI (at least from the technical point
disclosed the sedimentation of racist and sexist categories of view) is understood in the corporate world too. Head of
in ImageNet taxonomy: see the legitimation of the category Facebook AI and godfather of convolutional neural net-
‘failure, loser, nonstarter, unsuccessful person’ for a hundred works Yann LeCun reiterates that current AI systems are
arbitrary pictures of people (Crawford and Paglen 2019). not sophisticated versions of cognition, but rather, of percep-
The voracious data extractivism of AI has caused an tion. Similarly, the Nooscope diagram exposes the skeleton
unforeseeable backlash on digital culture: in the early 2000s, of the AI black box and shows that AI is not a thinking
Lawrence Lessig could not predict that the large repository of automaton but an algorithm that performs pattern recogni-
online images credited by Creative Commons licenses would tion. The notion of pattern recognition contains issues that
a decade later become an unregulated resource for face rec- must be elaborated upon. What is a pattern, by the way?
ognition surveillance technologies. In similar ways, personal Is a pattern uniquely a visual entity? What does it mean to
data are continually incorporated without transparency into read social behaviours as patterns? Is pattern recognition
privatised datasets for machine learning. In 2019 artist and AI an exhaustive definition of intelligence? Most likely not. To
researcher, Adam Harvey for the first time disclosed the non- clarify these issues, it would be good to undertake a brief
consensual use of personal photos in training datasets for face archaeology of AI.
recognition. Harvey’s disclosure caused Stanford University, The archetype machine for pattern recognition is Frank
Duke University and Microsoft to withdraw their datasets Rosenblatt’s Perceptron. Invented in 1957 at Cornell Aer-
amidst a major privacy infringement scandal (Harvey 2019; onautical Laboratory in Buffalo, New York, its name is a
Murgia 2019). Online training datasets trigger issues of data shorthand for ‘Perceiving and Recognizing Automaton’
sovereignty and civil rights that traditional institutions are (Rosenblatt 1957). Given a visual matrix of 20 × 20 photo-
slow to counteract (see the European General Data Protection receptors, the Perceptron can learn how to recognise simple
Regulation).17 If 2012 was the year in which the Deep Learn- letters. A visual pattern is recorded as an impression on a
ing revolution began, 2019 was the year in which its sources network of artificial neurons that are firing up in concert
were discovered to be vulnerable and corrupted. with the repetition of similar images and activating one

17
The GDPR data privacy regulation that was passed by the Euro-
pean Parliament in May 2018 is, however, an improvement compared
to the regulation that is missing in the United States.

13
1268 AI & SOCIETY (2021) 36:1263–1280

single output neuron. The output neuron fires 1 = true, if a 5 The learning algorithm: compressing
given image is recognised, or 0 = false, if a given image is the world into a statistical model
not recognised.
The automation of perception, as a visual montage of The algorithms of AI are often evoked as alchemic formu-
pixels along a computational assembly line, was originally las, capable of distilling ‘alien’ forms of intelligence. But
implicit McCulloch and Pitt’s concept of artificial neural what do the algorithms of machine learning really do? Few
networks (McCulloch and Pitts 1947). Once the algorithm people, including the followers of artificial general intel-
for visual pattern recognition survived the ‘winter of AI’ ligence (AGI), bother to ask this question. Algorithm is the
and proved efficient in the late 2000s, it was applied also to name of a process, whereby a machine performs a calcula-
non-visual datasets, properly inaugurating the age of Deep tion. The product of such machine processes is a statistical
Learning (the application of pattern recognition techniques model (more accurately termed an ‘algorithmic statistical
to all kinds of data, not just visual). Today, in the case of model’). In the developer community, the term ‘algorithm’
self-driving cars, the patterns that need to be recognised are is increasingly replaced with ‘model.’ This terminologi-
objects in road scenarios. In the case of automatic transla- cal confusion arises from the fact that the statistical model
tion, the patterns that need to be recognised are the most does not exist separately from the algorithm: somehow, the
common sequences of words across bilingual texts. Regard- statistical model exists inside the algorithm under the form
less of their complexity, from the numerical perspective of of distributed memory across its parameters. For the same
machine learning, notions such as image, movement, form, reason, it is essentially impossible to visualise an algorith-
style, and ethical decision can all be described as statistical mic statistical model, as is done with simple mathematical
distributions of pattern. In this sense, pattern recognition has functions. Still, the challenge is worthwhile.
truly become a new cultural technique that is used in various In machine learning, there are many algorithm architec-
fields. For explanatory purposes, the Nooscope is described tures: simple Perceptron, deep neural network, Support Vec-
as a machine that operates on three modalities: training, tor Machine, Bayesian network, Markov chain, autoencoder,
classification, and prediction. In more intuitive terms, these Boltzmann machine, etc. Each of these architectures has a
modalities can be called: pattern extraction, pattern recogni- different history (often rooted in military agencies and cor-
tion, and pattern generation. porations of the Global North). Artificial neural networks
Rosenblatt’s Perceptron was the first algorithm that paved started as simple computing structures that evolved into
the way to machine learning in the contemporary sense. At complex ones which are now controlled by a few hyperpa-
a time when ‘computer science’ had not yet been adopted rameters that express millions of parameters.18 For instance,
as definition, the field was called ‘computational geometry’ convolutional neural networks are described by a limited set
and specifically ‘connectionism’ by Rosenblatt himself. The of hyperparameters (number of layers, number of neurons
business of these neural networks, however, was to calcu- per layer, type of connection, behaviour of neurons, etc.)
late a statistical inference. What a neural network computes that project a complex topology of thousands of artificial
is not an exact pattern but the statistical distribution of a neurons with millions of parameters in total. The algorithm
pattern. Just scraping the surface of the anthropomorphic starts as a blank slate and, during the process called train-
marketing of AI, one finds another technical and cultural ing, or ‘learning from data’, adjusts its parameters until it
object that needs examination: the statistical model. What reaches a good representation of the input data. In image
is the statistical model in machine learning? How is it cal- recognition, as already seen, the computation of millions
culated? What is the relationship between a statistical model of parameters has to resolve into a simple binary output:
and human cognition? These are crucial issues to clarify. In 1 = true, a given image is recognised; or 0 = false, a given
terms of the work of demystification that needs to be done image is not recognised.19
(also to evaporate some naïve questions), it would be good
to reformulate the trite question ‘Can a machine think?’ into
the theoretically sounder questions ‘Can a statistical model
think?’, ‘Can a statistical model develop consciousness?’,
et cetera.

18
The parameters of a model that are learnt from data are called
‘parameters’, while parameters that are not learnt from data and are
fixed manually are called ‘hyperparameters’ (these determine number
and properties of the parameters.).
19
This value can be also a percentage value between 1 and 0.

13
AI & SOCIETY (2021) 36:1263–1280 1269

Source: https​://www.asimo​vinst​itute​.org/neura​l-netwo​rk-zoo

Attempting an accessible explanation of the relation- that occupy approximately 150 gigabytes of memory. On the
ship between algorithm and model, let us have a look at other hand, Inception v3, which is meant to represent the
the complex Inception v3 algorithm, a deep convolutional information contained in ImageNet, is only 92 megabytes.
neural network for image recognition designed at Google The ratio of compression between training data and model
and trained on the ImageNet dataset. Inception v3 is said partially describes also the rate of information diffraction.
to have a 78% accuracy in identifying the label of a pic- A table from the Keras documentation compares these val-
ture, but the performance of ‘machine intelligence’ in this ues (numbers of parameters, layer depth, file dimension and
case can be measured also by the proportion between the accuracy) for the main models of image recognition.20 This
size of training data and the trained algorithm (or model).
ImageNet contains 14 million images with associated labels 20
https​://keras​.io/appli​catio​ns (documentation for individual mod-
els.)

13
1270 AI & SOCIETY (2021) 36:1263–1280

is a brutalist but effective way to show the relation between community, and today, also beyond. 21 Machine learning
model and data, to show how the ‘intelligence’ of algorithms models, on the contrary, are opaque and inaccessible to
is measured and assessed in the developer community. community debate. Given the degree of myth-making and
social bias around its mathematical constructs, AI has indeed

inaugurated the age of statistical science fiction. Nooscope

Statistical models have always influenced culture and is the projector of this large statistical cinema.
politics. They did not just emerge with machine learning:
machine learning is just a new way to automate the tech-
nique of statistical modelling. When Greta Thunberg warns 6 All models are wrong, but some are useful
‘Listen to science.’ what she really means, being a good
student of mathematics, is ‘Listen to the statistical mod- ‘All models are wrong, but some are useful’—the canoni-
els of climate science.’ No statistical models, no climate cal dictum of the British statistician George Box has long
science: no climate science, no climate activism. Climate encapsulated the logical limitations of statistics and machine
science is indeed a good example to start with, in order to learning (Box 1979). This maxim, however, is often used
understand statistical models. Global warming has been to legitimise the bias of corporate and state AI. Computer
calculated by first collecting a vast dataset of temperatures scientists argue that human cognition reflects the capacity
from Earth’s surface each day of the year, and second, by to abstract and approximate patterns. Therefore, what’s the
applying a mathematical model that plots the curve of tem-
perature variations in the past and projects the same pattern 21
See the Community Earth System Model (CESM) that has been
into the future (Edwards 2010). Climate models are histori- developed by the National Center for Atmospheric Research in
cal artefacts that are tested and debated within the scientific Bolder, Colorado, since 1996. The Community Earth System Model
is a fully coupled numerical simulation of the Earth system consist-
ing of atmospheric, ocean, ice, land surface, carbon cycle, and other
components. CESM includes a climate model providing state-of-the-
art simulations of the Earth’s past, present, and future.’ https​://www.
cesm.ucar.edu

13
AI & SOCIETY (2021) 36:1263–1280 1271

problem with machines being approximate, and doing the The challenge of guarding the accuracy of machine learn-
same? Within this argument, it is rhetorically repeated that ing lays in calibrating the equilibrium between data underfit-
‘the map is not the territory’. This sounds reasonable. But ting and overfitting, which is difficult to do because of differ-
what should be contested is that AI is a heavily compressed ent machine biases. Machine learning is a term that, as much
and distorted map of the territory and that this map, like as ‘AI’, anthropomorphizes a piece of technology: machine
many forms of automation, is not open to community negoti- learning learns nothing in the proper sense of the word, as
ation. AI is a map of the territory without community access a human does; machine learning simply maps a statistical
and community consent.22 distribution of numerical values and draws a mathematical
How does machine learning plot a statistical map of the function that hopefully approximates human comprehension.
world? Let’s face the specific case of image recognition (the That being said, machine learning can, for this reason, cast
basic form of the labour of perception, which has been codified new light on the ways in which humans comprehend.
and automated as pattern recognition)23 (Beller 2012). Given The statistical model of machine learning algorithms is
an image to be classified, the algorithm detects the edges of an also an approximation in the sense that it guesses the miss-
object as the statistical distribution of dark pixels surrounded ing parts of the data graph: either through interpolation,
by light ones (a typical visual pattern). The algorithm does not which is the prediction of an output y within the known
know what an image is, does not perceive an image as human interval of the input x in the training dataset, or through
cognition does, it only computes pixels, numerical values of extrapolation, which is the prediction of output y beyond
brightness and proximity. The algorithm is programmed to the limits of x, often with high risks of inaccuracy. This
record only the dark edge of a profile (that is to fit that desired is what ‘intelligence’ means today within machine intelli-
pattern) and not all the pixels across the image (that would gence: to extrapolate a non-linear function beyond known
result in overfitting and repeating the whole visual field). A data boundaries. As Dan McQuillian aptly puts it: ‘There
statistical model is said to be trained successfully when it can is no intelligence in artificial intelligence, nor does it learn,
elegantly fit only the important patterns of the training data even though its technical name is machine learning, it is
and apply those patterns also to new data ‘in the wild’. If a simply mathematical minimization’ (McQuillan 2018a; b).
model learns the training data too well, it recognises only exact It is important to recall that the ‘intelligence’ of machine
matches of the original patterns and will overlook those with learning is not driven by exact formulas of mathematical
close similarities, ‘in the wild’. In this case, the model is over- analysis, but by algorithms of brute force approximation.
fitting, because it has meticulously learnt everything (including The shape of the correlation function between input x and
noise) and is not able to distinguish a pattern from its back- output y is calculated algorithmically, step by step, through
ground. On the other hand, the model is underfitting when it is tiresome mechanical processes of gradual adjustment (like
not able to detect meaningful patterns from the training data. gradient descent, for instance) that are equivalent to the dif-
The notions of data overfitting, fitting and underfitting can be ferential calculus of Leibniz and Newton. Neural networks
visualised on a Cartesian plane.

22
Post-colonial and post-structuralist schools of anthropology and
ethnology have stressed that there is never territory per se, but always
an act of territorialisation.
23
Pattern recognition is one among many other economies of atten-
tion. ‘To look is to labor’, as Jonathan Beller reminds us.

13
1272 AI & SOCIETY (2021) 36:1263–1280

are said to be among the most efficient algorithms, because tion within a space with several hundreds of dimen-
these differential methods can approximate the shape of any sions. The advantage of such a representation resides
function given enough layers of neurons and abundant com- in the numerous operations offered by such a transfor-
puting resources.24 Brute-force gradual approximation of a mation. Two terms whose inferred positions are near
function is the core feature of today’s AI, and only from this one another in this space are equally similar semanti-
perspective can one understand its potentialities and limita- cally; these representations are said to be distributed:
tions—particularly, its escalating carbon footprint (the train- the vector of the concept ‘apartment’ [− 0.2, 0.3, − 4.2,
ing of deep neural networks requires exorbitant amounts of 5.1…] will be similar to that of ‘house’ [− 0.2, 0.3,
energy because of gradient descent and similar training algo- − 4.0, 5.1…].[…] While natural language processing
rithms that operate on the basis of continuous infinitesimal was pioneering for ‘embedding’ words in a vectorial
adjustments) (Ganesh et al. 2019). space, today we are witnessing a generalization of the
embedding process which is progressively extending
to all applications fields: networks are becoming sim-
ple points in a vectorial space with graph2vec, texts
with paragraph2vec, films with movie2vec, meanings
of words with sens2vec, molecular structures with
mol2vec, etc. According to Yann LeCun, the goal of
the designers of connectionist machines is to put the
world in a vector (world2vec) (Cardon et al. 2018a).
Multi-dimensional vector space is another reason why
the logic of machine learning is difficult to grasp. Vector
space is another new cultural technique, worth becoming
familiar with. The field of Digital Humanities, in particular,
has been covering the technique of vectorialisation through
which our collective knowledge is invisibly rendered and
Multidimensional vector space. processed. William Gibson’s original definition of cyber-
space prophesized, most likely, the coming of a vector space
rather than virtual reality: ‘A graphic representation of data
7 World to vector abstracted from the banks of every computer in the human
system. Unthinkable complexity. Lines of light ranged in the
The notions of data fitting, overfitting, underfitting, inter- nonspace of the mind, clusters and constellations of data.
polation and extrapolation can be easily visualised in two Like city lights, receding’ (Gibson 1984, p. 69).
dimensions, but statistical models usually operate along
multidimensional spaces of data. Before being analysed,
data are encoded into a multi-dimensional vector space that
is far from intuitive. What is a vector space and why is it
multi-dimensional? Cardon, Cointet and Mazière describe
the vectorialisation of data in this way:
A neural network requires the inputs of the calculator
to take on the form of a vector. Therefore, the world
must be coded in advance in the form of a purely digi-
tal vectorial representation. While certain objects such
as images are naturally broken down into vectors, other
objects need to be ‘embedded’ within a vectorial space
before it is possible to calculate or classify them with
neural networks. This is the case of text, which is the
prototypical example. To input a word into a neural
network, the Word2vec technique ‘embeds’ it into a
vectorial space that measures its distance from the
other words in the corpus. Words thus inherit a posi-
24
As proven by the Universal Approximation Theorem.

13
AI & SOCIETY (2021) 36:1263–1280 1273

Vector space of seven words in three contexts.25 originary application of neural networks: with Deep Learn-
It must be stressed, however, that machine learning still ing, this technique is found ubiquitously in face recognition
resembles more craftsmanship than exact mathematics. AI is classifiers that are deployed by police forces and smartphone
still a history of hacks and tricks rather than mystical intui- manufacturers alike.
tions. For example, one trick of information compression is Machine learning prediction is used to project future
dimensionality reduction, which is used to avoid the Curse of trends and behaviours according to past ones, that is to com-
Dimensionality, that is the exponential growth of the variety of plete a piece of information knowing only a portion of it.
features in the vector space. The dimensions of the categories In the prediction modality, a small sample of input data (a
that show low variance in the vector space (i.e. whose values primer) is used to predict the missing part of the information
fluctuate only a little) are aggregated to reduce calculation costs. following once again the statistical distribution of the model
Dimensionality reduction can be used to cluster word meanings (this could be the part of a numerical graph oriented toward
(such as in the model word2vec) but can also lead to category the future or the missing part of an image or audio file). Inci-
reduction, which can have an impact on the representation of dentally, other modalities of machine learning exist: the sta-
social diversity. Dimensionality reduction can shrink taxono- tistical distribution of a model can be dynamically visualised
mies and introduce bias, further normalising world diversity through a technique called latent space exploration and, in
and obliterating unique identities (Samadi et al. 2018). some recent design applications, also pattern exploration.26

8 The society of classification and prediction Machine learning classification and prediction are becom-
bots ing ubiquitous techniques that constitute new forms of sur-
veillance and governance. Some apparatuses, such as self-
Most of the contemporary applications of machine learning driving vehicles and industrial robots, can be an integration
can be described according to the two modalities of clas- of both modalities. A self-driving vehicle is trained to rec-
sification and prediction, which outline the contours of a ognise different objects on the road (people, cars, obstacles,
new society of control and statistical governance. Classi- signs) and predict future actions based on decisions that a
fication is known as pattern recognition, while prediction human driver has taken in similar circumstances. Even if
can be defined also as pattern generation. A new pattern is recognising an obstacle on a road seems to be a neutral ges-
recognised or generated by interrogating the inner core of ture (it’s not), identifying a human being according to cat-
the statistical model. egories of gender, race and class (and in the recent COVID-
Machine learning classification is usually employed to 19 pandemic as sick or immune), as state institutions are
recognise a sign, an object, or a human face, and to assign increasingly doing, is the gesture of a new disciplinary
a corresponding category (label) according to taxonomy regime. The hubris of automated classification has caused
or cultural convention. An input file (e.g. a headshot cap- the revival of reactionary Lombrosian techniques that were
tured by a surveillance camera) is run through the model to thought to have been consigned to history, techniques such
determine whether it falls within its statistical distribution as automatic gender recognition (AGR), ‘a subfield of facial
or not. If so, it is assigned the corresponding output label. recognition that aims to algorithmically identify the gender
Since the times of the Perceptron, classification has been the of individuals from photographs or videos’ (Keyes 2018).
26
See the idea of assisted and generative creation in: Pieters and
25
Source: https​://corpl​ing.hypot​heses​.org/495. Winiger (2016).

13
1274 AI & SOCIETY (2021) 36:1263–1280

Recently, the generative modality of machine learning has The two main modalities of classification and genera-
had a cultural impact: its use in the production of visual arte- tion can be assembled in further architectures such as in the
facts has been received by mass media as the idea that arti- Generative Adversarial Networks. In the GAN architecture,
ficial intelligence is ‘creative’ and can autonomously make a neural network with the role of discriminator (a traditional
art. An artwork that is said to be created by AI always hides classifier) has to recognise an image produced by a neural
a human operator, who has applied the generative modal- network with the role of generator, in a reinforcement loop
ity of a neural network trained on a specific dataset. In this that trains the two statistical models simultaneously. For
modality, the neural network is run backwards (moving from some converging properties of their respective statistical
the smaller output layer toward the larger input layer) to gen- models, GANs have proved very good at generating highly
erate new patterns after being trained at classifying them, a realistic pictures. This ability has prompted their abuse in the
process that usually moves from the larger input layer to the fabrication of ‘deep fakes’.27 Concerning regimes of truth, a
smaller output layer. The generative modality, however, has similar controversial application is the use of GANs to gen-
some useful applications; it can be used as a sort of reality erate synthetic data in cancer research, in which neural net-
check to reveal what the model has learnt, i.e. to show how works trained on unbalanced datasets of cancer tissues have
the model ‘sees the world.’ It can be applied to the model started to hallucinate cancer where there was none (Cohen
of a self-driving car, for instance, to check how the road et al. 2018). In this case ‘instead of discovering things, we are
scenario is projected. inventing things’, Fabian Offert notices, ‘the space of discov-
A famous way to illustrate how a statistical model ‘sees ery is identical to the space of knowledge that the GAN has
the world’ is Google DeepDream. DeepDream is a convolu- already had.[…] While we think that we are seeing through
tional neural network based on Inception (which is trained on GAN—looking at something with the help of a GAN—we
the ImageNet dataset mentioned above) that was programmed are actually seeing into a GAN. GAN vision is not augmented
by Alexander Mordvintsev to project hallucinatory patterns. reality, it is virtual reality. GANs do blur discovery and inven-
Mordvintsev had the idea to ‘turn the network upside down’, tion’ (Offert 2020). The GAN simulation of brain cancer is a
that is to turn a classifier into a generator, using some random tragic example of AI-driven scientific hallucination.
noise or generic landscape images as input (Mordvintsev

et al. 2015). He discovered that ‘neural networks that were

trained to discriminate between different kinds of images
have quite a bit of the information needed to generate images
too.’ In DeepDream first experiments, bird feathers and dog
eyes started to emerge everywhere as dog breeds and bird
species are vastly overrepresented in ImageNet. It was also
discovered that the category ‘dumbbell’ was learnt with a
surreal human arm always attached to it. Proof that many 27
Deep fakes are synthetic media like videos in which a person’s
other categories of ImageNet are misrepresented. face is replaced with someone else’s facial features, often for the pur-
pose to forge fake news.

13
AI & SOCIETY (2021) 36:1263–1280 1275

Joseph Paul Cohen, Margaux Luck and Sina Honari. innovative act of disruption? The two modalities of machine
‘Distribution Matching Losses Can Hallucinate Features in learning display a limitation that is not simply bias.
Medical Image Translation’, 2018. Courtesy of the authors. A logical limit of machine learning classification, or
pattern recognition, is the inability to recognise a unique
anomaly that appears for the first time, such as a new meta-
9 Faults of a statistical instrument: phor in poetry, a new joke in everyday conversation, or an
the undetection of the new unusual obstacle (a pedestrian? a plastic bag?) on the road
scenario. The undetection of the new (something that has
The normative power of AI in the twenty first century has to never ‘been seen’ by a model and therefore never classi-
be scrutinised in these epistemic terms: what does it mean fied before in a known category) is a particularly hazardous
to frame collective knowledge as patterns, and what does problem for self-driving cars and one that has already caused
it mean to draw vector spaces and statistical distributions fatalities. Machine learning prediction, or pattern genera-
of social behaviours? According to Foucault, in early mod- tion, show similar faults in the guessing of future trends
ern France, statistical power was already used to measure and behaviours. As a technique of information compression,
social norms, discriminating between normal and abnor- machine learning automates the dictatorship of the past, of
mal behaviour (Foucault 2004, p. 26). AI easily extends past taxonomies and behavioural patterns, over the present.
the ‘power of normalisation’ of modern institutions, among This problem can be termed the regeneration of the old—the
others bureaucracy, medicine and statistics (originally, the application of a homogenous space–time view that restrains
numerical knowledge possessed by the state about its popu- the possibility of a new historical event.
lation) that passes now into the hands of AI corporations. Interestingly, in machine learning, the logical definition
The institutional norm has become a computational one: the of a security issue also describes the logical limit of its crea-
classification of the subject, of bodies and behaviours, seems tive potential. The problems characteristic of the prediction
no longer to be an affair for public registers, but instead of the new are logically related to those that characterise the
for algorithms and datacentres.28 ‘Data-centric rationality’, generation of the new, because the way a machine learn-
Paula Duarte has concluded, ‘should be understood as an ing algorithm predicts a trend on a time chart is identical
expression of the coloniality of power’ (Ricaurte 2019). to the way it generates a new artwork from learnt patterns.
A gap, a friction, a conflict, however, always persists The hackneyed question ‘Can AI be creative?’ should be
between AI statistical models and the human subject that reformulated in technical terms: is machine learning able to
is supposed to be measured and controlled. This logi- create works that are not imitations of the past? Is machine
cal gap between AI statistical models and society is usu- learning able to extrapolate beyond the stylistic boundaries
ally debated as bias. It has been extensively demonstrated of its training data? The ‘creativity’ of machine learning
how face recognition misrepresents social minorities and is limited to the detection of styles from the training data
how black neighbourhoods, for instance, are bypassed by and then random improvisation within these styles. In other
AI-driven logistics and delivery service (Ingold and Soper words, machine learning can explore and improvise only
2016). If gender, race and class discriminations are ampli- within the logical boundaries that are set by the training
fied by AI algorithms, this is also part of a larger problem data. For all these issues, and its degree of information com-
of discrimination and normalisation at the logical core of pression, it would be more accurate to term machine learn-
machine learning. The logical and political limitation of AI ing art as statistical art.
is the technology’s difficulty in the recognition and predic-
tion of a new event. How is machine learning dealing with
a truly unique anomaly, an uncommon social behaviour, an

28
On computational norms see: Pasquinelli (2017).

13
1276 AI & SOCIETY (2021) 36:1263–1280

Lewis Fry Richardson, Weather Prediction by Numerical correlation is enough for Google to run its ads business,
Process, Cambridge University Press, 1922. therefore, it must also be good enough to automatically
Another unspoken bug of machine learning is that the sta- discover scientific paradigms. Even Judea Pearl, a pioneer
tistical correlation between two phenomena is often adopted of Bayesian networks, believes that machine learning is
to explain causation from one to the other. In statistics, it is obsessed with ‘curve fitting’, recording correlations without
commonly understood that correlation does not imply cau- providing explanations (Mackenzie and Judea 2018). Such
sation, meaning that a statistical coincidence alone is not a logical fallacy has already become a political one, if one
sufficient to demonstrate causation. A tragic example can be considers that police forces worldwide have adopted predic-
found in the work of statistician Frederick Hoffman, who in tive policing algorithms.30 According to Dan McQuillan,
1896 published a 330-page report for insurance companies when machine learning is applied to society in this way, it
to demonstrate a racial correlation between being a black turns into a biopolitical apparatus of preemption, that pro-
American and having short life expectancy (O’Neil 2016). duces subjectivities which can subsequently be criminal-
Superficially mining data, machine learning can construct ized (McQuillan 2018a; b). Ultimately, machine learning
any arbitrary correlation that is then perceived as real. In obsessed with ‘curve fitting’ imposes a statistical culture
2008, this logical fallacy was proudly embraced by Wired and replaces the traditional episteme of causation (and polit-
director Chris Anderson who declared the ‘end of theory’, ical accountability) with one of correlations blindly driven
because ‘the data deluge makes the scientific method obso- by the automation of decision making.
lete’ (Anderson 2008).29 According to Anderson, himself no
expert on scientific method and logical inference, statistical

30
Experiments by the New York Police Department since the late
29
For a critique see: Mazzocchi (2015). 1980s. See: Pasquinelli, Arcana Mathematica Imperii.

13
AI & SOCIETY (2021) 36:1263–1280 1277

10 Adversarial intelligence vs. artificial

intelligence

So far, the statistical diffractions and hallucinations of

machine learning have been followed step by step through
the multiple lenses of the Nooscope. At this point, the orien-
tation of the instrument has to be reversed: scientific theories
as much as computational devices are inclined to consolidate
an abstract perspective—the scientific ‘view from nowhere’,
that is often just the point of view of power. The obsessive
study of AI can suck the scholar into an abyss of computa-
tion and the illusion that the technical form illuminates the
social one. As Paola Ricaurte remarks: ‘Data extractivism
assumes that everything is a data source’ (Ricaurte 2019).
How to emancipate ourselves from a data-centric view of
the world? It is time to realise that it is not the statistical
model that constructs the subject, but rather the subject that
structures the statistical model. Internalist and externalist
studies of AI have to blur: subjectivities make the mathemat-
ics of control from within, not from without. To second what
Guattari once said of machines in general, machine intel-
ligence too is constituted of ‘hyper-developed and hyper-
concentrated forms of certain aspects of human subjectivity’ Adam Harvey, HyperFace pattern, 2016.
(Guattari 2013, p. 2). Adversarial attacks exploit blind spots and weak regions
Rather than studying only how technology works, critical in the statistical model of a neural network, usually to fool a
inquiry studies also how it breaks, how subjects rebel against classifier and make it perceive something that is not there. In
its normative control and workers sabotage its gears. In this object recognition, an adversarial example can be a doctored
sense, a way to sound the limits of AI is to look at hacking image of a turtle, which looks innocuous to a human eye but
practices. Hacking is an important method of knowledge gets misclassified by a neural network as a rifle (Athalye
production, a crucial epistemic probe into the obscurity et al. 2017). Adversarial examples can be realised as 3D
of AI.31 Deep learning systems for face recognition have objects and even stickers for road signs that can misguide
triggered, for instance, forms of counter-surveillance activ- self-driving cars (which may read a speed limit of 120 km/h
ism. Through techniques of face obfuscation, humans have where it is actually 50 km/h) (Morgulis et al. 2019). Adver-
decided to become unintelligible to artificial intelligence: sarial examples are designed knowing what a machine has
that is to become, themselves, black boxes. The traditional never seen before. This effect is achieved also by reverse-
techniques of obfuscation against surveillance immediately engineering the statistical model or by polluting the training
acquire a mathematical dimension in the age of machine dataset. In this latter sense, the technique of data poisoning
learning. For example, AI artist and researcher Adam Har- targets the training dataset and introduces doctored data. In
vey has invented a camouflage textile called HyperFace that doing so, it alters the accuracy of the statistical model and
fools computer vision algorithms to see multiple human creates a backdoor that can be eventually exploited by an
faces where there is none (Harvey 2016). Harvey’s work adversarial attack.32
provokes the question: what constitutes a face for a human Adversarial attack seems to point to a mathematical
eye, on the one hand, and a computer vision algorithm, on vulnerability that is common to all machine learning mod-
the other? The neural glitches of HyperFace exploit such a els: ‘An intriguing aspect of adversarial examples is that
cognitive gap and reveal what a human face looks like to a an example generated for one model is often misclassified
machine. This gap between human and machine perception by other models, even when they have different architec-
helps to introduce the growing field of adversarial attacks. tures or were trained on disjoint training sets’ (Goodfellow
et al. 2014). Adversarial attacks remind us of the discrep-
ancy between human and machine perception and that the

31
The relationship between AI and hacking is not as antagonistic as
32
it may appear: it often resolves in a loop of mutual learning, evalua- Data poisoning can also be employed to protect privacy by enter-
tion and reinforcement. ing anonymised or random information into the dataset.

13
1278 AI & SOCIETY (2021) 36:1263–1280

logical limit of machine learning is also a political one. The that AI has to analyse and automate. Datasets for image rec-
logical and ontological boundary of machine learning is the ognition, for instance, record the visual labour that drivers,
unruly subject or anomalous event that escapes classification guards, and supervisors usually perform during their tasks.
and control. The subject of algorithmic control fires back. Even scientific datasets rely on scientific labour, experiment
Adversarial attacks are a way to sabotage the assembly line planning, laboratory organisation, and analytical observa-
of machine learning by inventing a virtual obstacle that can tion. The information flow of AI has to be understood as
set the control apparatus out of joint. An adversarial example an apparatus designed to extract ‘analytical intelligence’
is the sabot in the age of AI. from the most diverse forms of labour and to transfer such
intelligence into a machine (obviously including, within the
definition of labour, extended forms of social, cultural and
11 Labour in the age of AI scientific production).33 In short, the origin of machine intel-
ligence is the division of labour and its main purpose is the
The natures of the ‘input’ and ‘output’ of machine learn- automation of labour.
ing have to be clarified. AI troubles are not only about Historians of computation have already stressed the early
information bias but also labour. AI is not just a control steps of machine intelligence in the nineteenth century pro-
apparatus, but also a productive one. As just mentioned, an ject of mechanizing the division of mental labour, specifi-
invisible workforce is involved in each step of its assembly cally the task of hand calculation (Schaffer 1994; Daston
line (dataset composition, algorithm supervision, model 1994; Jones 2016). The enterprise of computation has since
evaluation, etc.). Pipelines of endless tasks innervate from then been a combination of surveillance and disciplining of
the Global North into the Global South; crowdsourced labour, of optimal calculation of surplus-value, and planning
platforms of workers from Venezuela, Brazil and Italy, for of collective behaviours (Pasquinelli 2019a). Computation
instance, are crucial to teach German self-driving cars ‘how was established by and still enforces a regime of visibility
to see’ (Schmidt 2019). Against the idea of alien intelli- and intelligibility, not just of logical reasoning. The geneal-
gence at work, it must be stressed that in the whole com- ogy of AI as an apparatus of power is confirmed today by
puting process of AI the human worker has never left the its widespread employment in technologies of identification
loop, or put more accurately, has never left the assembly and prediction, yet the core anomaly which always remains
line. Mary Gray and Siddharth Suri coined the term ‘ghost to be computed is the disorganisation of labour.
work’ for the invisible labour that makes AI appear artifi- As a technology of automation, AI will have a tremen-
cially autonomous. dous impact on the job market. If Deep Learning has a 1%
error rate in image recognition, for example, it means that
Beyond some basic decisions, today’s artificial intel-
roughly 99% of routine work based on visual tasks (e.g. air-
ligence can’t function without humans in the loop.
port security) can be potentially replaced (legal restrictions
Whether it’s delivering a relevant newsfeed or carry-
and trade union opposition permitting). The impact of AI
ing out a complicated texted-in pizza order, when the
on labour is well described (from the perspective of work-
artificial intelligence (AI) trips up or can’t finish the
ers, finally) within a paper from the European Trade Union
job, thousands of businesses call on people to quietly
Institute, which highlights ‘seven essential dimensions that
complete the project. This new digital assembly line
future regulation should address to protect workers: (1) safe-
aggregates the collective input of distributed workers,
guarding worker privacy and data protection; (2) addressing
ships pieces of projects rather than products, and oper-
surveillance, tracking and monitoring; (3) making the pur-
ates across a host of economic sectors at all times of
pose of AI algorithms transparent; (4) ensuring the exercise
the day and night.
of the ‘right to explanation’ regarding decisions made by
Automation is a myth, because machines, including AI, algorithms or machine learning models; (5) preserving the
constantly call for human help, some authors have suggested security and safety of workers in human–machine interac-
replacing ‘automation’ with the more accurate term hetero- tions; (6) boosting workers’ autonomy in human–machine
mation (Ekbia and Nardi 2017). Heteromation means that interactions; (7) enabling workers to become AI literate’
the familiar narrative of AI as perpetuum mobile is possible (Ponce 2020).
only thanks to a reserve army of workers. Ultimately, the Nooscope manifests in response to the
Yet, there is a more profound way in which labour con- need for a novel Machinery Question in the age of AI. The
stitutes AI. The information source of machine learning Machinery Question was a debate that sparked in England
(whatever its name: input data, training data or just data) during the industrial revolution, when the response to the
is always a representation of human skills, activities and
behaviours, social production at large. All training datasets
are, implicitly, a diagram of the division of human labour 33
For the idea of analytical intelligence see: Daston (2018).

13
AI & SOCIETY (2021) 36:1263–1280 1279

employment of machines and workers’ unemployment was Box G (1979) Robustness in the strategy of scientific model building.
a social campaign for more education about machines, that Technical report #1954, Mathematics Research Center, University
of Wisconsin-Madison
took the form of the Mechanics’ Institute Movement (Berg Campolo A, Crawford K (2020) Enchanted determinism: power with-
1980).34 Today, an Intelligent Machinery Question is needed out control in artificial intelligence. Engag Sci Technol Soc 6:1–19
to develop more collective intelligence about machine Cardon D, Cointet JP, Mazières A (2018a) Neurons spike back: the
intelligence, more public education instead of ‘learning invention of inductive machines and the artificial intelligence
controversy. Réseaux 5:211
machines’ and their regime of knowledge extractivism, Cardon D, Cointet JP, Mazières A (2018b) Neurons spike back. The
which crosses once again old colonial routes (if one looks invention of inductive machines and the artificial intelligence con-
at the network map of crowdsourcing). Also in the Global troversy. Réseaux 211:173–220
North, the colonial relationship between corporate AI and Cohen JP, Honari S, Margaux L (2018) Distribution matching losses
can hallucinate features in medical image translation. In: Inter-
the production of knowledge as a common good has to be national conference on medical image computing and computer-
brought to the forefront. The Nooscope’s purpose is to break assisted intervention, Springer, Cham. arXiv:1805.08841
into the hidden room of the corporate Mechanical Turk, and Corsani A, Paulré B, Vercellone C, Monnier JM, Lazzarato M,
to illuminate the invisible labour of knowledge that makes Dieuaide P, Moulier-Boutang Y (2004) Le Capitalisme cognitif
comme sortie de la crise du capitalisme industriel. Un programme
machine intelligence appear ideologically alive. de recherché, Laboratoire Isys Matisse, Maison des Sciences
Thanks to Jon Beller, Claire Glanois, Adam Harvey, Economiques, Paris
Leonardo Impett, Arif Kornweitz, Wietske Maas, Dan Crawford K (2017) The trouble with bias. Keynote lecture: conference
McQuillan, Fabian Offert, Godofredo Pereira, Mitch Speed on neural information processing systems
Crawford K, Paglen T (2019) Excavating AI: the politics of training
and the extended community around KIM HfG Karlsruhe for sets for machine learning. https​://excav​ating​.ai. Accessed 30 Apr
their inputs and comments. 2020
Daston L (1994) Enlightenment calculations. Crit Inq 21
Daston L (2018) Calculation and the division of labour 1750–1950.
Funding Open Access funding enabled and organized by Projekt Bull Ger Hist Inst 62:9–30
DEAL.. Edwards P (2010) A vast machine: computer models, climate data, and
the politics of global warming. MIT Press, Cambridge
Ekbia H, Nardi B (2017) Heteromation, and other stories of computing
Open Access This article is licensed under a Creative Commons Attri- and capitalism. MIT Press, Cambridge
bution 4.0 International License, which permits use, sharing, adapta- Eubanks V (2018) Automating inequality. St. Martin’s Press, New York
tion, distribution and reproduction in any medium or format, as long Foucault M (2004) Abnormal: Lectures at the Collège de France 1974–
as you give appropriate credit to the original author(s) and the source, 1975. Picador, New York, p 26
provide a link to the Creative Commons licence, and indicate if changes Foucault M (2005) The order of things. Routledge, London
were made. The images or other third party material in this article are Galstyan A, Lerman K, Mehrabi N, Morstatter F, Saxena N (2019) A
included in the article’s Creative Commons licence, unless indicated survey on bias and fairness in machine learning. arxiv preprint
otherwise in a credit line to the material. If material is not included in https​://arxiv​.org/abs/1908.09635​. Accessed 30 Apr 2020
the article’s Creative Commons licence and your intended use is not Ganesh A, McCallum A, Strubell E (2019) Energy and policy consider-
permitted by statutory regulation or exceeds the permitted use, you will ations for deep learning in NLP. arXiv preprint arXiv:1906.02243
need to obtain permission directly from the copyright holder. To view a Gibson W (1984) Neuromancer. Ace Books, New York, p 69
copy of this licence, visit http://creat​iveco​mmons​.org/licen​ses/by/4.0/. Gitelman L (ed) (2013) Raw data is an oxymoron. MIT Press,
Cambridge
Goodfellow I, Shlens J, Szegedy C (2014) Explaining and harness-
References ing adversarial examples. arXiv preprint. https​: //arxiv​. org/
abs/1412.6572. Accessed 30 Apr 2020
Guattari F (2013) Schizoanalytic cartographies. Coninuum, London,
Anderson C (2008) The end of theory: the data deluge makes the sci- p2
entific method obsolete. Wired Magazine 23 June Guttag J, Suresh H (2019) A framework for understanding unintended
Athalye A et al (2017) Synthesizing robust adversarial Examples. arXiv consequences of machine learning. arxiv preprint. https​://arxiv​
preprint. https:​ //arxiv.​ org/abs/1707.07397.​ Accessed 30 Apr 2020 .org/abs/1901.10002​. Accessed 30 Apr 2020
Beller J (2006) The cinematic mode of production: attention economy Harvey A (2016) HyperFace project. https:​ //ahproj​ ects.​ com/hyperf​ ace.
and the society of the spectacle. University Press of New England, Accessed 30 Apr 2020
Lebanon, NH, p 2 Harvey A (2019) Megapixel project. https​://megap​ixels​.cc/about​/.
Benjamin R (2019) Race after technology: abolitionist tools for the new Accessed 30 Apr 2020
jim code. Polity, Cambridge, p 5 Ingold D, Soper S (2016) Amazon doesn’t consider the race of its cus-
Berg M (1980) The machinery question and the making of political tomers. Should it?. https​://www.bloom​berg.com/graph​ics/2016-
economy. Cambridge University Press, Cambridge amazo​n-same-day. Accessed 21 Apr 2016
Jones ML (2016) Reckoning with matter: calculating machines, innova-
tion, and thinking about thinking from Pascal to Babbage. Univer-
sity of Chicago Press, Chicago
Keyes O (2018) The misgendering machines: trans/HCI implications
of automatic gender recognition. In: Proceedings of the ACM on
34
In fact, even the Economist has recently warned about ‘the return human-computer interaction, vol 2, n CSCW, article 88. https​://
of the machinery question’ in the age of AI. See: Standage (2016). doi.org/10.1145/32743​57

13
1280 AI & SOCIETY (2021) 36:1263–1280

Krizhevsky A, Sutskever I, Hinton GE (2017) Imagenet classifica- Pasquinelli M (2017) Arcana mathematica imperii: the evolution of
tion with deep convolutional neural networks. Commun ACM western computational norms. In: Hlavajova M et al (eds) Former
60(6):84–90 west. MIT Press, Cambridge
Leibniz GW (1677) Preface to the general science. In: Wiener P (ed) Pasquinelli M (2019a) On the origins of Marx’s general intellect. Radic
Selections, 1951. Scribner, New York Philos 2(6):43–56
Lipton ZC (2016) The mythos of model interpretability. arXiv preprint Pasquinelli M (2019b) Three thousand years of algorithmic rituals.
https​://arxiv​.org/abs/1606.03490​. Accessed 30 Apr 2020 e-flux 101
Mackenzie D, Judea P (2018) The book of why: the new science of Pasquinelli M (forthcoming) The eye of the master. Verso, London
cause and effect. Basic Books, New York Pieters R, Winiger S (2016) Creative AI: on the democratisation and
Malik MM (2002) A hierarchy of limitations in machine learning. escalation of creativity. https​://www.mediu​m.com/@creat​iveai​/
arxiv preprint, 2020. https​://arxiv​.org/abs/2002.05193​. Accessed creat​iveai​-9d4b2​346fa​f3
30 Apr 2020 Ponce A (2020) Labour in the age of AI: why regulation is needed to
Mazzocchi F (2015) Could Big Data be the end of theory in science? A protect workers. ETUI Research Paper - Foresight Brief #08. https​
few remarks on the epistemology of data-driven science. EMBO ://doi.org/10.2139/ssrn.35410​02
Rep 16(10):1250–1255 Pontin J (2007) Artificial intelligence, with help from the humans. The
McCulloch W, Pitts W (1947) How we know universals: the perception New York Times, 25 Mar 2007
of auditory and visual forms. Bull Math Biophys 9(3):127–147 Ricaurte P (2019) Data epistemologies, the coloniality of power, and
McQuillan D (2018a) Manifesto on algorithmic humanitarianism. Pre- resistance. Television & New Media, 7 Mar 2019
sented at the symposium reimagining digital humanitarianism, Rosenblatt F (1957) The perceptron: a perceiving and recognizing
Goldsmiths, University of London, February 1 automaton. Cornell Aeronautical Laboratory Report 85-460-1
McQuillan D (2018b) People’s councils for ethical machine learning. Samadi S, Tantipongpipat U, Morgenstern JH, Singh M, Vempala S
Soc Media Soc 4(2):3 (2018) The price of fair pca: one extra dimension. In: Advances in
Mezzadra S, Neilson B (2019) The politics of operations: excavating neural information processing systems, pp 10976–10987
contemporary capitalism. Duke University Press, Durham Schaffer S (1994) Babbage’s intelligence: calculating engines and the
Mitchell M (2019) Artificial intelligence: a guide for thinking humans. factory system. Crit Inq 21(1):203–227
Penguin, London Schmidt FA (2019) Crowdsourced production of AI training data: how
Mordvintsev A, Olah C, Tyka M (2015) Inceptionism: going deeper human workers teach self-driving cars to see. Hans-Böckler-Stif-
into neural networks. https​://ai.googl​eblog​.com/2015/06/incep​ tung, Düsseldorf
tioni​sm-going​-deepe​r-into-neura​l.html. Accessed 17 June 2015 Standage T (2016) The return of the machinery question. The Econo-
Moretti F (2013) Distant reading. Verso, London mist, 23 June 2016
Morgulis N et al (2019) Fooling a real car with adversarial traffic signs. Winner L (2001) Autonomous technology: technics-out-of-control as
arXiv preprint. https:​ //arxiv.​ org/abs/1907.00374.​ Accessed 30 Apr a theme in political thought. MIT Press, Cambridge
2020 Zuboff S (2019) The age of surveillance capitalism: the fight for a
Murgia M (2019) Who’s using your face? The ugly truth about facial human future at the new frontier of power. Profile Books, London
recognition. Financial Times, 19 Apr 2019
O’Neil C (2016) Weapons of math destruction, 9th edn. Broadway Publisher’s Note Springer Nature remains neutral with regard to
Books, New York jurisdictional claims in published maps and institutional affiliations.
Offert F (2020) Neural network cultures panel, transmediale festival
and KIM HfG Karlsruhe. https​://kim.hfg-karls​r uhe.de/event​s/
neura​l-netwo​rk-cultu​res. Accessed 1 Feb 2020

13