Paladyn, Journal of Behavioral Robotics 2020; 11: 199–216

Research Article

Carmen Fernández-Martínez* and Alberto Fernández

AI and recruiting software: Ethical and

legal implications
https://doi.org/10.1515/pjbr-2020-0030 recently, Kapoor and Sherif [2] stated that the business
received February 11, 2019; accepted March 12, 2020 environment is evolving into a more complex system with
Abstract: In this article, we examine the state-of-the- a diverse workforce and that it is essential to work with
art and current applications of artificial intelligence (AI), business intelligence and AI to improve HR practices.
specifically for human resources (HR). We study whether, Traditionally devoid of bias, AI proved very valuable for
due to the experimental state of the algorithms used and résumé and keywords scanning and the extraction of
the nature of training and test samples, a further control candidate skills. There has been a recent trend towards
and auditing in the research community is necessary to video-interview analysis in HR. Researchers such as
guarantee fair and accurate results. In particular, we Strohmeier and Piazza have noted that there is a lack of
identify the positive and negative consequences of the comprehensive studies on the potential of AI in the field
usage of video-interview analysis via AI in recruiting [3]. A survey by Personnel Today found that 38% of
processes as well as the main machine learning techni- enterprises are already using AI in their workplace with
ques used and their degrees of efficiency. We focus on 62% expected to be using it by 2018 [4].
some controversial characteristics that could lead to Nowadays, the idea that AI will replace humans in
ethical and legal consequences for candidates, companies the workplace is widespread. A job consists of a number
and states regarding discrimination in the job market (e.g. of tasks. There is evidence from research on the
gender and race). There is a lack of regulation and a need practicality of labour displacement due to automatiza-
for external and neutral auditing for the type of analyses tion and robots [5]. In HR, more and more tasks are now
done in interviews. We present a multi-agent architecture fully automatized and powered by algorithms, though
that aims at total legal compliance and more effective HR still relying on human recruiters to make the final
processes management. decision of who is interviewed on-site or who is hired.
This matter has reduced candidate time and the
Keywords: domain-specific AI, ethics, human resources, recruitment process significantly.
recruiting The demands of a global society and the excessive
recruiting time for HR managers in multinational
companies have changed the traditional way of inter-
1 Introduction viewing. The field is now leaning towards algorithms
that produce rankings of candidates according to
In recent years, artificial intelligence (AI) has been different features, and several commercial products are
increasingly used to support recruiting and human being used extensively in HR departments.
resources (HR) departments. AI has transformed several Also, large multinationals have recognized the value
sectors of the society, but the impact has been largely of several video-interview products and companies:
unfelt in HR, where AI has had no more than an assistant • HireVue:¹ Start-up incorporated in 2004 in Utah (USA).
role so far. Over decades, authors have reinforced the Its leading voice and face recognition software com-
concept of strategic and global HR management [1]. More pares and grades candidate’s word choice, tone and
facial movements with the body language and voca-
bularies of a company’s best former employees. Using

models, HireVue analyses thousands of features and
* Corresponding author: Carmen Fernández-Martínez, CETINIA,
University Rey Juan Carlos, Madrid, Spain, their correlation with the company’s top performers.
e-mail: [email protected]
Alberto Fernández: CETINIA, University Rey Juan Carlos, Madrid, 
Spain, e-mail: [email protected] 1 https://www.hirevue.com/

Open Access. © 2020 Carmen Fernández-Martínez and Alberto Fernández, published by De Gruyter. This work is licensed under the Creative
Commons Attribution 4.0 Public License.
200  Carmen Fernández-Martínez and Alberto Fernández

Finally, it produces a ranking list and an insight score After careful analysis of systems and techniques
(0–100) for candidates. The final recruiting decision is in oriented to extract qualities and features in employees, in
the hands of human recruiting. The system is supported this work we discuss on some of them presumably
by a patented deep-learning analytics engine named controversial, both explicit features like physical appear-
HireVue Iris™. ance, voice, intonation, keywords and race and implicit
• Montage,² SparkHire,³ Wepow:⁴ Mobile and video traits such as optimism, customer-oriented personality,
interview software or SAAS (Software as a Service). sexual identity and similarity to former employees. These
There are preferences for the choice of a video- advances bring a broad range of ethical issues – accuracy
interview product depending on the geographic area. and privacy – for both candidates and companies who
For example, the customer base of SparkHire is trust a particular recruiting product to assist in the
predominantly based in the United States of America recruitment process in search of efficiency. The domain-
(USA), whereas Wepow has a bigger market in Europe specific AI for HR research covers a whole spectrum of
and some sectors like pharmaceuticals.⁵ disciplines and backgrounds, including cognitive science
and psychology and artificial vision.
• Blizz:⁶ Cloud-based tool for video interviewing offering
The primary aim of our work is to analyse the
the possibility of secure chat and two-way HD video
problem of recruiting and working towards a proposal of
and audio. It was launched in 2013.
automated multi-agent software architecture to support
• Human:⁷ Human, a start-up founded in London in auditing HR. We have designed the architecture as
2016, analyses subliminal emotions in video-based job general as possible in order to distribute different tasks
applications in order to predict human behaviour. of recruitment and auditing processes in different agents
• Affectiva:⁸ Worldwide emotion database with over five and also allowing the surveillance of different types
million faces that provides emotion measurement of scenarios that require user’s approval and legal
technologies, the basis for applications of image proces- checks.
sing in HR. In these software products, it is essential to The rest of the article is organized as follows. We
define the traits and facial gestures that are related to first present (Section 2) some controversial features
each competency assessed in a recruitment process. frequently measured in image and voice recognition
Some other features measured by Affectiva are listed in using AI techniques. In Section 3, we present basic
categories or ranks, e.g. age (18–24, 25–34, 35–44, 45–54, information about machine learning (ML) techniques to
55–64, 65+, under_18, unknown). Affectiva can track detect the above-mentioned controversial features.
emotions such as anger, contempt, disgust, engagement, Section 4 focuses on the advantages and disadvantages
joy, sadness, surprise and valence (a measure of the of AI image analysis frequently used in recruiting. In
positive or negative nature of the recorded person’s Section 5, we discuss the legal and ethical issues related
experience). As an example, engagement implies the to the inclusion of minorities, gender equality and
following set of facial gestures: brow raise, brow furrow, applicants’ right to a fair and informed selection process.
nose wrinkle, lip corner depressor, chin raise, lip pucker, Then, in Section 6, we present a software architecture to
lip press, mouth open, lip suck and smile. support auditing recruiting processes. Section 7 includes
a discussion of the analysis carried out in this article and
This article examines the effects of new trends in AI some future lines for AI-based recruiting algorithms. We
and algorithms in HR. We analyse the implications of conclude the article with Section 8.
video-interview analysis and ranking of candidates, the
inclusion of minorities in a diverse pool of candidates
and, finally, the effectiveness of recruitment algorithms 2 Controversial features of image
under different legislation.
and voice processing

2 https://www.montagetalent.com/ Recently, there has been some critical analysis of advanced
3 https://www.sparkhire.com/ AI and image processing and face recognition. A recent
4 https://www.wepow.com/es/
Stanford study [6] claimed that it is possible to guess sexual
5 https://siftery.com/product-comparison/spark-hire-vs-wepow
6 https://www.blizz.com/en/
identity and its traits through face recognition technology.⁹
7 https://citysail.co.uk/ The study had to be reviewed and approved by the
8 https://www.affectiva.com/ American College of Psychologists before publication, given
 AI and recruiting software: ethical and legal implications  201

the potential danger and ethical concerns it raised. This were gathered from sites like Hotornot,¹⁰ where users can
reminds us of the pseudoscience level of some of these rate females, choosing the over-average voted photos for
studies that could be considered similar to phrenology, the data set.
an outdated theory that connects skull, head and traits
with behaviour, personality and criminal tendencies.
The idea goes beyond morphology and questioned 2.2 Age
issues related to grooming, e.g. using less eye makeup
or less revealing clothes as evidence of homosexuality in Regardless of age, there should be some inspection of
women. This idea leads us to mention the concept “soft enterprises’ policies to avoid a preference for employees of
biometrics”, the analysis of body geometry, marks and a particular age group. Ageing follows the same patterns in
tattoos that, unlike primary biometric traits, can be every individual. Some of them, like the appearance of
obtained at a distance without cooperation [7]. Several wrinkles, can be delayed as long as the person follows
researchers [8–11] studied up to what point it is possible some cosmetic procedure, e.g. Botox or fillers. However,
to infer and model race in face recognition. “How some changes like thinning and the decrease in dental arcs
implicit, non-declarative racial category can be concep- cannot be avoided easily. It is easy to study morphologic
tually modelled and quantitatively inferred from the changes in faces and determine an approximate age. The
face?” as suggested in the survey on race analysis by issue of querying candidate age is forbidden in many
Siyao et al. [8]. They mention implicit race detection and countries to promote the inclusion of all citizens.
categorization and learning race implicitly from the face, Next, we include an analysis of the basic features,
going beyond empirical demonstration. Racial categories lines, wrinkles and morphologic changes in face measured
should not be taken for granted, so they propose a in image processing to detect age, as appear in recent
proper race categorization model following qualitative research such as Hayashi (Table 1) and colleagues
analyses (of features and so on) and not quantitative. [14,15] and research on age and gender using convolu-
Other significant studies concentrate on a particular tional neural networks [16]. Recently, Yoti [17] published
geographical group [9–11]. a White Paper related to the application of facial
recognition technology and age tracking with AI. It was
already suggested for its use in retail and supermarkets
2.1 Physical attractiveness in the UK and possibly hiring. The system relies on
indicators of age and screens out applicants according to
Facial symmetry is especially important when it comes a 10-year error margin. For example, screening out over
to detecting physical attractiveness and beauty [12]. For 28’s when it comes to detecting underage customers and
example, a beautiful person would have an intraocular proceeding with the request of ID cards for alcohol
distance that is coherent with eyes size, a good ratio face drinks purchase.
width/height and a mouth width proportional to the rest
of the face.
These observations pose different questions about 2.3 Race
the legitimacy of measuring the beauty of job appli-
cants – especially in countries where CVs without photos One of the most controversial characteristics that could be
are preferable. analysed with video interviewing is race. It is unfair, and it
Studies to evaluate facial beauty are common in the must be said that most algorithms and classifiers are
scientific community [12,13], and some apps have trained with images of White people, not performing well
become popular among Internet users. ML is the key to with Black people. As Buolamwini and Gebru [18] pointed
this kind of automatic evaluation and is supported by out, there is enormous controversy regarding the inclusion
subsets of training and test data. According to automatic of racial discrimination in algorithms. It has been shown
evaluation of facial attractiveness, the seed training data that there are soap dispensers that detect easier White
people than others [19]. Recent research and theories on
analysis of race, like Siyao et al. [8], recall about the


9 https://www.washingtonpost.com/news/morning-mix/wp/2017/
09/12/researchers-use-facial-recognition-tools-to-predict-sexuality- 
lgbt-groups-arent-happy/?utm_term=.524d8912daab 10 www.hotornot.com
202  Carmen Fernández-Martínez and Alberto Fernández

Table 1: Wrinkle progression with age to mention some recent experiments carried out in Italy
and the USA. Both studies needed ethical supervision.
Event type Description Kosinski and Wangs’ work [6] shows theories based
10 years male None on a morphologic study of homosexuality. The study
10 years female None argues that there is a relationship between homosexu-
20 years male Under the eyes ality and exposure to a particular concentration of
20 years female None hormones in the womb and that morphological features
30 years male Under the eyes, at the corner of the eye
can determine the sexual orientation (for example, more
30 years female Under the eyes
40 years male Under the eyes, at the corner of the eye,
masculine or feminine jawline and forehead). This
on the cheek research was contrasted with what users disclose in
40 years female Under the eyes, at the corner of the eye dating applications. The opaque and too invasive nature
50 years male Under the eyes, at the corner of the eye, of this research required advice and permission of the
on the cheek, on the brow American College of Psychologists before publication.
50 years female Under the eyes, at the corner of the eye,
The voice-based categorization study [25] reminds
on the cheek, on the brow
us, however, that every language has its particularities
The table was extracted from Hayashi et al. (2002) [14]. and that a voice that sounds more feminine in a man or
vice versa may be due to anatomical reasons or more
complexities of facial recognition of non-White indivi- exposure to feminine voices during childhood. The
duals, especially women, during the night or with bad sexual orientation guessing proved to be more explicit
light conditions. This survey argues that the development for listeners in Italian or Romanic languages, but the
of systematic methods for race detection is more difficult in categorization works similarly both in German and in
comparison with other attributes. Collecting sample images Italian, where some gay speakers were consistently
for races is not that easy on the grounds of prejudice. The misidentified as heterosexual and vice versa.
analysis is more qualitative than quantitative, and the
definition is ambiguous for the race. How many racial
categories do exist? Whereas for other facial analysis, the 2.5 Use of keywords in speech-affirmative/
categories are known beforehand: one-to-one matching in negative connotation of words
face detection, two genders and a closed list of six
universal emotion types [8]. The lack of accuracy in a AI systems have been able to detect the frequent words
subgroup of dark-skinned women stands out in the that could identify an optimistic and convincing sales-
study by Muthukumar et al. [20]. Other authors [21,22] person. Since every language has its connotations, and
have noted the difficulties in face detection and race in the use of more or less optimistic phrasal constructions,
unconstrained captures, especially the blurred ones. We proverbs and popular expressions could lead to mis-
recall briefly that ML techniques were used for these interpretations of what candidates want to express or
systems. They were initially designed for White people generalize to the English language.
or trained predominantly with images of White indivi- According to recent sources [26], Google translator
duals, both training and test data sets. We need to carry still kept gender bias until the end of 2018 and is
out further studies to include accuracy concerning race currently working towards removing it. The last updates
for law enforcement algorithms, mainly when law include multiple translations for neutral meanings.¹¹ For
enforcement authorities are excessively controlling the instance, so far it links “she” to nouns and adjectives
Afro-American and non-White community in countries traditionally feminine (e.g. “nurse”, “teacher”), “he” with
like the USA. Past works have claimed inaccuracies in “hardworking”, the Chinese pronoun to refer to “they” is
the face recognition systems used by police [23] and naturally translated into English as masculine “he”, etc.
surveillance cameras [24]. According to Leavy [27], the presence of a majority of men
in the design of these technologies is the root of the
problem. These male-dominated environments could
2.4 Analysis of gender and sexual unbalance the percentage of masculine and feminine
orientation in images and voice

As an illustration of the advances of sexual orientation 11 https://www.blog.google/products/translate/reducing-gender-
recognition both in images and in sound, it is important bias-google-translate/
 AI and recruiting software: ethical and legal implications  203

words. There is a gender perspective of translation [28]. authors for real-time gender and ethnicity classification of
Individuals tend to translate neutral words according to videos in Shi and Tomasi [32].
their own gender. This is very often the case of automatic A study mentioned above targeting sexual orienta-
translations involving Chinese language. tion [32] gives importance to the ratio of the forehead
area, whereas the aforementioned works apply this
analysis for race detection. Presumably, the ML techni-
3 AI techniques for identifying ques used could be interchangeable.
Concerning physical attractiveness and according to
controversial features relevant authors [13], some main ML algorithms used to
train classifiers are K-nearest neighbours (k-NN), artificial
Following, we give an insight into the existing literature neural networks (ANN) and Adaboost. k-NN is dependent
and the main ML techniques used for face/patterns on choosing the class according to a local neighbourhood
recognition in domain HR, and more specifically video- and offers better results than ANN, where it is difficult to
interview systems over the last few years. Going into understand the interpretation. The philosophy behind
detail, we analyse traits and characteristics, race, gender Adaboost is combining several weak classifiers to make a
and so on, mentioned in the previous sections. strong classifier and offers positive results.
The initial interest in the analysis of surveillance of With detecting race, as with physical attractiveness,
video images and streaming has promoted the current k-NN, ANN and Adaboost are some of the ML algorithms
state-of-the-art in facial recognition related to video- used in race detection. Siyao et al. [8] acknowledged in
interview systems. However, current systems give weight their main paper about race recognition having difficul-
to the specific responses and subtle details from ties in detecting mixed race.
candidates, even considering controversial features like Recognition of emotion has always been crucial in
race or age. While techniques for facial feature analyses domain human–computer interaction, interactive video
multiply and the video-interview systems gather more games or artificial vision for health. As of today, an
data points, related to image or audio analysis, survey essential part of the video-interview analyses is the
and review become important. Before starting, it is feature extraction and emotion recognition tool.
important to acknowledge the heterogeneity and dupli- Abdulsalam et al. [33] acknowledges that an influential
city of studies that call into the question of which ML milestone in the analysis of facial expressions was the
technique is the most efficient. work of Paul Ekman [34] who described a set of six basic
First, it is worth mentioning the research conducted emotions (anger, fear, disgust, happiness, sadness and
by Viola and Jones [29]. Actually, they do not present an surprise) that are universal in terms of expressing and
ML technique but a rapid method to detect a face in an understanding them. According to the comparative
image, namely, an image-processing technique. The study [35], it is important to differentiate among
article sets the ground for new research making use of techniques used for feature extraction and those used
the Adaboost classifier for this purpose. The next step for classification. The main ML techniques used for
would be identifying traits and contours. Masood et al. features are Gabor filters, a histogram of oriented
[30] proposed a normalized forehead area calculation gradients and local binary pattern for feature extraction,
using the Sobel edge detection method [31]. The and for classification, support vector machine, random
following prediction of human ethnicity study [30] forest and nearest neighbour algorithm (k-NN).
acknowledges that some races, Black for instance, have
higher forehead ratios.
It is important to keep in mind that we address an
experimental field. We have given a summary of the main
4 Pros and cons of image
controversial features in image processing in the previous analysing in HR
sections. Even though there is extensive literature on the
matter, we should not generalize on which ML methods are We begin a description of the primary and controversial
the best for every characteristic. According to the previous features that could be measured through image and
literature related to the recognition of human gender, video analyses. An AI-based video interview system
the Viola–Jones is considered an important milestone. could be programmed to gather the following types of
Viola and Jones [29] introduced rectangle or Haar-like features – lighting, eye contact, tone of voice, cadence,
features for rapid face detection and was used by other keywords used (substantial conversation), etc. and infer
204  Carmen Fernández-Martínez and Alberto Fernández

features such as age, or states, e.g. mood and behaviour was initially fed with images from North American, more
(eccentric, movement or quite calmed and not talkative). concretely Super Bowl¹⁴ viewers, expressing different
Nowadays, there is much emphasis on emotions. They emotions, in English, as a matter of fact, rich in intonations
could detect specific personality traits that employers too. Thereby, it is difficult to guarantee that other races and
want in their teams, sign of customer oriented or cultures – like Asians – show the same level of expres-
salesperson. Recently, both companies and researchers siveness as North American, traditionally shown as more
have worked on emotions and sentiment detection, e.g. expressive and talkative than other cultures. This could be
company Affectiva, Dehghan et al. [36] or algorithm mistaken by inaction or lack of drive – e.g. necessary for
Emotionet [37]. commercial roles. There are recent works related to
Some benefits of domain-specific AI for HR and emotions detection such as Deghan et al. [36] that labelled
more specifically AI-powered video interviewing, are as photos with up to seven emotional levels, “happy”, “sad”,
follows: “neutral”, “disgusted”, “fearful”, “angry”, “surprised”. It is
• Time. Reducing selection process time and candidate understood that the categories are a subset not general-
time/travel distances. izable to every culture. Table 2 shows the main problems
identified.
• Recruiting task easier. Possibility of reviewing videos
and interviews several times.
• Customized candidate experience and customized ques- 5 Ethical and legal implications of
tions. Possibility of repeating and clarifying questions.
HR managers can customize questions and answers AI in recruiting
and set an individual score for each trait too.
In Section 4, we approached the pros and cons of
• Attention to detail. Systems like HireVue can detect up to
recruiting software. The pros are related to technical
25,000 human feature data points in each interview.¹² It
limitations, user cognitive experience and HR business
can outperform the HR recruiter in the number of
processes. Next, we focus mainly on ethical and legal
characteristics to be analysed simultaneously.
implications of software, not only caused by software
• Attention to eye time, emotions/intonation and body limitations but also related to companies’ deliberated
language crucial for customer-facing roles. choices for interview design.
• Lack of human bias related to physical appearance, The analysis of some features poses challenges for
accessories, tattoos and regional accents. Recruiter bias has governments and regulatory entities. In certain countries, it is
been empirically examined in previous works [38]. If well illegal to ask some questions to the candidates (e.g. age) and
designed, the system should favour a focus on responses the candidates should not include personal details, like the
and keywords rather than being influenced by such bias. birthdate, in their resumes. They are asked to fill informative
questionnaires asking about ethnicity, gender, sexual orien-
Concerning problems that accompany the use of these tation or upbringing at the end of the selection process. It is
technologies, it is important to highlight the imprecision of advisable to look at frequently asked questions in the USA
this technology and gender and racial bias. That prevents interview settings that are illegal [39]. HR managers often ask
of hiring intuitively or causes selection of individuals of questions on issues such as marital status, children, previous
similar characteristics to the initial training set. These arrests, country of origin, first language, debt status, social
observations pose challenges if the predicting AI is fed drinking and military discharges. The anti-discrimination
with data based on, for instance, top performers, blue-eyed laws cover different sectors, from employment to insurance,
blonde candidate residents in London and close-to-native housing and banking [40].
English. Over time, companies and start-ups will develop Although law scholars have used the term discrimi-
their own AI algorithms. For the time being, they are nation by proxy extensively for years, we are seeing a
dependent on limited public databases that do not stand trend on the analysis of intentional and unintentional
out for their diversity. Database Affectiva,¹³ for instance, proxy discrimination related to the effects of AI and big
data analysis [40,41]. Next, we describe in brief the
possibility of discrimination by proxy, but it is

12 https://www.hirevue.com/blog/assessments-and-big-data-can-
we-predict-better-hires 
13 https://www.affectiva.com/ 14 https://www.nfl.com/super-bowl
 AI and recruiting software: ethical and legal implications  205

Table 2: Problem types of image analysis in video interviewing

Problem type Description

Candidates are unfamiliar with In selection processes, there are two central values for candidates: trust and
video-interviewing analysis information. The candidates are usually unfamiliar with this type of technology and the
settings to perform well in the interview and maximize success rate, even though they
are given the option of exploring the tool beforehand. Some gestures like looking away
from camera are certainly mistaken for global performance. Self-confidence is
compromised due to a distorted image in smartphones

Imprecisions of technology and imperfect The technology does not reflect reality at a maximum. AI is imprecise because a limited
training sets and somehow biased data set supports it. The first bias could influence the algorithm
in subsequent iterations

Gender and nationality bias Google is deemed to be biased towards men’s voices. When detecting gay/straight,
the cues and sounds of some languages/accents are more prone to be considered
homosexual by listeners and classifiers

Training classifiers with too generalist Very generalist data sets were used for the training test, for instance, human. The
and broad data sets companies use “proprietary data” from former employees who performed well for
contrasting profiles, leading to endogamy and unfair results. Racially biased learning
algorithms constitute a fundamental problem in computer science too. Problems of
race and mixed-race recognition

Technical limitations The inaccuracy of race detection (or skin colour) in AI analysis if the flashlight falls
upon directly on the face or with night conditions. Discrimination based on actual
races without ever knowing it

understandable that it also leads to unfair decisions in discrimination in automatic decision support systems
HR. Proxy discrimination means to discriminate in favour (DSS), used for screening purposes in social-sensitive
or against a protected class or a protected group (e.g. tasks, including access to credit or mortgage, tracking if
race, people with higher incomes and so on) but based the input (innocuous candidate data like ZIP) could lead
first on legitimate or innocuous grounds (e.g. zip code). to a wrong yes/no decision. Hajian and Domingo-Ferrer
These reasons for discrimination are highly correlated [44] propose techniques which are effective at removing
with belonging to a protected class (e.g. racial minorities direct and/or indirect discrimination biases. So they
living in the same areas or ghettos). So this segregation reinforce Pedreschi’s proposal, in their paper, of
finishes affecting the protected groups and being the inducting patterns that avoid discriminatory decisions
reason for more discrimination. When there is a human despite drawing from training-biased data sets.
actor behind this favourable treatment or less favourable, These cases lead us to think how future generations
we speak of “intentional” discrimination by proxy [40]. of students and candidates should prepare for these
However, the nature and mix of the data sets used for interviews and how the HR field will develop?
testing and training could also lead to “unintentional” Companies are given the option of customizing their
discrimination by proxy. Relying on external sources systems and have their own agency for selecting the
often causes more algorithm bias. It is often the case that features of their future employees. Would it be ethical to
the company holds that they can justify a practice as a classify men or another subset in the first place? This
business necessity if it is a predictor of future perfor- idea of black box assessment does not help in the
mance and therefore is not infringing equal employment empowerment of some collectives, which could be
legislation. Even though the practices resemble historic marginalized or be given unfair treatment in the round
patterns of discrimination or use limited data sets [42]. of interviews.
It is noted that algorithms can detect, and some- Even though reports have shown that the results of
times protect against, indirect discrimination resulting HireVue systems are highly diverse and show even more
from automated analyses. Pedreschi et al. [43] describe presence of non-White candidates and gender equality,
well in their work the possibility of finding evidence of there is an element of disinformation about what exactly
206  Carmen Fernández-Martínez and Alberto Fernández

this software targets for every position, junior, manage- employer discretion or based on arbitrary or temporary
rial and so forth. This is in line with the arguments employment. Sexual orientation detection follows si-
discussed in the previous section. Not only does milar patterns. In this light, some critics – including the
technology bring with it drawbacks related to the American society of psychologists – have issued a review
technical limitations and biased data sets, but also the before permitting the publication of the work of Wang
disinformation and the prejudices worsen the problem. and M. Kosinski [6], arguing that it is illegal and against
Legal protection related to information gathering from the fundamental civil rights to use this kind of software.
minorities or protected groups causes more inaccuracy.
Typical and classic algorithms in computer science have
a better effect in White people, just because the 6 Towards automated support for
technology was initially created in their origins by
White men and it was tested initially with a majority of auditing HR
White people.
Another issue of debate is the legal implications of It is evident that the legal issues concerning candidates,
some analysis. Image recognition technology is so such as the identification of race and sexual orientation
widely developed to the point that it could even identify in the selection process due to the advances in image
age or sexual orientation of a candidate through the processing, entail ethical questions that cannot be
analysis of their facial features [6,45,46]. As mentioned ignored. Careful analysis from auditing bodies, govern-
earlier, Google is deemed to be biased towards men ments, ethics committees and psychologies is needed.
voices, which questions the legal validity of most of In this section, we describe a proposal of a multi-agent
these advances. Algorithms could be tuned to detect real software architecture for auditing, which is depicted in
age according to the biological factors, skin type, Figure 1. Coordination [47] and interoperability of agents
baldness and so on. Under Spanish law, it is possible in heterogeneous domains have been widely used in
to pay fewer taxes for under 30-year-olds and over different domains, such as healthcare [48], emergencies
40-year-olds. For cost-saving factors, AI could be tuned due to natural disasters [49], smart cities [50], etc. Multi-
to discriminate against senior people or young people, agent systems (MASs) have also been used to improve
could seek for women or minorities or reject them at business processes in complex organizations. Practicalities

Figure 1: Multi-agent system architecture for HR auditing.

 AI and recruiting software: ethical and legal implications  207

of big multinational corporations and the difference in The additional auditing checks are requested on demand
formats and regulations make it necessary to create by internal auditing checks or carried out by agents
architectures and models to rule over changing specific without prior request. We intend it to consider normative
agent populations. Architectures have been proposed to reasoning to regulate human and artificial agents.
deal with the random and quick changes in a particular Arguably, we assumed that agents behave as they should.
productive section, such as the manufacturing industry, It is highly recommendable to consider normative MASs
but could be applied to other dynamic corporate environ- or normative reasoning for cases like the one we handle.
ments [51]. The recruiting scenario in a multinational The interview auditing agent requires the approval of
context is also quick and complex and needs models. So candidates in case of law infringement. Once all
far there is not much literature related to applications of necessary checks are completed, the information flows
HR and MAS as an enabling technology, but these types of back to the selection process agent responsible for
architectures have been extensively used, as noted above, closing the selection process or cancelling it on legal/
in manufacturing and corporate control production (e.g. ethical grounds. The candidate data agent could conduct
Ciortea et al. [52]). The domain of law is open to MAS some additional checks at their own risk, though.
applications and has been addressed over the last decade The workflow of data analysis related to agents,
by law scholars such as Walker [53]. managers, auditors and authorities (e.g. legal and
Figure 1 shows an abstract MAS architecture that must nationality checks) is fundamental to guarantee inter-
be adaptable to different international corporate environ- national legal compliance. The final aim is keeping the
ments and by recruiters of different nationalities in search of system as efficient, transparent and neutral as possible,
international compliance. The challenge of storing legal i.e. reduce or mitigate racial and gender discrimination
knowledge and doing sound checks and reasoning becomes and cases where the company wants to profit certain
complicated in cross-jurisdictional cases. legal benefits by applying laws from the country where it
It requires full interoperability of all the agents to is incorporated and not the laws of the country where it
achieve a functional system. The core of the architecture is recruiting. An interesting fact of employment law is
comprises three different parties that must collaborate: that some countries, like Brazil, keep certain jobs for
(i) a recruiter/company, (ii) an external auditor and (iii) their nationalities. It is understandable that there are
(if necessary) the government/authorities of the country many necessary checks to guarantee a fair process.
where the company is incorporated or recruiting and the
country of origin of the candidate. Our vision of auditing
system targets varied selection processes and works the 6.1 Interview design agent
same way for different job categories, from technical to
managerial positions. Nevertheless, it should be noted In order to design and compare different and fair inter-
that video-interview systems are specially designed to views, an interview design agent is required. The design
reduce candidate time and shortlist candidates in selec- agent is located in the company’s main headquarters.
tion processes with thousands of applicants. These According to the company requirements, some interview
situations occur with more frequency in the lowest levels profiles could be designed to be recruited after that
of the business hierarchy, e.g. selection of interns of internationally (e.g. software engineer, marketing expert).
graduates. Managers are often pointed out by head- The input of the agent would be the specific characteristics
hunters and directly selected by the board of directors. AI of the open position, controversial or not (e.g. experience,
in HR can be useful to create models that predict commercial role, age > 40, gender = woman, English/
successful promotions among the working staff though. Spanish native speaker). The output would be the interview
There is no specific hierarchy among all the agents and questions in the natural language as well as a structured
no co-dependencies. However, the interview design agent format for a particular role together with scores for possible
seems responsible for starting the selection process answers and rankings expected for the role (optimism 50%,
and taking first steps by making public an offer. It is a commercial profile 60%, etc.). It is expected that the
precursor to an interview in the same way the selection complete implementation will require the use of technol-
process agent is responsible for closing one. However, ogies such as Resource Description Framework (RDF) or
some doubts might arise about how human recruiters and ontologies as a starting point to define and harmonize the
agents interact. Human recruiters and auditors delegate formats. Once the interview has been designed, it is
some of their duties and checks on agents, and additional submitted to the database to be reviewed thereafter by
reasoning or test could be done by agents on their own. the interview in-company auditing agent.
208  Carmen Fernández-Martínez and Alberto Fernández

6.2 Interview in-company auditing agent a candidate denies confirmation or does not give consent
to the data being used in a particular way.
The interview in-company auditing agent manages the
selection processes in different branches. Human recrui-
ters select a particular generalist interview from the 6.5 Labour law external auditing agent
corporate database for a particular role, for instance, a
The human auditor or the internal auditing agent could
software analyst. It applies the general format to every
optionally proceed with legal tests. The analyses are very
regional scenario, e.g. Europe, Asia Pacific and the
convenient since the labour law agent runs different
Middle East, adapting the prospective candidate criteria
checks in different legislation, specifically focusing on
to be selected to the culture of the recruiting company’s
employment laws. For the proposed architecture, we
country and local culture. This agent controls interac-
focus on the US and Spain Labour Law. There will be a
tions among human recruiters and decides if it is
database storing the laws, conveniently represented in a
essential to pass interviews to additional stages if they
rule-based format. In particular, we chose an if-then rule
handle controversial data. This agent receives an inter-
format condition ⇒ decision, where conditions is a
view in a structured format (later redistributed among
logical formula and decision can be (i) include, (ii)
the HR team) and returns whether the process requires
warning or (iii) exclude. If a rule is fired (conditions =
auditing. If the information needs further validation, it is
true) with decision = exclude, then it means the
passed to other agents. All the legal and ethical
exclusion of the selection process, while the other two
transactions are hosted outside of the company, in
possible decisions mean continuity. Warning decision is
external auditing bodies and governmental premises.
useful for future reporting, auditing and statistical
The recruiting process could end at this point if the
purposes. In the first example below (R1), a warning is
questions are simple and do not need legal handling.
used to signal cases that qualify for benefits (e.g.
younger unemployed candidates).
6.3 Selection process agent We assume that the set of rules in the knowledge
base is consistent. This means there is no possibility
The selection process agent is the central element of the that different rules could be fired (e.g. with conditions
proposed MASs. It is hosted in the company branch where “age < 16” and “age < 60”), obtaining different
the selection process is being held. It is in charge of outcomes with the same input data. However, we allow
processing the different events that occur in the system that situation if they get the same conclusion.
(new interviews, new auditing and legal checks, authorities, Therefore, the knowledge engineer in charge of filling
etc.), triggering new events when necessary and providing a the knowledge bases must be careful of keeping
ranking of candidates to the human interviewer. consistency (she might be supported by a consistency
It could occasionally reuse interviews from other checking or editing tool).
companies; it coordinates different companies, external The following rules consider both fiscal benefits for
neutral auditors and authorities, but only if necessary. enterprises according to citizen groups and laws
In case no external auditing is needed – no controversial infringement in Spain. Warnings signalize situations
issues are found – the selection process agent provides likely to qualify for benefit or not benefits that should be
direct feedback of the candidate. reviewed afterwards:
R1: Age < 30 ∧ first job ⇒ warning
R2: Age > 45 ∧ unemployed ⇒ warning
6.4 Candidate data check agent R3: Terrorism victim ⇒ warning
R4: Gender violence victim ⇒ warning
Once a candidate dossier arrives at the external auditors, R5: Age < 30 ∧ first job ⇒ warning
it may happen that, due to the sensitivity or inaccuracy The labour law agent allows to adapt to new legal
of the data, some confirmation or permission from the scenarios considering multiple countries and jurisdic-
candidate is required to proceed with the data handling. tions. In the case of law infringement, it would inform
Since the main purpose of this MAS is to assess the the main agent, i.e. the selection process agent. In the
selection process in each phase, the candidate data case of American legislation, which is particularly
check agent would inform the company side promptly if protective of racial discrimination, any subtle reference
 AI and recruiting software: ethical and legal implications  209

to race in a hiring process and exclude in one iteration the preselection of individuals by gender or age merely
could be marked to be watched (warning), e.g.: based on fiscal benefit for the employers and positive
R1: Non-White ∧ rejection ⇒ warning discrimination on economic grounds. Likewise, the fol-
R2: Mixed race ∧ rejection ⇒ warning lowing typology of rules could have statistical and reporting
R3: Minority ∧ rejection ⇒ warning purposes, that is, recording the number of candidates
One of the main objectives of an auditing will be the eligible for each type of benefit or reduction of tax rate:
legal check carried out by the labour law agent when R1: Age < 30 ∧ first job ⇒ warning
necessary. The in-company auditing agent passes R2: Age > = 45 ∧ long-term unemployment ⇒ warning
information to the labour law agent only when there R3: Gender = Woman ∧ domestic violence victim ⇒
are suspicions of legal infringements in the recruitment warning
processes. R4: Terrorism victim ⇒ warning
Typically, one of the first analyses to be carried out In our work, we have resolved to include a rule
is checking the legal age of candidates and then the engine to implement the legal classifier with up to 100
analysis proceeds more accurately on advanced criteria rules so far organized in rules’ files specifically for two
to avoid discrimination. The first cut will be based on countries at first: Spain and the USA. When it comes to
valid reasons. For example, checking if the candidate preselecting which legislation could be considered being
has the minimum age to access the job market in that the most practical subset. The USA and Spain are two
country or work permit. major examples. These countries present some of the
The Spanish Labour Law is more lenient for other most extensive ranges of codes, statutes and formal
age prerequisites. In Spain, it is possible to be working sources that consider “protected attributes” [41]. Spain
past the retirement age (active retirement, a formula for offers subsidies for business owners that hire workers
both being a pensioner and earning a reduced salary according to protected attributes, such as age. It
simultaneously). We formalize this fragment of law with constitutes positive discrimination to easy access to the
the following rules: job market to some disadvantaged collectives, like
R1: Age > = 18 ∧ Age < = 60 ⇒ include women and young unemployed people. In the complex
R2: Age < 18 ∧ emancipated ∧ consent parents/tutor case of the USA, several instances of discrimination are
⇒ include prohibited by law. The case of race must be especially
R3: Age < 18 ∧ emancipated ∧ authorized ⇒ exclude highlighted. Title VII of the US Civil Rights Act prohibits
R4: Age < 18 ∧ emancipated ∧ authorized mother ∧ making employment decisions based on race, sex and
consent father ⇒ include other protected attributes. However, it does not prohibit
R5: Age < 18 ∧ emancipated ∧ authorized father ∧ the interviewer from knowing beforehand the race of the
consent mother ⇒ include candidate, only making discriminatory decisions. Luckily,
R6: Age < 18 ∧ emancipated ∧ total orphan ∧ the use of protected attributes for decisions in wider
authorized legal tutor ⇒ include society, beyond the workplace, like housing and credit
R7: Age < 18 ∧ emancipated ∧ total orphan ∧ decisions, is being restricted progressively by law [41]. In
authorized legal tutor ⇒ exclude the UK, the Equality Act 2010 and further legislation have
R8: Age < 18 ∧ emancipated ∧ partial orphan ∧ a protective role too.
authorized mother ⇒ include Therefore, we narrowed down our analysis to the US
R9: Age < 18 ∧ emancipated ∧ partial orphan ∧ and Spain cases, with an intent to cover the Brazil case
authorized father ⇒ include soon, which makes up one of the most interesting cases
R10: Age < 18 ∧ emancipated ∧ partial orphan ∧ in labour law worldwide, with several national laws that
authorized mother ⇒ exclude give preference to Brazilian nationals for employment.
R11: Age < 18 ∧ emancipated ∧ partial orphan ∧
authorized father ⇒ exclude
R12: Age > 60 ∧ legally retired ⇒ exclude 7 Ethical agent
R13: Age > 60 ∧ legally retired ⇒ include
R14: Age > 60 ∧ legally retired ∧ active retirement In the case the user data are very controversial and
status ⇒ include address ethical issues, e.g. the interview format contains
The legal agent could trigger warnings as well. It could terms or analyses traits designed to check age or sexual
proceed with integrity checks in order to detect the granting orientation, the internal auditing agent would pass data
of benefits, illegal or not. As an illustration, it would detect to the ethical agent. This would only happen if enough
210  Carmen Fernández-Martínez and Alberto Fernández

evidence is found to complete an ethical test. The ethical 7.2 System operation
agent carries out a similar analysis to the labour law
agent, following a rule-based approach. The ethical In this section, we include a more detailed explanation
agent could additionally support the analysis with of the choices towards a full legal audit of interviews so
ethical committees and experts to make a verdict on as to give a brief description of the inputs and outputs of
the excessive invasion of interviewee privacy. the systems. The final outcome would be an auditing
The input of this agent would be a request put report. The warning messages and other results of
forward by the internal auditing agent and it returns the agents’ reasoning will be displayed for human and
ethical approval or disapproval (include/exclude or agent auditors during the whole process and used for
include with warnings). The ethical agent would be reporting purposes afterwards.
addressed by the mediation of the interview auditing The system is composed of two different kinds of
agent. In case it checks any subtle irregularity in the agents: corporate agents and auditing and authorities
candidate criteria database, both positive and negative agents. Following this, we offer an insight on how the
discriminations, the auditing agent launches a check agents interact. It is important to mention that all agents
with the ethical agent. The candidate could be accepted work at the same level and are not codependent, and
for legal reasons, the algorithm would include the rule although there is no hierarchy, two agents stand out.
and the possible bias would be detected by means of The interview design agent, the one that comes first,
warning, as the following example: opening the selection process and the selection process
R1: Candidate ∧ homosexual ⇒ warning agent, responsible for finalizing the hiring process. The
R2: Candidate ∧ heterosexual ⇒ warning positive intrinsic characteristic of MASs is the possibility
R3: Candidate ∧ transgender ⇒ warning of initializing different agents without the need of
These three rules would select similarly. First, the restarting the others. A human auditor or recruiter could
interview content is evaluated. If the auditing result shows request a check to the agents and they could also carry
that the terms “sexual orientation”, “heterosexual”, “trans- out reasoning or automated tasks on their own. The
gender”, “homosexual” come up in the interview – excluded ethical and legal analyses are integrated with the rest of
survey or statistical treatment of confidential data – then the the system, but there is a clear separation of roles:
candidate is likely to be preselected or excluded on grounds corporate analysis on one hand and external auditing on
of sexuality. The system launches a warning to signal the other hand. The legal checks will be independent of
evidence of inappropriate interview content. the corporate procedures and only carried out in specific
Sexual orientation and religion could be considered circumstances.
more serious cases of discrimination than other rules of We propose a rule-based system for automatized
the ethical rule base. Tracking the term “race” in an legal and ethical analyses. The bias is suitably detected
interview would be less negligent, given that companies by “warnings”. Candidate exclusion for valid reasons
retain the right to hire candidates of a particular race, for results in an immediate firing of exclusion rules and
example, in the fashion industry or for statistical or notification for posterior dismissal of the recruitment
survey purposes, for example, representation of mino- process. Inclusion with a warning would contribute to
rities in their workforces. negative reporting. After this analysis, the internal
auditing agent would determine if there is enough
evidence for data requests or negative auditing results.
7.1 Authorities agent The central elements of the architecture are the
selection process agent – responsible for closing the
In case a company is expatriated and recruiting inter- selection process – and the internal auditing agent. The
national candidates, an authorities agent coordinates auditing agent will request an ethical and legal check if
different governments’ decisions and must decide necessary. In the rare case that unethical keywords
whether a company must comply with additional pro- appear in the interview template (e.g. “religion”), the
ceedings or if it is needed to check the register of a foreign internal auditing would proceed to request ethical agent
candidate in a city census. The civil servant or company analysis.
could additionally contact with the candidate and request For example, the identification of the word “sexual
the visa, passport or refugee status necessary to move orientation” or “religion” in the designed interview will
forward to other phases in the selection process. suppose the inclusion for valid reasons but the
 AI and recruiting software: ethical and legal implications  211

automatic firing of a warning or negative auditing, Concerning the proposed corrections related to
whereas other words like race will mean firing a warning rules, the use case includes a specification of rules for
for later reviews. For example, it could be possible that a Spanish Labour Law in check. However, the proposed
company actually seeks a Black or Asiatic person, for architecture and system are developed for implementa-
instance, in the modelling or fashion industry, but tion in different legislation and settings, with sharp
supposedly no company should recruit in terms of differences in anti-discrimination laws. Therefore, the
religion and sexual orientation. As a result, the ethical rules included are a limited subset for an illustration of
auditing would end with the result “non-ethical”. The the system versatility in future use cases.
main goal of discrimination-aware data mining research Below, we globally describe the most relevant steps in
is to avoid unlawful discrimination. According to Calders the procedure and workflow, starting with the company’s
and Verwer’s work [54], the topic of the discrimination- decision to open a new position in a branch. Following a
aware classification was first introduced due to the step-by-step timeline, the initial entry of the system is an
observation of unwanted dependencies among the interview format in plain text associated with the particular
attributes. Additionally, the article suggests that split- position. As for the details of the interview, there is an
ting a group of candidates according to a particular important record field: language of the interview and the
feature would favour one group over another. The so- definition of STAC (skills, traits, abilities and characteristics)
called stratification and red-lining effect cause first to questions related to skills.
prioritize minority groups and problems in future
iterations; the classifier then favours one group and 1. The central HR manager creates a position (general
uses features that correlate with these preselected specification) [internal activity] to be applied in
features to apply the same type of rules. different branches, for example:
Sometimes, mechanical application of discrimina-
tion-aware data mining to avoid overall discrimination Example 1.
could lead to miss out genuine occupational require- Level: Analyst.
ments for fashion or companies needs, so, under- Division: Investment banking.
standably, it could be convenient to track sensitive Situation: Asia-Pacific.
attributes for business purposes occasionally. Pedreschi Position: Risk analyst.
et al. [43] deal in-depth with cases of supposed direct
Example 2.
discrimination that could be refuted by the respondent
Level: Manager.
(company) to be lawful. In the examples provided, they
Division: Wealth management.
assume a case that claims for discrimination against
Situation: EMEA.
women among applicants for a job position (e.g. sex =
Position: Wealth manager.
female ∧ city = NYC ⇒ exclude).
The company could support the defence arguing
2. The regional branch recruiter adapts the general
that the rule is an instance of a more general rule:
specification to their specific context and needs. She
drive_truck ∧ city = NYC ⇒ exclude. This is a legitimate
is assisted by the interview design agent to create an
rule and indeed a genuine occupational requirement for
interview script. The interview comprises STAC ques-
a specific job and as a consequence, exclude women
tions. The interview will be defined for a specific
lawfully.
language and country, and there will be three clear
The main objective of the auditory will be indeed the
subcategories in the questions: behavioural questions
legal check carried out by the labour law agent and
(body language and response time), leadership style
ethical check when necessary. It is assumed that the
(suitable for commercial role) and optional corporate
interviews will have a great amount of confidential
questions (willingness to travel, dream job and
information and private responses and often, they will
economic compensation). As described before, these
be processed by the external law agent to make sure
are possible and generalistic categories, but the design
there are no infringements of labour law of the countries
of the interview is at the discretion of the company.
carrying out the process.
Likewise, if the scanning of interview format or 3. The company publishes the call.
candidate data seems to be noncompliant with the basic 4. The selection process agent analyses the interview
regulations, the internal corporate auditing agent would with the aim of detecting major discriminations (e.g.
put forward a request for external legal testing. religion).
212  Carmen Fernández-Martínez and Alberto Fernández

5. Candidate interviews take place and are stored in the The future of AI and its consequences in organiza-
interviews’ database. tional change management should be more inclusive
6. The selection process agent carries out reasoning and informative. Likewise, ML classification methods
based on interview questions and candidate answers. remind us of the importance of the attention to detail. To
It follows an analysis to exclude candidates for valid achieve good hires using ML algorithms training, data
reasons (e.g. non-legal working age). The shortlisted sets should be diverse to support the idea behind
candidates are passed to the next step. learning, i.e. generalization and subsequent application
to new individuals.
7. In-company auditing agent passes information to
In this article, we have presented the pros and cons
labour law auditing agent and ethical agent in case of
of domain-specific AI for HR. It is noteworthy to outline
warning or irregularities of the shortlisted candidates.
some present challenges such as the lack of reasoning
If the candidate data concerning citizenship status need
and inference ability of current domain-specific AI and
to be checked against the analysis, it would be passed on
the fact that it remains in the experimental state. As
to the candidate data check agent. If there is no need for
mentioned above, the training data sets are imperfect as
ethical or legal processing, the internal auditing agent
they lead to an imperfect AI.
closes the analysis and passes the shortlisted candidates
Some future lines regarding domain-specific AI for
to the selection process agent.
HR are the following:
8. The labour law agent, ethical agent and candidate
• Ensuring that ML and domain-specific AI work equally
data request to carry out a sound check if necessary
for many people and are neutral, guaranteeing diverse
at the request of a human external auditor. The
training sets.
candidate data check agent contacts the candidate
self or authorities to confirm citizen or expatriate • Conditions to classify well are uncertain, but there are
status, if necessary. The tests are independent. They conditions to record and perform well technically in an
produce an acknowledge document and give back interview that could avoid false positives (e.g. good
information about the citizen and legal and ethical microphone and distance to camera).
warning to the selection process agent.
• Enhancing candidate consciousness of the machine
9. The selection process agent is responsible for closing hardware and wearables for better results. Informing
the selection process, giving back a report with a candidates of language requirements of the company.
ranking of candidates to the human recruiter. The Well-informed users would avoid misclassification
human recruiter decides the hire depending on the problems.
scores. The final say is at his/her own discretion.
• Fostering proper auditing in domain-specific AI for HR
to avoid manipulative uses. Systems are slightly opaque
so far and not officially audited.
8 Discussion • It is preferably reducing the importance of body
language in candidates to focus on answers. Technical
AI will be a turning point in the previous way of characteristics, such as light and behaviour, are not that
recruiting, especially when combined with cognitive important. Tracking how many times they look away
psychology. It saves candidate time and reduces the from the camera disadvantages the chances of candi-
geographic distance and time of face-to-face interviews. dates who are natural and spontaneous and not used to
It is questionable if scientists and, in particular, direct gaze. Another suggestion for interviewing is
companies can make gradual improvements to the experimenting with different rankings, not just pre-
capabilities of algorithms or ride them to the limit selecting candidates based on top performers.
according to their agenda. It is essential to foster proper
auditing and supervision of these systems, which, in any • Addressing the issue of gender bias. Blind auditions
way, perform well. Technology for recruiting enables the have proved to be useful.
detection of over 11 movements/minute surpassing • Controlling the pace of digital change for small- and
human recruiters. Thereby, wearables or cognitive help medium-size companies. Some are concerned with not
like microphones, and augmented reality glasses will be being innovative enough, but unnecessary adoption of
needed to give candidates the chance to perform well video interviewing could cause business disruptions:
and make up for their individual handicaps. “A 2013 research from Oxford University states that
 AI and recruiting software: ethical and legal implications  213

almost 43% of jobs will be automatized in the USA and applicants by race” in selection processes. There are still
up to 70% in developing countries by 2033”.¹⁵ many territories where the US Employment Law applies,
like American Samoa, Guam, the Commonwealth of the
Despite so many expectations in the field of HR, it should Northern Mariana Islands, Puerto Rico and the US Virgin
be noted that AI is a tool and the ethics of it will always Islands. Even though the recruiter and company are
depend on the person creating the tool. Image processing is foreigners, they should comply with the US Law, which is
very powerful and looks beyond ordinary things. especially protective of discrimination by race and age
In this context, more governance is needed. In a over 40 years.
broader perspective, governments should track selection We have illustrated in the previous sections that
processes if there is an infringement of fundamental current technology enables to detect race easily in images.
employment laws and human rights. In particular, due The discrimination could be subtle, especially when the
to the diverse nature of the global job markets, an ever- selection process is about promoting an employee. The
growing mobility of employees exposed to different same goes with discrimination related to sexual orienta-
approaches in regulations (for instance, some countries tion. The technology has no limits of privacy. Even though
keep some jobs only for their nationals) is expected. a minimum level of fairness is reached concerning the
Concerning the idea of identifying sexual orientation, main attributes or protected classes, like race and gender,
it reminds us of the difficulty level. The cues and sounds it is difficult to assess the side effects and the interaction
in every language vary. The same applies to gender, and among other subclasses and combination of them in ML
males produce shorter vowel duration than females. analyses. The overall fairness is somehow unreachable.
Problems could arise not just with the technicalities Thus, there is growing concern about the limitations of AI
of the sounds for females, males and nationalities. It technology and the effects of “intersectionality”, especially
could be potentially dangerous that a company or state when it becomes mainstream and its use spreads over
can predict the sexual orientation of employees. It is different and miscellaneous fields, from criminal justice to
critical when keeping in mind the number of countries finances [55].
that condemn homosexuality. What ultimately needs to be discussed is if we should
rely entirely on private companies and AI solutions instead
of seeing it for what it is today: a tool with limitations,
9 Conclusion computer-aided recruitment. Technology in the HR has
evolved enormously but often has lost the human aspect.
The conflict behind AI for recruiting is that it relies on In this article, we analysed controversial character-
proprietary products trained with limited data sets. Even istics that are measured by commonly used recruiting
though they offer accuracy to look at certain character- software. This article serves as a ground and summarizes
istics, they were not thought of as mainstream recruiting our research following both a technical and an informa-
tools in their beginnings. As a matter of fact, they are tive approach. It is precise to mention the increment
progressively adopted by large corporations with thou- with respect to our previously published work that offers
sands of candidates for efficiency reasons. The software a brief insight of the proposal [56,57] or focuses more
could not control potentially discriminatory outcomes if on semantic technologies and technicalities of legal
recruitment is carried out by the company under the reasoning [58]. Here we explore in greater detail the
wrong reasons or controlled by non-democratic state, advantages and disadvantages of image analysis during
e.g. being selective against minorities, women, people interviews and discuss their legal and ethical implications.
under or over a certain age, senior citizens, immigrants To overcome these problems, we think auditing should be
or customer with accents. Image processing could even carried out on recruiting processes. We have proposed an
filter candidates by appearance reasons. MAS architecture, so as to support humans on those
This reality is ultimately in a clash with employment auditing procedures. The most notable issue raised by the
laws in most jurisdictions. For example, the US Law is prototype is, being self-critical enough, the limitations of
especially protective of racial discrimination. The Civil the proposal here mentioned, a consequence of differ-
Rights Act 1964 gathers the idea that is forbidden ences in formats and the need of interoperability,
“improperly classifying or segregating employees or proprietary HR software products that do not favour
auditing and lack of access to real corporate HR business
 scenarios. The proposal is therefore limited to basic
15 https://www.oxfordmartin.ox.ac.uk/publications/view/1314 agents’ checks and reasoning. The auditing of AI poses
214  Carmen Fernández-Martínez and Alberto Fernández

challenges that are in line with the current state of the art [5] E. P. Bruun and A. Duka, “Artificial intelligence, jobs and the
of AI technology. future of work: racing with the machines,” Basic Income Stud.,
Taking due account of the inherent limitations of this vol. 13, no. 2, Art. 20180018, 2018, https://doi.org/10.1515/
bis-2018-0018.
proposal and the difficulty to explain a complex system in
[6] Y. Wang and M. Kosinski, “Deep neural networks are more
just one paper, we include a comprehensive description of accurate than humans at detecting sexual orientation from
the system functioning. Open issues are remaining that facial images,” J. Abnorm. Psychol. Soc. Psychol., vol. 114,
could be beneficial, such as the full description of no. 2, pp. 246–257, 2018.
knowledge representation, e.g. formalizing interviews [7] D. A. Reid, S. Samangooei, C. Chen, M. S. Nixon, and A. Ross,
“Soft biometrics for surveillance: an overview,” Handbook of
and job description with ontologies and fostering inter-
Statistics, 2013, pp. 327–352.
operability and a variety of formats. Relying on a common [8] F. Siyao, H. Haibo and H. Zeng-Guang. “Learning race from
vocabulary could smooth the interaction among agents. face: a survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 36,
In this line, we may need to use some extensions to no. 12, pp. 2483–2509, 2014.
the rule format approach we are currently using or even [9] Y. Wang, Y. Feng, H. Liao, J. Luo, and X. Xu, “Do they all look
the same? Deciphering Chinese, Japanese and Koreans by
adopt a different paradigm. For example, priorities can
fine-grained deep learning,” in 2018 IEEE Conference on
be useful to solve rule inconsistencies while keeping
Multimedia Information Processing and Retrieval (MIPR),
rules as simple as possible. Another option that we are Apr. 2018.
considering is having several warning degrees, e.g. the [10] M. C. Roh and S. W. Lee, “Performance analysis of face
higher the number of rules supporting a warning, the recognition algorithms on Korean face database,” Int. J. Pattern
stronger the belief that a warning should be raised. Recognit. Artif. Intell., vol. 21, no. 6, pp. 1017–1033, Sep. 2007.
[11] A. Bastanfard, M. A. Nik, and M. M. Dehshibi, “Iranian face
We continue analysing which agents’ duties could
database with age, pose and expression,” 2007 International
be automatized and semi-automatized to further Conference on Machine Vision, Dec. 2007.
strengthen full automatization. At this point, the system [12] D. Sutic, I. Breskovic, R. Huic, and I. Jukic, “Automatic
is arranged in some instances as a decision-making tool evaluation of facial attractiveness,” in The 33rd International
or a tool to assist decision for HR managers. Some Convention MIPRO, 2010, pp. 1339–1342.
analyses remain semi-automatized. In the future, we [13] J. Gan, L. Li, Y. Zhai, and Y. Liu, “Deep self-taught learning for
facial beauty prediction,” Neurocomputing, vol. 144,
plan to investigate the development of the proposed
pp. 295–303, Nov. 2014.
architecture and test it in operation. [14] J. Hayashi, M. Yasumoto, H. Ito, Y. Niwa, and H. Koshimizu,
“Age and gender estimation from facial image processing,” in
Acknowledgements: This work was partially supported Proceedings of the 41st SICE Annual Conference, 2002,
by the Spanish Ministry of Science, Innovation and pp. 13–18.
[15] S. E. Padme and P. S. Desai, “Estimation of age from face
Universities and co-funded by EU FEDER Funds through
images,” Int. J. Sci. Res., vol. 4, no. 12, pp. 1927–1931, 2015.
Grants TIN2015-65515-C4-4-R and RTI2018-095390-B-C33 [16] G. Levi and T. Hassner, “Age and gender classification using
(MCIU/AEI/FEDER, UE). convolutional neural networks,” in 2015 IEEE Conference on
Computer Vision and Pattern Recognition Workshops
(CVPRW), 2015, pp. 34–42.
[17] Yoti, “White paper. Yoti age scan – public version,” https://
References www.yoti.com/wp-content/uploads/2019/09/Age-Scan-
White-Paper-Executive-Summary-December19.pdf [accessed
[1] N. J. Adler and E. Ghadar, “Strategic human resource manage- March 14, 2020].
ment: a global perspective,” Human Resource Management in [18] J. Boulamwini and T. Gebru, “Gender shades: intersectional
International Comparison, Berlin: Walter de Gruyter, 1990. accuracy disparities in commercial gender classification,” in
[2] B. Kapoor and J. Sherif, “Human resources in an enriched Conference on Fairness, Accountability and Transparency,
environment of business intelligence,” Kybernetes, vol. 41, 2018, pp. 77–91.
no. 10, pp. 1625–1637, 2012. [19] S. Buell, “MIT researcher: artificial intelligence has a race
[3] S. Strohmeier and F. Piazza, “Artificial intelligence techni- problem and we need to fix it,” The Boston Magazine, Jan. 24
ques in human resource management – a conceptual 2019, https://www.bostonmagazine.com/news/2018/02/23/
exploration,” in Intelligent Techniques in Engineering artificial-intelligence-race-dark-skin-bias/ [accessed
Management, C. Kahraman and S. Çevik Onar, Eds, March 14, 2020].
Intelligent Systems Reference Library, vol. 87, Cham: [20] V. Muthukumar, et al., “Understanding unequal gender
Springer, 2015. classification accuracy from face images,” arXiv preprint
[4] J. Min, Ten ways HR tech leaders can make the most of arXiv:1812.00099, 2018.
artificial intelligence, 2017, https://www.personneltoday. [21] S. P. Zafeiriou, C. Zhang, and Z. Zhang, “A survey on face
com/hr/ten-ways-hr-tech-leaders-can-make-artificial- detection in the wild: past, present and future,” Comput. Vis.
intelligence/ [accessed March 14, 2020]. Image Und., vol. 138, pp. 1–24, 2015.
 AI and recruiting software: ethical and legal implications  215

[22] Y. Bai, Y. Zhang, M. Ding, and B. Ghanem, “Finding tiny faces judgments and decisions,” Organ. Behav. Hum. Decis.
in the wild with generative adversarial network,” in 2018 IEEE/ Process., vol. 101, no. 2, pp. 152–167, 2006.
CVF Conference on Computer Vision and Pattern Recognition, [39] V. Ging, “11 common interview questions that are actually
Jun. 2018. illegal, business insider,” https://www.businessinsider.com/
[23] C. Garvie, The perpetual line-up: unregulated police face 11-illegal-interview-questions-2013-7?IR=T [accessed
recognition in America, Georgetown Law, Center on Privacy & March 14, 2020].
Technology, Washington, DC, 2016. [40] A. Prince and D. Schwarcz, “Proxy discrimination in the age of
[24] S. Cosar, G. Donatiello, V. Bogorny, C. Garate, L. O. Alvares, artificial intelligence and big data,” Iowa Law Review, vol. 105,
and F. Bremond, “Toward abnormal trajectory and event no. 3, pp. 1257–1318, 2020.
detection in video surveillance,” IEEE Trans. Circuits Syst. [41] A. Datta, M. Fredrikson, G. Ko, P. Mardziel, and S. Sen, “Proxy
Video Technol., vol. 27, no. 3, pp. 683–695, 2017. discrimination in data-driven systems,” arXiv preprint
[25] S. Sulpizio, et al., “The sound of voice: voice-based categor- arXiv:1707.08120, 2017.
ization of speakers’ sexual orientation within and across [42] S. Barocas and A. Selbst, “Big data’s disparate impact,” Calif.
languages,” PLOS One, vol. 10, no. 7, pp. 1–38, Jul. 2015. Law Rev., vol. 104, no. 3, pp. 671–732, 2016.
[26] J. Kuczmarski, “Reducing gender bias in google translate,” [43] D. Pedreschi, S. Ruggieri, and F. Turini, “Integrating induction
Google Blog, Dec. 6 2018, https://www.blog.google/ and deduction for finding evidence of discrimination,” in
products/translate/reducing-gender-bias-google-translate/ Proceedings of the 12th International Conference on Artificial
[accessed March 14, 2020]. Intelligence and Law (ICAIL ‘09), 2009.
[27] S. Leavy, “Gender bias in artificial intelligence: the need for [44] S. Hajian and J. Domingo-Ferrer, “A methodology for direct
diversity and gender theory in machine learning,” in and indirect discrimination prevention in data mining,” IEEE
2018 IEEE/ACM 1st International Workshop on Gender Trans. Knowl. Data Eng., vol. 25, no. 7, pp. 1445–1459,
Equality in Software Engineering (GE), Gothenburg, 2018, 2013.
pp. 14–16. [45] D. Hawkins, “Researchers use facial recognition tools to
[28] L. Yang, “A gender perspective of translation: taking three predict sexual orientation. LGBT groups aren’t happy,” The
chinese versions of the purple color as an example,” J. Lang. Washington Post, September 12, 2017, https://www.
Teach. Res., vol. 5, no. 2, pp. 371–375, 2014. washingtonpost.com/news/morning-mix/wp/2017/09/12/
[29] P. Viola and M. Jones, “Robust real-time face detection,” Int. J. researchers-use-facial-recognition-tools-to-predict-sexuality-
Comput. Vision, vol. 57, no. 2, pp. 137–154, 2004. lgbt-groups-arent-happy/?utm_term=23020459ddcd
[30] S. Masood, S. Gupta, A. Wajid, S. Gupta, and M. Ahmad, [accessed March 14, 2020].
“Prediction of human ethnicity from facial images using [46] H. Murphy, “Why Stanford researchers tried to create a
neural networks,” in Data Engineering and Intelligent “gaydar” machine,” The New York Times, Oct. 9, 2017,
Computing, S. Satapathy, V. Bhateja, K. Raju, and B. https://www.nytimes.com/2017/10/09/science/stanford-
Janakiramaiah, Eds., Advances in Intelligent Systems and sexual-orientation-study.html [accessed March 14, 2020].
Computing, vol. 542, Springer, Singapore, 2017, pp. 217–226. [47] S. Ossowski and A. Omicini, “Coordination knowledge engineering,”
[31] Z. Jin-Yu, C. Yan, and H. Xian-Xiang, “Edge detection of Knowl. Eng. Rev., vol. 17, no. 4, pp. 309–316, 2002.
images based on improved Sobel operator and genetic [48] S. Mahmoud, et al., “Multi-agent system for recruiting
algorithms,” in 2009 International Conference on Image patients for clinical trials,” in Proceedings of the 2014
Analysis and Signal Processing, 2009, pp. 31–35. international conference on Autonomous agents and multi-
[32] J. Shi and C. Tomasi, “Good features to track,” in Proceedings agent systems, 2014, pp. 981–988.
of IEEE Conference on Computer Vision and Pattern [49] E. Mas, A. Suppasri, F. Imamura, and S. Koshimura, “Agent-based
Recognition, Seattle, 1994. simulation of the 2011 great east Japan earthquake/tsunami
[33] W. H. Abdulsalam, R. S. Alhamdani, and M. Najm Abdullah, evacuation: an integrated model of tsunami inundation and
“Facial emotion recognition from videos using deep convolu- evacuation,” J. Nat. Disaster Sci., vol. 34, no. 1, pp. 41–57,
tional neural networks,” Int. J. Mach. Learn. Comput., vol. 9, 2012.
no. 1, pp. 14–19, 2019. [50] H. Billhardt, A. Fernández, M. Lujak, and S. Ossowski,
[34] P. Ekman, “Darwin, deception, and facial expression,” Ann. N. “Agreement technologies for coordination in smart cities,”
Y. Acad. Sci., vol. 1000, no. 1, pp. 205–221, 2003. Appl. Sci., vol. 8, no. 5, Art. 816, 2018, DOI: https://doi.org/
[35] D. Mehta, M. F. Siddiqui, and A. Y. Javaid, “Recognition of 10.3390/app8050816.
emotion intensities using machine learning algorithms: a [51] J. Debenham, “A multi-agent architecture for business
comparative study,” Sensors, vol. 19, no. 8, Art. 1897, 2019, process management adapts to unreliable performance,”
https://doi.org/10.3390/s19081897. Adaptive Computing in Design and Manufacture V, Springer,
[36] A. Dehghan, E. G. Ortiz, G. Shu, and S. Z. Masood, “Dager: London, 2002, pp. 369–380.
deep age, gender and emotion recognition using convolu- [52] A. Ciortea, S. Mayer, and F. Michahelles, “Repurposing
tional neural network,” arXiv preprint arXiv:1702.04280, 2017. manufacturing lines on the fly with multi-agent systems for
[37] C. F. Benitez-Quiroz, R. Srinivasan, Q. Feng, Y. Wang, and the web of things,” in Proceedings of the 17th International
A. M. Martinez, “EmotioNet challenge: recognition of facial Conference on Autonomous Agents and Multiagent Systems,
expressions of emotion in the wild,” arXiv preprint 2018, pp. 813–822.
arXiv:1703.01210, 2017. [53] V. R. Walker, “A default-logic framework for legal reasoning
[38] S. L. S. Purkiss, P. L. Perrewé, T. L. Gillespie, B. T. Mayes, and in multiagent systems,” AAAI Fall Symposium, 2006,
G. R. Ferris, “Implicit sources of bias in employment interview pp. 88–95.
216  Carmen Fernández-Martínez and Alberto Fernández

[54] T. Calders and S. Verwer, “Three naive Bayes approaches for [57] C. Fernández and A. Fernández, “AI in recruiting. Multi-agent
discrimination-free classification,” Data Min. Knowl. Discov., systems architecture for ethical and legal auditing,” in
vol. 21, no. 2, pp. 277–292, 2010. Proceedings of the Twenty-Eight International Joint Conference
[55] M. Kearns, S. Neel, A. Roth, and Z. S. Wu, “Preventing Artificial Intelligence (IJCAI), 2019.
fairness gerrymandering: auditing and learning for [58] C. Fernández and A. Fernández, “Ontologies and AI in
subgroup fairness,” arXiv preprint arXiv:1711.05144, 2017. recruiting. A rule-based approach to address ethical and legal
[56] C. Fernández and A. Fernández, “Ethical and legal implications of ai auditing,” in Proceedings of the International Semantic Web
recruiting software,” ERCIM News, vol. 116, pp. 22–23, 2019. Conference (ISWC), 2019.