Original research

Open AI meets open notes: surveillance capitalism,

J Med Ethics: first published as 10.1136/jme-2023-109574 on 23 November 2023. Downloaded from http://jme.bmj.com/ on August 11, 2024 by guest. Protected by copyright.
patient privacy and online record access
Charlotte Blease ‍ ‍ 1,2
1
Women’s and Children’s ABSTRACT they do not want visible on their online portals
Health, Uppsala Universitet, Patient online record access (ORA) is spreading and decisions about when to ethically block access
Uppsala, Sweden
2
Digital Psychiatry, Beth Israel worldwide, and in some countries, including Sweden, among adults or children in domestic abuse situa-
Deaconess Medical Center, and the USA, access is advanced with patients obtaining tions.9 13–15 A different privacy consideration is how
Boston, Massachusetts, USA rapid access to their full records. In the UK context, ORA intersects with ‘surveillance capitalism’.16 This
from 31 October 2023 as part of the new NHS England phrase, coined by the social scientist Zuboff refers
Correspondence to general practitioner (GP) contract it will be mandatory to the monetisation of human experiences and
Dr Charlotte Blease, Women’s for GPs to offer ORA to patients aged 16 and older. behaviours on the internet.17 For example, personal
and Children’s Health, Uppsala
Universitet, Uppsala, 752 37, Patients report many benefits from reading their clinical data from internet searches may be sold to medical
Sweden; c​ harlotte.​blease@​uu.​se records including feeling more empowered, better health insurance companies, direct to consumer
understanding and remembering their treatment plan, drug advertisers or other for-­profit enterprises.
Received 9 September 2023 and greater awareness about medications including An overlooked, unintended consequence of ORA
Accepted 30 October 2023 is that detailed medical internet searches, social
Published Online First
possible adverse effects. However, a variety of indirect
23 November 2023 evidence suggests these benefits are unlikely to accrue media use and the uptake of a new generation of
without supplementation from internet-­based resources. chatbots powered by large language models (LLMs),
Using such routes to augment interpretation of the exemplified by OpenAI’s GPT-­4, Google’s Bard and
data and notes housed in electronic health records, Microsoft’s Bing AI, might expose patient data to
however, comes with trade-­offs in terms of exposing new privacy risks. While these kinds of privacy risks
sensitive patient information to internet corporations. do not constitute a robust reason to deny ORA (and
Furthermore, increased work burdens on clinicians, very strong additional argumentation would be
including the unique demands of ORA, combined with required to justify such a stance, taking us beyond
the easy availability and capability of a new generation the scope of the present paper), nonetheless, the
of large language model (LLM)-­powered chatbots, create practical realities of everyday internet use, patient
a perfect collision course for exposing sensitive patient accessible records and privacy regulations deserve
information to private tech companies. This paper surveys scrutiny. This paper attempts to initiate that task.
how ORA intersects with internet associated privacy risks The paper surveys the risks associated with patient
and offers a variety of multilevel suggestions for how internet use, before examining the specific privacy
these risks might be better mitigated. risks associated with clinician documentation prac-
tices in the era of ORA. It concludes with a variety
of multilevel suggestions for how privacy risks
Online record access (ORA) is expanding world- might be better mitigated.
wide. Already patients in an estimated 30 countries
can access at least some of their online records via
secure portals and apps. Access can include list of POTENTIAL PRIVACY RISKS OF PATIENTS USING
medications, vaccinations, laboratory results and OPEN NOTES
even the very narrative reports written by clinicians During clinic visits, patients misremember around
(the latter commonly referred to as ‘open notes’). half of what is communicated,18 and often fail to
In the Nordic countries and the USA, this innova- understand the health information conveyed by
tion is advanced.1 For example, implementation in doctors.19 It is hypothesised that access to electronic
Sweden began in 2010 and since 2018 adults have health records may present a workaround solving
had access to their online records.2 3 In Norway, a variety of face-­ to-­face communication break-
implementation started in 2015 expanding to downs that can arise in face-­to-­face visits.20 21 The
patients in three out of four regions by 2019.4 Since suggestion is patients are afforded opportunities
2021 in the USA, the federally enacted 21st Century to pore over their records, and to remember, and
Cures Act mandated that providers offer all patients better grasp what was conveyed by clinicians. While
access to download their electronic health records this seems like a sound supposition, it also seems
without charge.5 Patients report many benefits from reasonable to suggest access to the records alone
reading their clinical records including feeling more may not facilitate improved understanding about
empowered, better understanding and remembering what is documented—for at least two reasons.
their treatment plan, and greater awareness about First, while patients have experiential knowledge
© Author(s) (or their medications including possible adverse effects.1 2 6–8 about their condition and health, most do not have
employer(s)) 2024. Re-­use
permitted under CC BY. While there is a growing interest in examining in-­depth medical expertise or a command of medical
Published by BMJ. the risks of ORA with respect to patient privacy, vernacular, even though some do become experts on
this research agendum that largely focuses on safe- their condition(s).22 23 Clinicians as domain experts,
To cite: Blease C.
J Med Ethics 2024;50:84–89.
guarding and proxy access9–12—for example, how have a tendency to overestimate patients’ knowl-
patients might selectively hide information that edge of specialist or technical language, failing to
84   Blease C. J Med Ethics 2024;50:84–89. doi:10.1136/jme-2023-109574
 Original research
calibrate it effectively to lay levels of understanding24—a cluster health records, with online access such searches are likely to be

J Med Ethics: first published as 10.1136/jme-2023-109574 on 23 November 2023. Downloaded from http://jme.bmj.com/ on August 11, 2024 by guest. Protected by copyright.
of problems collectively referred to as ‘the curse of expertise’.25 more detailed: patients may use the internet to decipher detailed
This epistemic imbalance suggests it is unlikely that patients are test results, to translate the technical language embedded in
fully equipped to understand all the detail housed in their elec- documentation and/or to avoid confusions.
tronic records.
Second, and relatedly, clinical records traditionally served as
Risks from search engines and social media
an aide memoire for clinicians, or as a tool for communicating
This usage invites questions about the extent of privacy expo-
detailed medical information to other providers, and not as a way
sures arising when patients use search engines or social media,
to convey accessible health information to patients.26 Although
and privacy risks differ between countries and regions. In the
significant proportions of surveyed physicians report changing
USA, the 1996 Health Insurance Portability and Accountability
how they write clinical information in the era of ORA including
Act (HIPAA) created national standards in the USA to protect
changing language perceived as critical of the patient, and modi-
patients’ health information from being shared by ‘covered enti-
fying how they document sensitive information,27–29 there is
ties’—that is providers—to other third parties. In the epoch of
evidence that the medical terminology embedded in notes is
the internet in the USA, HIPAA lacks teeth to protect patient
largely preserved. For example, in a US survey led by DesRoches
privacy from surveillance capitalism. While generic symptom
of clinicians’ experiences with writing open notes, of 1628 clini-
searches risk exposing personal consumer information to tech
cians (response rate 27%), 76% (n=966) reported that open
giants, using search engines or social media as vehicles for more
notes did not affect the value of their notes for other clinicians.29
precise understanding about the information housed in their
Furthermore, beyond the narrative reports written by clinicians,
records—perhaps ‘cutting and pasting’ detailed information into
lab and test results, and medications, can be logged without any
browsers or on social media—could lead to more meaningful
corresponding, clarificatory notes offered to patients.
exposures. Furthermore, in the USA, the 21st century Cures Act
Still, patients do report multiple benefits from ORA including
mandates that patient’s ORA is also available in a downloadable
(as noted above) feeling more empowered and better under-
form for use with other apps.39
standing their treatment plans and medications.1 2 6–8 This
In the European Union, under the General Data Protection
intimates that increased understanding might not derive from
reading the records in isolation, but arises in conjunction with Regulation (GDPR), citizens have greater control over their
supplementary resources. Preliminary findings strongly indicate personal information.40 Without informed consent, or unless
this could indeed be the case. For example, in the largest patient exceptional circumstances are met, such as public health justifi-
survey of open notes conducted in the USA at three health cations, companies are prohibited from processing and trading
centres, of 29 656 adult patients (response rate: 22%), 23 576 in, ‘data concerning health’ defined as ‘personal data related to
(79%) reported reading at least 1 note, and among them, 19 411 the physical or mental health of a natural person, including the
(82%) reported they were taking or had been prescribed a medi- provision of healthcare services, which reveal information about
cation in the past 12 months; of them, 32% said access made his or her health status.’40
them ‘seek more information’ about their medications.8 Further- Against the foregoing considerations, if patients are using
more, among patients with serious mental health diagnoses, the search engines, social media, or other smartphone applications
proportion was higher: 39% reported seeking more information (apps) to strengthen understanding about their care, do any such
after reading their clinicians’ notes.30 privacy exposures constitute an ethical problem or one that
constitutes a privacy violation? On the face of it, it might seem
Prevalence of internet use for health information searches that the individual responsibility is with the actor who chooses
This constellation of factors invites nontrivial questions about to use the internet to supplement their understanding of their
where and how patients might supplement deficits in their under- medical information—that any related exposures do not consti-
standing. Notably, it is now well established that consumers use tute privacy violations and instead constitute freely made deci-
the internet to seek health information31 with an estimated 7% sions, exercised without external pressures.
of Google searches health related.32 Recent findings show, more This, however, oversimplifies the sociotechnical context of
four in ten Americans use Google instead of seeing a doctor,33 and internet use, exaggerating the freedom that consumers have
around half of Europeans search for health information online.34 when they turn to online health resources. One reason for scep-
When it comes to social media, the statistics are even more ticism is that consent processes do not meet threshold elements
arresting with an estimated 4.8 billion social media users glob- or preconditions for consent in terms of competence (of
ally, comprising 6 in 10 of the world’s population.35 Exchanging consumers’ capacity to understand and decide), and of voluntari-
and accessing health information is a common reason for using ness (in making decisions), and of requisite information elements
social media36; for example, in in one recent US survey, three in (the disclosure of salient information relevant to consent).41
four respondents said they relied on Facebook and Twitter for Consider the first of these, competence to decide. Even in the
COVID-­19 health information.37 In a study in Germany, Braun EU context where data harvesting is more stringently regulated,
et al found that the internet was the most important source of when offered the option to give consent to data collection via
information for patients with cancer (75%, n=308), with half apps, consumers are also obliged to read through and under-
the respondents (49.2%, n=196) reporting that a cancer diag- stand labyrinthine terms and conditions. The reality is these are
nosis was the stimulus for starting to engage with social media.38 often unfeasibly onerous. In 2008, it was estimated the average
To reiterate, although the connection between ORA and internet user would need 76 days per year to wade through every
patient internet use has not been directly explored by health internet privacy policy before deliberating over consenting, a
services researchers, since patients frequently use online figure undoubtedly higher today.42 Zuboff argues that the very
resources to strengthen knowledge about their health, it is not minutiae and scale of the small print is one reason consumers
a leap to suppose many may be inputting sensitive information are inclined to put their privacy at risk.17 Simply put, the stan-
into websites to augment their understanding. Although they dard of competence required to meaningfully consent is often
may already be doing so regardless of access to their electronic unreasonable.
Blease C. J Med Ethics 2024;50:84–89. doi:10.1136/jme-2023-109574 85
 Original research
When it comes to voluntariness—of consumers making deci- these chatbots are not without significant limitations: the quality

J Med Ethics: first published as 10.1136/jme-2023-109574 on 23 November 2023. Downloaded from http://jme.bmj.com/ on August 11, 2024 by guest. Protected by copyright.
sions without external pressures—there are also reasons for of the data on which they are trained means they may generate
doubt. The internet is now a domestic necessity akin to running discriminatory advice (so-­called ‘algorithmic biases’) that could
water, electricity, or plumbing. Most internet users appear cogni- worsen racial, ageist or gender discrimination in care; they are
zant that their personal data might be monetised or sold to third prone to making things up (‘hallucinations’); and could offer
parties. In 2018, a survey by Rock Health reported that only 1 harmful medical advice.57
in 10 Americans were willing to share their health data with tech The extent to which patients use generative-­AI chatbots to
giants such as Facebook and Amazon.43 Recent Pew Research augment understanding about their healthcare is only beginning
studies show most Americans distrust private companies with an to be explored. In June 2023, in a survey conducted by Medical
estimated 8 in 10 believing their personal data is less secure now, Economics, 8 in 10 Americans surveyed believed these tools
and that it is not possible to go through daily life without being had the potential to improve the quality of healthcare, reduce
tracked.44 Despite these reservations, as we have seen, health-­ costs and increase the accessibility of care.58 Precisely because
related internet searches and social media use are commonplace. these chatbots are user-­friendly, however, consumers may be
As Zuboff argues, the explanation for this apparent tension seduced into anthropomorphising the interactions. For example,
between risking data exposure and using online resources, is that for low risk health complaints, one experimental study demon-
the choice to opt out is essentially an ersatz one: the internet is strated ChatGPT responses to patients’ queries were only weakly
so essential for social participation that it, ‘produces a psychic distinguishable from clinician responses with patients trusting
numbing that inures us to the realities of being tracked, parsed, chatbot responses.59 These attributes could render LLMs such
mined and codified’.17 as Chat-­GPT particularly potent extractors of sensitive and
A related reason to doubt that threshold conditions of volun- detailed health information.55 60
tariness in consent processes are met comes from the increasing
pressures on health systems and their staff. Patients prefer
to receive health information directly from their doctor; for POTENTIAL PRIVACY RISKS OF CLINICIANS WRITING OPEN
example, in the study by Braun et al, 85% said they wanted to NOTES
get information from their doctors (n=342), yet the internet Potential privacy exposures do not only arise from the patient
was reportedly the most important source of information for side: using LLM-­powered chatbots clinicians may inadvertently
patients with cancer (75%, n=308).38 Physician time is now one expose sensitive patient data too. For example, in July 2023,
of the scarcest resources in medicine. The COVID-­19 pandemic five hospitals in Australia’s South Metropolitan Health Service,
oversaw a flight of physicians from primary care45; and globally, in Perth, were instructed to stop using ChatGPT after it was
the WHO predicts a worldwide shortage of 10 million health- discovered some staff had been using the chatbot to write clinical
care workers by 2030.46 In England, for example, as of April notes.61 In the USA, in a June 2023 Medical Economics study,
2023, nearly 5 million patients waited more than 2 weeks to more than 1 in 10 surveyed healthcare professionals reported
obtain a general practitioner (GP) appointment.47 Even prior adopting generative AI chatbots such as ChatGPT, and nearly
to the pandemic, health systems have increasingly been strained 50% expressed an intent to use these technologies in the future
with ageing populations, and with more people suffering chronic for tasks such as data entry, medical scheduling or research.58
illnesses for longer. Perhaps unsurprisingly, surveys in the UK and Despite the (aforementioned) limitations associated with these
the USA show that physician burnout is at an all-­time high.48 49 tools, their greatest current promise lies in assisting clinicians
Combined, studies show people are often acutely aware of the with administrative tasks. Here, the potential should not be
pressures on doctors, and of the rationing involved in health underemphasised.56 LLM-­powered chatbots have the capacity
systems,50 with many reporting using internet resources to avoid to summarise material in a requested style or tone suitable for
asking too many questions in clinic visits, out of fear of taking a wide range of lay readers even removing jargon or phrasing
up too much time in visits, or to avoid ‘doctor-­bothering’.51 52 that may interfere with medical understanding, by couching
A third reason to doubt consent processes are being met comes responses at different literacy levels, or adopting an empathic
from analyses of informational elements in health apps.53 For tone.62
example, a recent analysis of privacy policies of 36 top-­ranked Moreover, given the pressures on them, clinician motivations
apps for smoking cessation and depression, available for down- for clinicians adopting LLMs such as Chat-­GPT are clear, and
load in popular app stores, found that 29 transmitted data to as we’ve seen, preliminary evidence suggests that at least some
services provided by Facebook or Google, yet only 12 accurately doctors are turning to generative AI to assist with workplace
disclosed this in a privacy policy.54 demands. Furthermore, ORA may invite new work burdens
of its own since the electronic health record now serves as a
Risks from LLM-powered chatbots communication tool, not just for other providers but also for
Aside from the range of elevated exposures via search engines patients. A key trend in physician surveys is widespread fear
and social media, the discussed risks to patient privacy are that these measures, and subsequent patient contact as a result
likely to be even graver with LLMs-­powered chatbots, such as of ORA, will increase in workloads.27 63–65 Evidence suggests
ChatGPT. This new generation of chatbots offers an unprece- that clinicians recognise adjustments to documentation may be
dented level of conversational fluency in computer-­human inter- needed so that information is recorded in understandable and
actions.55 Unlike traditional search engines which offer lists of sensitive ways.9 27 63 Following implementation of open notes, in
webpages, LLM-­based chatbots facilitate exchanges that mimic the survey by DesRoches, 36% (n=463) of clinicians reported
dialogue and ‘remember’ previous prompts, helping to create the spending longer writing notes, with 58% (n=422) changing
perception of smoother exchanges. LLMs use massive amounts language that might be perceived as critical of patients and
of past data to predict the next word in a sequence. This prob- 49% (n=372) changing how they documented sensitive clinical,
abilistic process combined with other technical advances means mental health or social information.29 While there is a paucity
these models are well suited to recognising, summarising and of objective evidence to demonstrate patient online access does
generating content.56 However, it is important to emphasise that increase work burdens, some findings do suggest an increase in
86 Blease C. J Med Ethics 2024;50:84–89. doi:10.1136/jme-2023-109574
 Original research
clerical workload with patient email traffic increasing, in partic- strongly advised that such tools might be valuable in assisting

J Med Ethics: first published as 10.1136/jme-2023-109574 on 23 November 2023. Downloaded from http://jme.bmj.com/ on August 11, 2024 by guest. Protected by copyright.
ular, following the release of test results.66–68 At least some of this with writing documentation but not replace clinicians in under-
increased contact, it seems reasonable to surmise, may be driven taking this task.
by patients who are confused or desire clarifications about their Third, at a civic level, questions about data ownership and
medical information. consent for using health data need to be resolved.79 It is worth
Again, however, when clinicians adopt these chatbots to assist emphasising that, even when it comes to the USA’s HIPAA-­
with documentation they risk exposing sensitive patient infor- compliant LLM-­powered chatbots, establishing whether patients
mation. For these reasons, in June 2023, the American Psychi- consent to use their electronic record as training data, is worthy
atric Association issued guidance strongly opposing physicians of further scrutiny. Indeed, beyond health data, legal challenges
entering patient data into generative AI systems.69 And in July relating to copyright infringements, by writers and artists,80
2023, responding to reports of physicians using ChatGPT to have already been raised on the grounds that LLM models
write medical notes, the Australian Medical Association, called scrape public data from the internet to train their models.81
for stronger AI regulations.70 Authorities in the EU are currently reviewing whether, without
obtaining informed consent, OpenAI’s ChatGPT complies with
GDPR regulations and meets the requirement of public health
SUGGESTIONS TO REDUCE PRIVACY RISKS OF OPEN NOTES justifications.82
Patients, and clinicians, may be concerned about the potential
privacy exposures associated with sharing clinical notes and
internet use, and to mitigate risks, several suggestions could be CONCLUSIONS
considered. ORA affords patients many benefits including feeling more in
First, there are serious deficits in health and social care profes- control of, and knowledgeable about, their healthcare. However,
sionals’ digital literacy, including with respect to the ethical currently, these benefits are unlikely to accrue without supple-
implications of digital tools.71–73 Similarly, patients may also mentation from internet-­based resources. ORA access combined
need training to become more knowledgeable about how to with the intrinsicality of internet use in daily life, surveillance
use digital innovations in healthcare including the positive and capitalism, the pressures on health systems, and the challenges
negative aspects of these tools.74 Brief educational interventions of readily availing of health services, create the perfect storm for
could help clinicians and patients become more aware about the privacy exposures. It is imperative for patients and clinicians to
risks associated with surveillance capitalism, including of using become more aware of these privacy risks. Health systems should
internet search engines, social media and generative AI chatbots, also, as a matter of exigency, outline policies that uphold privacy
to supplement understanding of information housed in elec- in the use of LLM chatbots to assist clinicians with documenta-
tronic medical records, or—in the case of clinicians—to assist tion. While debates about regulating the power of tech giants
with writing documentation patients might read. Increasingly, continue to rage at political and legislative levels, the elevated
health organisations are becoming aware of the need to offer risks of exposing private health information with online record
guidance to patients about the benefits and risks of accessing access should also be considered.
ORA, for example, via leaflets and practice websites9 11 12; and
guidance about privacy risks could also be discussed. Similarly, Contributors CB is the sole author and guarantor.
in undergraduate medical education, GP training programmes,
Funding This study was funded by Forskningsrådet om Hälsa, Arbetsliv och Välfärd
and continuing professional development courses, brief training (Beyond Implementation—’no award/grant number’).
about the internet and privacy could improve clinicians’ under-
Competing interests None declared.
standing about why innovations such as ChatGPT pose risks to
Patient consent for publication Not applicable.
patient data.75
Second, health systems could strive to improve the under- Provenance and peer review Not commissioned; externally peer reviewed.
standability of health information curated in patient accessible Data availability statement No data are available.
records. This could be achieved via improved patient codesign Open access This is an open access article distributed in accordance with the
of portals, including embedding and curating access to patient-­ Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits
friendly resources that are linked to, and housed securely within, others to copy, redistribute, remix, transform and build upon this work for any
purpose, provided the original work is properly cited, a link to the licence is given,
the electronic records.20 As noted, patients also prefer to receive
and indication of whether changes were made. See: https://creativecommons.org/​
information from their physicians. Therefore, to reduce the risks licenses/by/4.0/.
of patients actively supplementing their understanding with
internet or chatbot searches beyond the portal, it would be valu- ORCID iD
Charlotte Blease http://orcid.org/0000-0002-0205-1165
able to augment the quality of the narrative reports written by
clinicians. To that end, given that LLM-­powered chatbots may
offer significant promise in assisting with writing medical notes,
health systems could seek to embed such tools within electronic REFERENCES
record systems only if these tools comply with patient privacy 1 Hägglund M, McMillan B, Whittaker R, et al. Patient empowerment through online
access to health records. BMJ 2022;378:e071531.
laws. Already there are promising efforts in this direction. For 2 Blease C, Salmi L, Rexhepi H, et al. Patients, clinicians and open notes: information
example, in the USA, an Azure HIPAA compliant GPT-­4 service blocking as a case of epistemic injustice. J Med Ethics 2022;48:785–93.
now exists,76 and Epic—which has the largest share of the 3 Hägglund M, Scandurra I. Patients’ online access to electronic health records: Current
status and experiences from the implementation in Sweden. MedInfo 2017:723–7.
hospital medical record market77—is piloting the integration 4 Zanaboni P, Kummervold PE, Sørensen T, et al. Patient use and experience with online
of HIPAA-­compliant GPT services.78 Given the limitations and access to electronic health records in Norway: results from an online survey. J Med
challenges associated with LLM-­powered chatbots, including the Internet Res 2020;22:e16144.
tendency to ‘hallucinate’, the potential for bias, and inconsis- 5 Salmi L, Blease C, Hägglund M, et al. US policy requires immediate release of records
to patients. BMJ 2021;372:n426.
tencies in summative responses, clinicians may need guidance 6 Moll J, Rexhepi H, Cajander Å, et al. Patients’ experiences of accessing their electronic
about the careful adoption and use of these tools and should be health records: national patient survey in Sweden. J Med Internet Res 2018;20:e278.

Blease C. J Med Ethics 2024;50:84–89. doi:10.1136/jme-2023-109574 87

 Original research
7 Walker J, Leveille S, Bell S, et al. Opennotes after 7 years: patient experiences 39 Health and human services Department, USA. 21st century cures act: Interoperability,

J Med Ethics: first published as 10.1136/jme-2023-109574 on 23 November 2023. Downloaded from http://jme.bmj.com/ on August 11, 2024 by guest. Protected by copyright.
with ongoing access to their clinicians’ outpatient visit notes. J Med Internet Res information blocking, and the ONC health IT certification program. 2020. Available:
2019;21:e13876. https://www.govinfo.gov/content/pkg/FR-2020-05-01/pdf/2020-07419.pdf [Accessed
8 DesRoches CM, Bell SK, Dong Z, et al. Patients managing medications and 15 Jul 2020].
reading their visit notes: a survey of opennotes participants. Ann Intern Med 40 The European Parliament and The Council of The European Union. On the
2019;171:69–71. protection of natural persons with regard to the processing of personal data and
9 Blease C, McMillan B, Salmi L, et al. Adapting to transparent medical records: on the free movement of such data, and repealing directive 95/46/EC [General
International experience with "open notes". BMJ 2022;379:e069861. Data Protection Regulation]. 2016. Available: https://gdpr-info.eu/ [Accessed 23 Jan
10 OpenNotes. How to use open notes. 2023. Available: https://howtouse.opennotes.org/ 2023].
[Accessed 03 Sep 2023]. 41 Beauchamp TL, Childress JF. Principles of biomedical ethics. USA: Oxford University
11 Hannan A. Learn about how to sign up for online services including access to your Press, 2001.
electronic records [Haughton Thornley Medical Centres]. 2017. Available: http://www.​ 42 McDonald AM, Cranor LF. The cost of reading privacy policies. Isjlp 2008;4:543.
htmc.co.uk/pages/pv.asp?p=htmc700 [Accessed 29 Jan 2022]. 43 Farr C. Tech companies see health data as a huge opportunity, but people don’t trust
12 Blease C, Torous J. Opening mental health notes: 7 tips to prepare clinicians them [CNBC]. 2019. Available: https://www.cnbc.com/2019/02/13/consumers-dont-​
[Psychology Today]. 2020. Available: https://www.psychologytoday.com/us/blog/​ trust-tech-companies-with-health-data-rock-health.html [Accessed 14 Jan 2023].
digital-mental-health/202010/opening-mental-health-notes-7-tips-prepare-clinicians 44 Auxier B, Rainie L, Anderson M, et al. Americans and privacy: concerned, confused and
[Accessed 11 Dec 2020]. feeling lack of control over their personal information [Pew Research Center]. 2019.
13 Bourgeois FC, DesRoches CM, Bell SK. Ethical challenges raised by opennotes for Available: https://www.pewresearch.org/internet/2019/11/15/americans-and-privacy-​
pediatric and adolescent patients. Pediatrics 2018;141:e20172745. concerned-confused-and-feeling-lack-of-control-over-their-personal-information/
14 Hagström J, Scandurra I, Moll J, et al. Minor and parental access to electronic health [Accessed 14 Jan 2023].
records: differences across four countries. In: Challenges of Trustable AI and Added-­ 45 Abbasi J. Pushed to their limits, 1 in 5 physicians intends to leave practice. JAMA
Value on Health. IOS Press, 2022: 495–9. 2022;327:1435–7.
15 Hagström J, Blease C, Haage B, et al. Views, use, and experiences of web-­based 46 World Health Organization. Health workforce. 2023. Available: https://www.who.int/​
access to pediatric electronic health records for children, adolescents, and parents: health-topics/health-workforce#tab=tab_1 [Accessed 22 Aug 2023].
scoping review. J Med Internet Res 2022;24:e40328. 47 BBC News. Millions wait more than a fortnight to see a GP in England [BBC News].
16 Blease C. Response to commentaries on sharing online clinical notes with patients: 2023. Available: https://www.bbc.co.uk/news/health-65275367 [Accessed 07 Sep
implications for Nocebo effects and health equity. J Med Ethics 2023;49:32–3. 2023].
17 Zuboff S. The Age of Surveillance Capitalism: The Fight for a Human Future at the 48 Tonkin T. Burnout hits record high [BMA]. 2022. Available: https://www.bma.org.uk/​
New Frontier of Power: Barack Obama’s Books of 2019. Profile Books, 2019. news-and-opinion/burnout-hits-record-high [Accessed 07 Sep 2023].
18 Kessels RPC. Patients’ memory for medical information. J R Soc Med 49 Berg S. Pandemic pushes U.S. doctor burnout to all-­time high of 63% [AMA]. 2022.
2003;96:219–22. Available: https://www.ama-assn.org/practice-management/physician-health/​
19 McCarthy DM, Waite KR, Curtis LM, et al. What did the doctor say? Health literacy pandemic-pushes-us-doctor-burnout-all-time-high-63 [Accessed 07 Sep 2023].
and recall of medical instructions. Med Care 2012;50:277–82. 50 Llanwarne N, Newbould J, Burt J, et al. Wasting the doctor’s time? A video-­elicitation
20 Blease C, Fernandez L, Bell SK, et al. Empowering patients and reducing inequities: is interview study with patients in primary care. Soc Sci Med 2017;176:113–22.
there potential in sharing clinical notes. BMJ Qual Saf 2020;29:1–2. 51 Tan S-­L, Goonawardene N. Internet health information seeking and the patient-­
21 Blease C. We need to talk about “closed notes." [BMJ]. 2021. Available: https://blogs.​ physician relationship: a systematic review. J Med Internet Res 2017;19:e9.
bmj.com/bmj/2021/10/15/charlotte-blease-we-need-to-talk-about-closed-notes/ 52 Chiu Y-­C. Probing, impelling, but not offending doctors: the role of the Internet
22 Carel H, Kidd IJ. Epistemic injustice in healthcare: a philosophical analysis. Med Health as an information source for patients’ interactions with doctors. Qual Health Res
Care Philos 2014;17:529–40. 2011;21:1658–66.
23 Kidd IJ, Carel H. Epistemic injustice and illness. J Appl Philos 2017;34:172–90. 53 Powell AC, Singh P, Torous J. The complexity of mental health app privacy policies: a
24 Castro CM, Wilson C, Wang F, et al. Babel babble: physicians’ use of unclarified potential barrier to privacy. JMIR Mhealth Uhealth 2018;6:e158.
medical jargon with patients. Am J Health Behav 2007;31 Suppl 1:S85–95. 54 Huckvale K, Torous J, Larsen ME. Assessment of the data sharing and privacy practices
25 Fisher M, Keil FC. The curse of expertise: when more knowledge leads to miscalibrated of Smartphone Apps for depression and smoking cessation. JAMA Netw Open
explanatory insight. Cogn Sci 2016;40:1251–69. 2019;2:e192542.
26 McMillan B, Eastham R, Brown B, et al. Primary care patient records in the 55 Haupt CE, Marks M. AI-­generated medical advice—GPT and beyond. JAMA
United Kingdom: past, present, and future research priorities. J Med Internet Res 2023;329:1349–50.
2018;20:e11293. 56 Lee P, Goldberg C, Kohane I. The AI Revolution in Medicine: GPT-­4 and Beyond.
27 Dobscha SK, Denneson LM, Jacobson LE, et al. VA mental health clinician experiences Pearson Education, Limited, 2023.
and attitudes toward opennotes. Gen Hosp Psychiatry 2016;38:89–93. 57 Chen S, Kann BH, Foote MB, et al. Use of artificial intelligence Chatbots for cancer
28 Ralston JD, Yu O, Penfold RB, et al. Changes in clinician attitudes toward sharing visit treatment information. JAMA Oncol 2023;9:1459–62.
notes: surveys pre-­and post-­implementation. J Gen Intern Med 2021;36:3330–6. 58 Shryock T. AI special report: what patients and doctors really think about AI in health
29 DesRoches CM, Leveille S, Bell SK, et al. The views and experiences of clinicians care [Medical Economics]. 2023. Available: https://www.medicaleconomics.com/​
sharing medical record notes with patients. JAMA Netw Open 2020;3:e201753. view/ai-special-report-what-patients-and-doctors-really-think-about-ai-in-health-care
30 Blease C, Dong Z, Torous J, et al. Association of patients reading clinical notes with [Accessed 22 Aug 2023].
perception of medication adherence among persons with serious mental illness. JAMA 59 Nov O, Singh N, Mann D. Putting ChatGPT’s medical advice to the (Turing) test: survey
Netw Open 2021;4:e212823. study. JMIR Med Educ 2023;9:e46939.
31 Jia X, Pang Y, Liu LS. Online health information seeking behavior: a systematic review. 60 Marks M, Haupt CE. AI Chatbots, health privacy, and challenges to HIPAA compliance.
Healthcare (Basel) 2021;9:1740. JAMA 2023;330:309–10.
32 Drees J. Google receives more than 1 billion health questions every day [Becker’s 61 Taylor J. AMA calls for stronger AI regulations after doctors use ChatGPT to write
Health IT]. 2019. Available: https://www.beckershospitalreview.com/healthcare-​ medical notes [The Guardian]. 2023. Available: https://www.theguardian.com/​
information-technology/google-receives-more-than-1-billion-health-questions-every-​ technology/2023/jul/27/chatgpt-health-industry-hospitals-ai-regulations-ama
day.html [Accessed 22 Aug 2023]. [Accessed 22 Aug 2023].
33 Krištopaitytė E. Nearly 40% of Americans Google instead of seeing a doctor 62 Ayers JW, Poliak A, Dredze M, et al. Comparing physician and artificial intelligence
[Health News]. 2023. Available: https://healthnews.com/news/nearly-40-of-​ Chatbot responses to patient questions posted to a public social media forum. JAMA
americans-google-instead-of-seeing-physician/#:~:text=A%20new%20survey%​ Intern Med 2023;183:589–96.
20reveals%20that,of%20seeing%20a%20medical%20professional [Accessed 12 63 Petersson L, Erlingsdóttir G. Open notes in Swedish psychiatric care (part 2): survey
May 2023]. among psychiatric care professionals. JMIR Ment Health 2018;5:e10521.
34 EU. What did we use the Internet for in 2022? [EuroStat]. 2022. Available: https://ec.​ 64 Blease C, Kharko A, Dong Z, et al. Patients accessing their online records in England:
europa.eu/eurostat/web/products-eurostat-news/w/ddn-20221215-2 [Accessed 12 a survey of general practitioners’ experiences and opinions. Health Informatics
May 2023]. [Preprint].
35 Nyst A. 134 social media Statistics you need to know for 2023 [Search Engine 65 Blease C, Torous J, Dong Z, et al. Patient online record access in English primary
Journal]. 2023. Available: https://www.searchenginejournal.com/social-media-​ care: qualitative survey study of general practitioners’ views. J Med Internet Res
statistics/480507/#:~:text=Social%20Media%20Statistics%20Worldwide,increase%​ 2023;25:e43496.
20year%2Dover%2Dyear [Accessed 03 Sep 2023]. 66 Mold F, de Lusignan S, Sheikh A, et al. Patients’ online access to their electronic health
36 Chen J, Wang Y. Social media use for health purposes: systematic review. J Med records and linked online services: a systematic review in primary care. Br J Gen Pract
Internet Res 2021;23:e17917. 2015;65:e141–51.
37 Neely S, Eldredge C, Sanders R. Health information seeking behaviors on social media 67 Ferguson K, Fraser M, Tuna M, et al. The impact of an electronic portal on patient
during the COVID-­19 pandemic among American social networking site users: survey encounters in primary care: interrupted time-­series analysis. JMIR Med Inform
study. J Med Internet Res 2021;23:e29802. 2023;11:e43567.
38 Braun LA, Zomorodbakhsch B, Keinki C, et al. Information needs, communication and 68 Steitz BD, Sulieman L, Wright A, et al. Association of immediate release of test
usage of social media by cancer patients and their relatives. J Cancer Res Clin Oncol results to patients with implications for clinical workflow. JAMA Netw Open
2019;145:1865–75. 2021;4:e2129553.

88 Blease C. J Med Ethics 2024;50:84–89. doi:10.1136/jme-2023-109574

 Original research
69 American Psychiatric Association. The basics of augmented intelligence: some factors 76 Boyd E. Introducing GPT-­4 in Azure Openai service [Azure Microsoft]. 2023. Available:

J Med Ethics: first published as 10.1136/jme-2023-109574 on 23 November 2023. Downloaded from http://jme.bmj.com/ on August 11, 2024 by guest. Protected by copyright.
psychiatrists need to know now [American Psychiatric Association]. 2023. Available: https://azure.microsoft.com/en-us/blog/introducing-gpt4-in-azure-openai-service/
https://www.psychiatry.org/News-room/APA-Blogs/The-Basics-of-Augmented-​ [Accessed 31 Jul 2023].
Intelligence [Accessed 13 Aug 2023]. 77 Adams K. 31 numbers that show how big epic, cerner, allscripts & meditech are in
70 Attwooll J. Extremely unwise’: warning over use of ChatGPT for medical notes [Royal Healthcare [2021]. Available: https://www.beckershospitalreview.com/healthcare-​
Australian College of General Practitioners]. 2023. Available: https://www1.racgp.org.​ information-technology/31-numbers-that-show-how-big-epic-cerner-allscripts-​
au/newsgp/clinical/extremely-unwise-warning-over-use-of-chatgpt-for-m [Accessed meditech-are-in-healthcare.html [Accessed 31 Jul 2023].
08 Sep 2023]. 78 Adams K. Epic to integrate GPT-­4 into its EHR through expanded Microsoft
71 Blease C, Kharko A, Bernstein M, et al. Machine learning in medical education: a partnership [MedCity News]. 2023. Available: https://medcitynews.com/2023/04/epic-​
to-integrate-gpt-4-into-its-ehr-through-expanded-microsoft-partnership/ [Accessed 31
survey of the experiences and opinions of medical students in Ireland. BMJ Health
Jul 2023].
Care Inform 2022;29:e100480.
79 Cohen IG. What should ChatGPT mean for Bioethics? SSRN Journal 2023.
72 Blease C, Kaptchuk TJ, Bernstein MH, et al. Artificial intelligence and the future of 80 Davis W. Sarah Silverman is suing Openai and meta for copyright infringement [The
primary care: exploratory qualitative study of UK general practitioners’ views. J Med Verge]. 2023. Available: https://www.theverge.com/2023/7/9/23788741/sarah-​
Internet Res 2019;21:e12802. silverman-openai-meta-chatgpt-llama-copyright-infringement-chatbots-artificial-​
73 Hswen Y, Abbasi J. AI will—and should—change medical school says Harvard’s Dean intelligence-ai [Accessed 09 Sep 2023].
for medical education. JAMA 2023. 81 Lucchi N. ChatGPT: a case study on copyright challenges for generative artificial
74 Wisniewski H, Torous J. Digital navigators to implement Smartphone and Digital tools intelligence systems. Eur j Risk Regul 2023;1–23:1–23.
in care. Acta Psychiatr Scand 2020;141:350–5. 82 Minssen T, Vayena E, Cohen IG. The challenges for regulating medical use of ChatGPT
75 Opennotes. 2022. Available: https://www.opennotes.org/ and other large language models. JAMA 2023;330:315–6.

Blease C. J Med Ethics 2024;50:84–89. doi:10.1136/jme-2023-109574 89

 Interactive Learning Environments

ISSN: (Print) (Online) Journal homepage: https://www.tandfonline.com/loi/nile20

A review of opportunities and challenges of

chatbots in education

Gwo-Jen Hwang & Ching-Yi Chang

To cite this article: Gwo-Jen Hwang & Ching-Yi Chang (2021): A review of opportunities
and challenges of chatbots in education, Interactive Learning Environments, DOI:
10.1080/10494820.2021.1952615

To link to this article: https://doi.org/10.1080/10494820.2021.1952615

View supplementary material

Published online: 18 Jul 2021.

Submit your article to this journal

Article views: 122

View related articles

View Crossmark data

Full Terms & Conditions of access and use can be found at

https://www.tandfonline.com/action/journalInformation?journalCode=nile20
INTERACTIVE LEARNING ENVIRONMENTS
https://doi.org/10.1080/10494820.2021.1952615

REVIEW ARTICLE

A review of opportunities and challenges of chatbots in education

a b
Gwo-Jen Hwang and Ching-Yi Chang
a
Graduate Institute of Digital Learning and Education, National Taiwan University of Science and Technology,
Taipei, Taiwan; bSchool of Nursing, College of Nursing, Taipei Medical University, Taipei, Taiwan

ABSTRACT ARTICLE HISTORY

This study explores the trends of chatbots in education studies by Received 11 November 2020
conducting a literature review to analyze relevant papers published in Accepted 4 July 2021
the Social Science Citation Index (SSCI) journals by searching the Web
KEYWORDS
of Science (WoS) database. From the analysis results, it was found that Chatbot; artificial
the United States, Taiwan and Hong Kong are the top three intelligence; chatbot in
contributing countries or regions. In addition, most studies adopted education; systematic
quantitative methods in their research design, such as ANOVA (Analysis review; trend analysis
of variance), descriptive statistics, t test, and correlation analysis.
ANCOVA (Analysis of covariance) was the most frequently adopted
approach for comparing the performances or perceptions of different
groups of students. From the analysis results, the greatest proportion of
studies adopted guided learning, followed by no learning activities. It
was determined that the studies related to chatbots in education are
still in an early stage since there are few empirical studies investigating
the use of effective learning designs or learning strategies with
chatbots. This implies much room for conducting relevant research to
drive innovative teaching in terms of improving the learning process
and learning outcomes. Finally, we highlight the research gaps and
suggest several directions for future research based on the findings in
the present study.

Introduction
Chatbots are considered to be a technology application model that effectively promotes interperso-
nal communication and learning; they provide various types of information and knowledge through
interactive methods and easy-to-operate interfaces, and can even be used as a tool for personal con-
sultation (Muniasamy & Alasiry, 2020; Poncette et al., 2020; Yamada et al., 2016). As the popularity of
mobile technology grows, chatbots’ interactive learning method and the characteristics of not being
restricted by time and place are making their use increasingly popular (Zhou et al., 2020). In recent
years, a growing number of studies have explored the ways and effects of chatbot application in edu-
cation (Ferrell & Ferrell, 2020). Several studies have revealed the benefits of using chatbots in school
settings, including providing users with a pleasant learning experience by allowing for real-time
interaction (Kim et al., 2019), enhancing peer communication skills (Hill et al., 2015), and improving
learners’ learning efficiency (Wu et al., 2020).
With the advancement of Artificial Intelligence (AI) technology, scholars have begun to apply
machine learning and natural language technology to the development of chatbots, making their
application in education a new topic of academic research (Følstad & Brandtzæg, 2017). Smutny
and Schreiberova (2020) pointed out that these new technologies will enable chatbots to become

CONTACT Ching-Yi Chang [email protected]

Supplemental data for this article can be accessed https://doi.org/10.1080/10494820.2021.1952615.
© 2021 Informa UK Limited, trading as Taylor & Francis Group
2 G.-J. HWANG AND C.-Y. CHANG

a smart teaching assistant in the future; they also encouraged teachers to use AI-based chatbots in
classroom activities. On the other hand, although some recent studies have conducted literature
reviews on chatbot-related studies (Abd-alrazaq et al., 2019, 2020; Bendig et al., 2019; Kennedy
et al., 2012), these review studies mainly focused on the research of chatbots in health care rather
than in education. That is, there has been no review conducted to analyze the trends and challenges
of chatbots in the field of education. Scholars have pointed out that systematic reviews can help
scholars in this field, in particular novice scholars, to understand important research foci and appli-
cation directions, which can be used as a reference for planning future research as well as curriculum
design (Bond et al., 2019; Lai, 2020; Smith & Hill, 2019). To review the comparative study opportu-
nities (e.g. application domains and potential learning strategies) and challenges (e.g. research
design and research method) of the application of chatbots in education, this study reviewed articles
published in SSCI journals by referring to the Technology-based Learning Review (TLR) model
(Chang & Hwang, 2019; Hsu et al., 2012). We aimed to identify gaps in the application of and research
on chatbots in education, as well as proposing possible solutions to the challenges and gaps.

Literature review
The term chatbot refers to a computer program that provides services through dialogue (Brennan,
2006; Chopra et al., 2016). Early chatbots used keyword matching mechanisms (Weizenbaum, 1966)
or natural language processing mechanisms (Brennan, 2006; Hsieh, 2011; Melián-González et al.,
2021). Later, due to the advancement of computer and speech recognition technology, voice inter-
active interfaces began to appear (Guttormsen et al., 2011). For example, the Amazon Echo in 2014 is
a chatbot based entirely on voice interactions (Teja, 2020). Chatbots can be used for various pur-
poses, such as chatting, entertainment, data query, agent execution tasks, answering questions,
and dialogue exercises (Copulsky, 2019; Quarteroni & Manandhar, 2009). For example, a chatbot
can play the role of an agent to collect information from conversations to complete a form-filling
task such as booking air tickets or purchasing goods (Moriuchi et al., 2020).
Chatbots have been used in both formal and informal learning, with a view to analyzing the
content of learner conversations as well as providing learning content and feedback in an interactive
manner (Jia, 2009; Lee et al., 2009). Scholars have also found that interacting with chatbots generally
increases students’ learning interests, which has encouraged the development and application of
educational chatbots (Johnson, 2001). For example, Kerly and Bull (2006) used a chatbot to train uni-
versity students in how to negotiate with people; Tegos et al. (2015) used a chatbot to promote aca-
demically productive talk in a multimedia course, and found it effective in terms of improving
learners’ performance. Recently, Shorey et al. (2019) used a chatbot as a virtual patient to train
nursing students’ communication skills. On the other hand, some studies have reported the limit-
ations of using chatbots; for example, Fryer et al. (2017) conducted a study in a language course
to compare the performances of the students using chatbots for practice and those who practiced
via peer interactions; they found that after 3 weeks, students’ interest in using chatbots for practice
decreased, while their interest in interacting with peers remained the same. It can be seen from these
studies that using chatbots in educational settings has great potential; however, the limitations of
the approach also need to be considered.
At the same time, scholars have pointed out that a complete systematic review of specific topics can
help researchers better see important research trends in the field (Bond et al., 2019; Lai, 2020). Several
reviews of chatbot-related studies have already been conducted; for example, Abd-alrazaq et al. (2019)
reviewed the publications of chatbots in mental health, Abd-alrazaq et al. (2019) reviewed the studies
related to chatbots in health care, and Bendig et al. (2019) reviewed the studies of chatbots in clinical
psychology and psychotherapy. These review studies mainly focused on the use of chatbots in medical
applications rather than in the education domain. Therefore, this study aimed at analyzing the research
topics and trends of the application of chatbots in education through a systematic review. Based on
the results of this review, directions for future research can be provided.
 INTERACTIVE LEARNING ENVIRONMENTS 3

Research methods
Resources
In order to ensure full coverage of the target articles, the search time range for this article is from 1999
to 2020. According to the previous research recommendations of Chang et al. (2018), the Web of
Science (WoS) database was used due to the fact that it is a reliable and authoritative database.
Figure 1 shows the procedure of searching for the relevant studies from the WoS. On September 5,
2020, the authors used the keywords “chatbot*” or “Conversational Agent*” to search for the relevant
studies related to chatbots. A total of 551 publications were found. By limiting the publications to the
“article” type, 482 journal papers were obtained. Furthermore, by selecting the publications belonging
to the “Education Educational Research,” “Education Scientific Disciplines” or “Psychology Edu-
cational” categories, 30 publications were derived. After manual screening by two researchers, one
irrelevant article was excluded, and a total of 29 articles were selected for analysis.

Data distribution
Figure 2 shows the distribution of chatbots in education papers. The earliest research on chatbots in
education was published in 2005 by Graesser et al. (2005), who presented the use of a chatbot, Auto-
Tutor, with a natural language interface to talk to learners in order to enhance their learning engage-
ment and performance. From 2006 to 2012, several studies were published which reported different
research foci of using chatbots in education; for example, Hsieh (2011) designed a chatbot for a uni-
versity computer course by taking students’ cognitive styles into account. From 2013 to 2020, it was
found that the number of studies began to increase, and the fields of application became more diverse;
for example, Wang et al. (2017) added a Virtual Immersive Language Learning chat dialogue in a game
environment to support students’ English learning, while Feine et al. (2019) developed conversational
agents to guide learners to conduct scientific inquiry learning activities. In addition, some scholars have
applied chatbots to clinical psychology and psychotherapy consultation (Bendig et al., 2019).

Coding schemes
In order to analyze the research trend of chatbots, a coding scheme was determined based on the
Technology-based Learning Review (TLR) model (Chang & Hwang, 2019; Hsu et al., 2012), as shown in

Figure 1. Flowchart of data collection.

4 G.-J. HWANG AND C.-Y. CHANG

Figure 3. Accordingly, several dimensions have been taken into account in this review study, such as
learning domains, learning strategies, research design and analysis methods, as shown in Table 1.
In addition, Lai (2020) pointed out that the author’s productivity is valuable information for
helping researches in the field to identify and learn from those who have designed and conducted
a series of relevant studies. Therefore, this study also considers the author’s productivity. The formula
is based on Howard et al. (1987), who quantitatively analyzed the author’s contribution as follows:
(1.5n−i )
Score(i) = Number of citations ×
n n−k
k=1 1.5

In this formula, the contribution score for the i-th author of a paper is calculated based on the
weighting of the author order and the number of citations of the paper. For example, considering
the study of Hwang, Chu, and Lai (2017), the author order weightings for the three authors are
0.47, 0.32 and 0.21, based on the formula of setting i = 1 (first author), i = 2 (second author) and i
= 3 (third author), respectively, with n = 3 (number of authors). If the number of citations of the
paper is 100, in this case, the first author Hwang contributed 47 points, while Chu and Lai contrib-
uted 32 and 21 points, respectively.

Results
Learning domains
Figure 4 presents the data distribution for the learning domains. From 1999 to 2020, the most fre-
quently applied learning domains of chatbots in education studies were “Languages” (9 papers), fol-
lowed by “Engineering or computers” (6 papers). It was also found that chatbots had been applied to
various courses such as Science, Health, Medical or Nursing, Social science or social studies, Business
and management, Arts or design, and Mathematics. Five studies were categorized into “Non-
specified” including an investigation of students’ perceptions of using chatbots in learning.

Learning strategies
The distribution of the adopted learning strategies is shown in Figure 5. The only strategy adopted is
guided learning, with a total of 25 papers, while none of the other strategies was adopted in the
chatbot-based learning activities.

Figure 2. Distribution of chatbots in education publications.

 INTERACTIVE LEARNING ENVIRONMENTS 5

Figure 3. Coding scheme based on the Technology-based learning model.

Table 1. Classification descriptions.

Categories Explanation Reference
Learning Engineering or computers, Science (Physics, Chemistry, Biology, Environmental Science or Cheng et al.
domains natural Science), Health, Medical or Nursing, Social science or social studies (e.g. local (2020)
culture, history), Arts or design, Languages, Mathematics, Business and management, No
learning activities (position paper or review papers), Non-specified
Learning Guided learning, Peer assessment, Video sharing, Synchronous sharing, Issue-based Chang et al.
strategies learning, Computers as Mindtools, Project-based learning, Inquiry-based learning, (2018)
Contextual mobile learning, Game-based learning, Community service training, No
learning activities (position paper or review papers)
Research design quantitative method, qualitative method, mixed method, position paper, review paper Cheng et al.
(2020)
Analysis Descriptive statistics, t test, X2, ANOVA, ANCOVA Interview, CFA, co-citation network Cheng et al.
methods analysis, correlation, Generalized Linear, Latent and Mixed Models (GLLAMM), interview, (2020)
position papers

Research design
Figure 6 shows the research designs used in these chatbot in education studies. Among them, there
are 20 (69%) using quantitative methods, three (10%) using mixed methods, and two (7%) using quali-
tative methods. It can be seen that most of the researchers used an experimental research design to
investigate the issues of chatbots in education, and answered the research questions through the col-
lection and analysis of quantitative data. For instance, Bibauw et al. (2019) implanted a dialogue-based
chatbot to engage students in foreign language practice, and found it helpful for reducing the stu-
dents’ learning nervousness. In addition, there are four position papers (14%), but there are no
review papers, indicating that the development of this field is still in its initial stage (Figure 6).
These studies show that although various chatbots have been developed, indicating that the technol-
ogy is mature, there is still a gap in the application and study of chatbots in education.

Analysis methods
Table 2 shows the distribution of analysis methods adopted in the studies. It was found that most
studies used ANCOVA/MANCOVA to analyze the data (17%), followed by ANOVA (14%), interviews
(14%), descriptive statistics (10%) and t test (10%), correlation analysis (7%), GLLAMM (3%), CFA
(3%), X 2 (3%) and co-citation network analysis (3%) in the research on chatbots in education.
6 G.-J. HWANG AND C.-Y. CHANG

Figure 4. Data distribution of the chatbot research field in the WoS database.

This study further investigated the use of different analysis methods in three time periods, that is,
2005-2010, 2011-2016, and 2017-2020. In the first time period, the researchers mainly utilized
ANCOVA/MANCOVA, interview and t tests to analyze the data in their chatbot studies. The use of
ANOVA analysis began to increase in the second time period. In the third time period, from 2017 to
2020, when more than half of the studies were published, the analysis methods of ANCOVA/
MANCOVA and interview were mainly used. For example, Hsu (2020) conducted an experimental
study with a AI-based chatbot, and analyzed the collected data with the one-way multivariate of variance
(MANOVA). It was also found that X 2 was seldom adopted in these studies. One study related to X 2 was
conducted by Ortega-Morán et al. (2020) in a nursing course for minimally invasive surgery training.

Nationalities and journals

In our analysis of author nationality, we only counted the nationalities of the first author of the pub-
lished chatbot articles. According to the results, a great number of researchers have attempted to
implement chatbots. Figure 7 illustrates the distribution of the nationalities and areas which are
the sources of the published articles, the top three of which are the United States (7), Taiwan (4)
and Hong Kong (3).
Figure 8 shows the international journals which published articles on the use of chatbots in edu-
cation between 2005 and 2020. They included the British Journal of Educational Technology, Com-
puter Assisted Language Learning, Educational Technology and Society, IEEE Transactions on
Learning Technologies, and the Journal of Educational Computing Research. The British Journal of
Educational Technology, Computer Assisted Language Learning, and Educational Technology and
Society were the journals with the largest number of published papers, each with a total of four,
 INTERACTIVE LEARNING ENVIRONMENTS 7

Figure 5. Learning strategies adopted in the chatbot education studies.

followed by IEEE Transactions on Learning Technologies, with three, and the Journal of Educational
Computing Research, with a total of two articles.

Productive authors
Table 3 calculates the score and determines the top 10 research authors based on the citation fre-
quency of the journal articles used in the research and ranking of the authors. In 2005-2010, the most
productive researchers were from the United States. It is interesting to find articles published in the
early stage. During this period of 2005-2010, researchers did not show much interest in introducing
chatbots into education to promote learning; however, they were very interested in providing com-
puter technology to fully support students’ online learning, such as Jia’s (2009) computer-assisted
English learning chatbots based on text knowledge and reasoning. In the second phase of 2011-
2016, researchers were mainly from Sweden and Taiwan. By 2017-2020, it is interesting to find
that both the citation frequency of the articles and the ranking of the authors in the time period
are higher than those in the two earlier stages. This implies that the publications in 2017–2020

Figure 6. Percentage of research designs in chatbots.

8 G.-J. HWANG AND C.-Y. CHANG

Table 2. Percentage of analysis methods used in each stage.

Analysis methods 2005∼2010 (N = 4) 2011∼2016 (N = 7) 2017∼2020 (N = 20) 2005∼2020 (N = 31)
ANCOVA/MANCOVA 1(25%) 1(14%) 3(17%) 5(17%)
Interview 1(25%) 0 3(17%) 4(14%)
Descriptive statistics 0 1(14%) 2(11%) 3(10%)
ANOVA/MANOVA 0 2(29%) 2(11%) 4(14%)
t test 1(25%) 0 2(11%) 3(10%)
Correlation 0 1(14%) 1(6%) 2(7%)
Generalized Linear 0 1(14%) 0 1(3%)
Latent and Mixed Models (GLLAMM)
CFA 0 0 1(6%) 1(3%)
X2 0 0 1(6%) 1(3%)
co-citation network analysis 0 0 1(6%) 1(3%)
position papers 1(25%) 1(14%) 2(11%) 4(14%)

attracted more attention and had higher impacts than earlier studies, showing that researchers in
educational technology have greater interest in using chatbots to promote learning. This period
mainly focused on the advantages of teaching through action and conducting experiments, such
as Fryer et al.’s (2017) study which stimulated students’ interest in language courses, and compared
tasks performed with chatbots and peers. The trend of author productivity is also consistent with the
results of this research. On the whole, the top three authors with the highest research contributions
are Fryer, Tegos, and Veletsianos; they continue to propose the development and use of chatbot
education systems and to verify the relevant research.
On the other hand, the impact of a single citation with the analysis of the VOSviewer software in
Figure 9 shows that the article with the highest citation rate is Graesser et al. (2005) who proposed
chatbot education research; this paper was published in IEEE Transactions on Education, and had a
total of 231 citations at the time of this study. It was followed by Kumar and Rose (2011) with 45
citations, Veletsianos and Miller (2008) with 32 citations, Segedy et al. (2013) with 29 citations,
and Engwall (2012) with 20 citations.

Discussion
This research aims to review the research on the use of chatbots in education as a potential future
direction, and to discuss the benefits and research trends in this domain. The analysis results show

Figure 7. Countries with published chatbot articles.

 INTERACTIVE LEARNING ENVIRONMENTS 9

Figure 8. Journals with published articles on chatbots in education.

that researchers are more focused on comparing different chatbot systems to find more effective
chatbots in education application methods.

Learning domains
After reviewing and categorizing the articles, it was found that, regarding the application learning
domains, most of the studies on the use of chatbots in education have discussed the effectiveness
of chatbots in different domains. Most of the research in this domain has focused on Language
courses, followed by Engineering or computers, with little consideration of the topics of Arts or
design and Mathematics. By referring to the literature, it was found that arts or design courses gen-
erally focus more on fostering students’ hands-on competences than merely delivering knowledge;
moreover, the students and teachers in the field generally have less experience of using technologies
(Luo, 2017), which could be the reason why fewer studies reported the use of chatbots in Art or
design courses. Moreover, Mathematics courses focus more on computations or problem-solving
competences (Hardin & Horton, 2017), and hence it is much more challenging to implement the
required chatbots for mathematics than for those courses which mainly focus on the cognitive
levels of memorizing and comprehension. This implies the need to incorporate artificial intelligent

Table 3. Rankings of the top 10 highly cited authors (2005-2020).

2005∼2010 2011∼2016 2017∼2020 2005∼2020
Author Score Author Score Author Score Author Score
1 Veletsianos, G 102 Engwall, O 100 Fryer, LK 180 Fryer, LK 180
2 Graesser, AC 42 Hsieh, SW 100 Tseng, JJ 100 Tegos, S 102
3 Miller, C 40 Kumar, R 60 Hsu, LW 100 Veletsianos, G 102
4 De Carolis, B 38 Segedy, JR 47 Lin, MPC; Chang, D 60 Tseng, JJ 100
5 Chipman, P 28 Hassani, K 47 Tegos, S 60 Hsu, LW 100
6 Pizzutilo, S 26 Dyke, G 42 Wang, YF 47 Engwall, O 100
7 Haynes, BC 18 Tegos, S 42 Bibauw, S 47 Hsieh, SW 100
8 Cozzolongo, G 17 Rose, CP 40 Benotti, L 47 Demetriadis, S 68
9 Olney, A 12 Kinnebrew, JS 32 Chen, HL 47 Lin, MPC; Chang, D 60
10 Drozda, P 11 Nahvi, A 32 Song, D 47 Kumar, R 60
10 G.-J. HWANG AND C.-Y. CHANG

or technologies into the development of such educational systems as chatbots in the future, as indi-
cated by Hwang et al. (2020).

Learning strategies
From the learning strategies analysis results, it is clear that the greatest proportion of the studies
adopted guided learning, followed by no learning activities. It was also found that most of the
research using chatbots was not matched with learning strategies to design activities, which is a
direction for future education researchers to think about in the design of empirical research.

Research design
Most of these studies used quantitative methods as their research design methods, followed by
mixed methods and qualitative methods. It can be seen that most of the researchers used an exper-
imental research design to discuss the topic of the application of chatbots in education, and
answered the research questions through the collection and analysis of quantitative data.

Analysis methods
In terms of analysis methods, the most frequently adopted method was ANCOVA/MANCOVA, while
there was only one study which used Generalized Linear Latent and Mixed Models (GLLAMM) in the
2011–2016 period. From 2017 to 2020, the number of studies on the use of chatbots in education
increased significantly; at the same time, scholars began to analyze CFA, X 2 or to perform co-citation
network analysis. For instance, Bibauw et al. (2019) used co-citation network analysis of foreign
language learning with research synthesis and a conceptual framework of dialogue-based CALL.

Nationalities and journals

Analysis of the distribution of the countries and areas which are the source of the published articles
in this domain indicated that the United States, Taiwan and Hong Kong were the main contributing
countries, while the articles were mainly published in the British Journal of Educational Technology,
Computer Assisted Language Learning, and Educational Technology and Society, as well as in a
number of other international journals. Fryer, the most productive author with the highest contri-
bution score, adopted high-order statistical methods and attracted the attention of other research-
ers. This study also found that articles published between 2005 and 2010 did not have complicated
experimental designs.

Productive authors
Scholars have pointed out that the findings of systematic review can be used as an important basis
for follow-up research (Cheng et al., 2020; Lai, 2020). Through the understanding of the overall
research status, suggestions for future research can be made. For example, the rankings of the
authors from 2005 to 2020 indicate the active level of authors such as Fryer, Tegos, and Veletsianos;

Figure 9. Citation rate of research designs in chatbots.

 INTERACTIVE LEARNING ENVIRONMENTS 11

their research findings can be used as references by future researchers. Therefore, it is recommended
that in future research, a comprehensive analysis of the authors’ rankings can be carried out, using
equal contribution criteria (EC), the contribution percentage method (PCI), h-index or Google Scholar
ranking (Cheng et al., 2020; Lai, 2020). In addition, it is recommended that research be conducted to
investigate the impacts of adopting chatbots in learning activities with more advanced strategies
such as peer assessment, concept mapping and inquiry-based learning, to foster students’ higher
order thinking (e.g. critical thinking, reflective thinking and problem-solving competence). In the
future, different types of technical analysis, such as social network analysis, can be used for
papers with high citation rates in order to gain a comprehensive understanding of the research
trends.

Conclusions
From our review, it can be seen that the current research has rarely analyzed students’ learning
behaviors or structural equation analysis of causality. This shows that although education research-
ers have begun to pay attention to the research of chatbots in education, most of the studies still
used pre- and post-tests and questionnaires to evaluate their effectiveness. Only a few studies
have begun to pay attention to the behavior of students in the learning process; for example,
Fryer et al. (2017) designed an experiment combined with a chatbot system, performing a 12-
week experiment for students in foreign language classes, and observed and monitored the learning
behavior of the students in the language course. Fryer’s study found that a speaking task delivered
with chatbots significantly improved the students’ speaking performance, and also verified the
benefits of chatbot systems in education.
On the other hand, it was found that the only strategy adopted in the chatbot-based educational
studies was “guided learning,” which is a straightforward approach in which students are asked to
practice or search for data using application programs to complete learning sheets or to perform
learning tasks requested by the teacher. Other learning strategies, such as peer assessment, video
sharing, synchronous sharing, issue-based learning, computers as Mindtools (e.g. concept
mapping), project-based learning, and inquiry-based learning, generally aim to engage students
in peer interactions and higher order thinking. This implies that there is room to improve the
chatbot-based learning designs in future research.
The main contribution of this literature review is the finding that there is room for the develop-
ment of the application of chatbots in education research; for example, some rarely discussed topics
and applications may be a reference for thinking about the future research directions of chatbots. In
addition, the current research has rarely discussed chatbots in K-12 domains or topics, and there is
little empirical research on educational issues (for example, learners’ advanced thinking and learning
behaviors). On the other hand, chatbot research related to language education is most often cited,
which shows that the demand in this area is greater and more direct; therefore, the application
characteristics of language education can be used as a reference for publishing about the use of
chatbots in education in the future. On the whole, this study found that the research on chatbots
in education is in the early stage of development and is growing steadily; this result shows that
the research on chatbots in education has great development potential and is worthy of more invest-
ment by researchers in education technology. In addition, it is recommended that in the future, the
performance of teachers and learners can be analyzed from different angles, including the graphical
learning behavior analysis presented by chatbots in education systems, and the application of chat-
bots to improve students’ learning process and learning effectiveness.

Disclosure statement
No potential conflict of interest was reported by the author(s).
12 G.-J. HWANG AND C.-Y. CHANG

Funding
This study is supported in part by the Ministry of Science and Technology of Taiwan under contract numbers MOST-108-
2511-H-011-005-MY3, MOST-109-2511-H-011-002-MY3, MOST 109-2635-H-227-001, MOST 110-2511-H-038 -008 as well
as the Taipei Medical University under contract number TMU109-AE1-B25.

Notes on contributors
Dr. Gwo-Jen Hwang is a chair professor at the Graduate Institute of Digital Learning and Education, National Taiwan
University of Science and Technology. His research interests include mobile learning, digital game-based learning,
flipped classroom and AI in education.
Dr. Ching-Yi Chang is a PhD, RN, Assistant professor in the School of Nursing, College of Nursing, Taipei Medical Uni-
versity. Her research interests include mobile learning, digital game-based learning, flipped classroom and medical edu-
cation, nursing education, midwifery education, and AI in education.

ORCID
Gwo-Jen Hwang http://orcid.org/0000-0001-5155-276X
Ching-Yi Chang http://orcid.org/0000-0002-3146-2270

References
Abd-alrazaq, A. A., Alajlani, M., Alalwan, A. A., Bewick, B. M., Gardner, P., & Househ, M. (2019). An overview of the features
of chatbots in mental health: A scoping review. International Journal of Medical Informatics, 132, 103978. https://doi.
org/10.1016/j.ijmedinf.2019.103978
Abd-Alrazaq, A., Safi, Z., Alajlani, M., Warren, J., Househ, M., & Denecke, K. (2020). Technical metrics used to evaluate
health care chatbots: Scoping review. Journal of Medical Internet Research, 22(6), e18301. https://doi.org/10.2196/
18301
Bendig, E., Erb, B., Schulze-Thuesing, L., & Baumeister, H. (2019). The next generation: Chatbots in clinical psychology
and psychotherapy to foster mental health–a scoping review. Verhaltenstherapie, 29(4), 266–280. https://doi.org/
10.1159/000499492
Bibauw, S., François, T., & Desmet, P. (2019). Discussing with a computer to practice a foreign language: Research syn-
thesis and conceptual framework of dialogue-based CALL. Computer Assisted Language Learning, 32(8), 827–877.
https://doi.org/10.1080/09588221.2018.1535508
Bond, M., Zawacki-Richter, O., & Nichols, M. (2019). Revisiting five decades of educational technology research: A
content and authorship analysis of the British Journal of Educational technology. British Journal of Educational
Technology, 50(1), 12–63. https://doi.org/10.1111/bjet.12730
Brennan, K. (2006). The managed teacher: Emotional labour, education, and technology. Educational Insights, 10(2), 55–
65.
Chang, C. Y., & Hwang, G. J. (2019). Trends in digital game-based learning in the mobile era: A systematic review of
journal publications from 2007 to 2016. International Journal of Mobile Learning and Organisation, 13(1), 68–90.
https://doi.org/10.1504/IJMLO.2019.096468
Chang, C. Y., Lai, C. L., & Hwang, G. J. (2018). Trends and research issues of mobile learning studies in nursing education:
A review of academic publications from 1971 to 2016. Computers & Education, 116, 28–48. https://doi.org/10.1016/j.
compedu.2017.09.001
Cheng, S. C., Hwang, G. J., & Lai, C. L. (2020). Critical research advancements of flipped learning: A review of the top 100
highly cited papers. Interactive Learning Environments, 1–17. https://doi.org/10.1080/10494820.2020.1765395
Chopra, S., Gianforte, R., & Sholar, J. (2016). Meet percy: The CS 221 teaching assistant chatbot. ACM Transactions on
Graphics, 1(1), https://doi.org/10.1109/GHCI47972.2019.9071832
Copulsky, J. (2019). Do conversational platforms represent the next big digital marketing opportunity? Applied
Marketing Analytics, 4(4), 311–316.
Engwall, O. (2012). Analysis of and feedback on phonetic features in pronunciation training with a virtual teacher.
Computer Assisted Language Learning, 25(1), 37–64. https://doi.org/10.1080/09588221.2011.582845
Feine, J., Gnewuch, U., Morana, S., & Maedche, A. (2019). A taxonomy of social cues for conversational agents.
International Journal of Human-Computer Studies, 132, 138–161. https://doi.org/10.1016/j.ijhcs.2019.07.009
Ferrell, O. C., & Ferrell, L. (2020). Technology challenges and opportunities facing marketing education. Marketing
Education Review, 30(1), 3–14. https://doi.org/10.1080/10528008.2020.1718510
Følstad, A., & Brandtzæg, P. B. (2017). Chatbots and the new world of HCI. Interactions, 24(4), 38–42. https://doi.org/10.
1145/3085558
 INTERACTIVE LEARNING ENVIRONMENTS 13

Fryer, L. K., Ainley, M., Thompson, A., Gibson, A., & Sherlock, Z. (2017). Stimulating and sustaining interest in a language
course: An experimental comparison of chatbot and human task partners. Computers in Human Behavior, 75, 461–
468. https://doi.org/10.1016/j.chb.2017.05.045
Graesser, A. C., Chipman, P., Haynes, B. C., & Olney, A. (2005). Autotutor: An intelligent tutoring system with mixed-
initiative dialogue. IEEE Transactions on Education, 48(4), 612–618. https://doi.org/10.1109/TE.2005.856149
Guttormsen, M., Bürger, A., Hansen, T. E., & Lietaer, N. (2011). The SiRi particle-telescope system. Nuclear Instruments &
Methods in Physics Research Section A-Accelerators Spectrometers Detectors and Associated Equipment, 648(1), 168–
173. https://doi.org/10.1016/j.nima.2011.05.055
Hardin, J. S., & Horton, N. J. (2017). Ensuring that mathematics is relevant in a world of data science. Notices of the
American Mathematical Society, 64(9), 986–990. https://doi.org/10.1090/noti1572
Hill, J., Ford, W. R., & Farreras, I. G. (2015). Real conversations with artificial intelligence: A comparison between human–
human online conversations and human–chatbot conversations. Computers in Human Behavior, 49, 245–250. https://
doi.org/10.1016/j.chb.2015.02.026
Howard, G. S., Cole, D. A., & Maxwell, S. E. (1987). Research productivity in psychology based on publication in the jour-
nals of the American psychological association. American Psychologist, 42(11), 975–986. https://doi.org/10.1037/
0003-066X.42.11.975
Hsieh, S. W. (2011). Effects of cognitive styles on an MSN virtual learning companion system as an adjunct to classroom
instructions. Educational Technology & Society, 14(2), 161–174.
Hsu, L. (2020). To CALL or not to CALL: Empirical evidence from neuroscience. Computer Assisted Language Learning, 1-
24. https://doi.org/10.1080/09588221.2020.1750429.
Hsu, Y. C., Ho, H. N. J., Tsai, C. C., Hwang, G. J., Chu, H. C., Wang, C. Y., & Chen, N. S. (2012). Research trends in technology-
based learning from 2000 to 2009: A content analysis of publications in selected journals. Journal of Educational
Technology & Society, 15(2), 354–370.
Hwang, G. J., Chu, H. C., & Lai, C. L. (2017). Prepare your own device and determination (PYOD): A successfully promoted
mobile learning mode in Taiwan. International Journal of Mobile Learning and Organisation, 11(2), 87–107.
Hwang, G. J., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, challenges, roles and research issues of artificial intelligence
in education. Computers & Education: Artificial Intelligence, 1. https://doi.org/10.1016/j.caeai.2020.100001
Jia, J. (2009). CSIEC: A computer assisted English learning chatbot based on textual knowledge and reasoning.
Knowledge-Based Systems, 22(4), 249–255. https://doi.org/10.1016/j.knosys.2008.09.001
Johnson, W. L. (2001). Pedagogical agent research at CARTE. AI Magazine, 22(4), 85–85.
Kennedy, C. M., Powell, J., Payne, T. H., Ainsworth, J., Boyd, A., & Buchan, I. (2012). Active assistance technology for
health-related behavior change: An interdisciplinary review. Journal of Medical Internet Research, 14(3), e80.
https://doi.org/10.2196/jmir.1893
Kerly, A., & Bull, S. (2006). The potential for chatbots in negotiated learner modelling: A wizard-of-oz study. Lecture Notes
in Computer Science, 4053, 443–452. https://doi.org/10.1007/11774303_44
Kim, N. Y., Cha, Y., & Kim, H. S. (2019). Future English learning: Chatbots and artificial intelligence. Multimedia-Assisted
Language Learning, 22(3), 32–53.
Kumar, R., & Rose, C. P. (2011). Architecture for building conversational agents that support collaborative learning. IEEE
Transactions on Learning Technologies, 4(1), 21–34. https://doi.org/10.1109/TLT.2010.41
Lai, C. L. (2020). Trends of mobile learning: A review of the top 100 highly cited papers. British Journal of Educational
Technology, 51(3), 721–742. https://doi.org/10.1111/bjet.12884
Lee, C., Jung, S., Kim, S., & Lee, G. G. (2009). Example-based dialog modeling for practical multi-domain dialog system.
Speech Communication, 51(5), 466–484. https://doi.org/10.1016/j.specom.2009.01.008
Luo, M. (2017). Exploring the practice of visual communication design in the new era based on the view of “ Art+ tech-
nology”. Asian Journal of Social Science Studies, 2(3), 8–11. https://doi.org/10.20849/ajsss.v2i3.175
Melián-González, S., Gutiérrez-Taño, D., & Bulchand-Gidumal, J. (2021). Predicting the intentions to use chatbots for
travel and tourism. Current Issues in Tourism, 24(2), 192–210. https://doi.org/10.1080/13683500.2019.1706457
Moriuchi, E., Landers, V. M., Colton, D., & Hair, N. (2020). Engagement with chatbots versus augmented reality interactive
technology in e-commerce. Journal of Strategic Marketing, 1-15. https://doi.org/10.1080/0965254X.2020.1740766.
Muniasamy, A., & Alasiry, A. (2020). Deep learning: The impact on future eLearning. International Journal of Emerging
Technologies in Learning, 15(1), 188–199. https://doi.org/10.3991/ijet.v15i01.11435
Ortega-Morán, J. F., Pagador, B., Maestre-Antequera, J., Arco, A., Monteiro, F., & Sánchez-Margallo, F. M. (2020).
Validation of the online theoretical module of a minimally invasive surgery blended learning course for nurses: A
quantitative research study. Nurse Education Today, 89. https://doi.org/10.1016/j.nedt.2020.104406
Poncette, A. S., Rojas, P. D., Hofferbert, J., Sosa, A. V., Balzer, F., & Braune, K. (2020). Hackathons as stepping stones in
health care innovation: Case study with systematic recommendations. Journal of Medical Internet Research, 22(3),
e17004. https://doi.org/10.2196/17004
Quarteroni, S., & Manandhar, S. (2009). Designing an interactive open-domain question answering system. Natural
Language Engineering, 15(1), 73–95. https://doi.org/10.1017/S1351324908004919
14 G.-J. HWANG AND C.-Y. CHANG

Segedy, J. R., Kinnebrew, J. S., & Biswas, G. (2013). The effect of contextualized conversational feedback in a complex
open-ended learning environment. Educational Technology Research and Development, 61(1), 71–89. https://doi.
org/10.1007/s11423-012-9275-0
Shorey, S., Ang, E., Yap, J., Ng, E. D., Lau, S. T., & Chui, C. K. (2019). A virtual counseling application using artificial intelli-
gence for communication skills training in nursing education: Development study. Journal of Medical Internet
Research, 21(10), e14658. https://doi.org/10.2196/14658
Smith, K., & Hill, J. (2019). Defining the nature of blended learning through its depiction in current research. Higher
Education Research & Development, 38(2), 383–397. https://doi.org/10.1080/07294360.2018.1517732
Smutny, P., & Schreiberova, P. (2020). Chatbots for learning: A review of educational chatbots for the facebook messen-
ger. Computers & Education, 151. https://doi.org/10.1016/j.compedu.2020.103862
Tegos, S., Demetriadis, S., & Karakostas, A. (2015). Promoting academically productive talk with conversational agent
interventions in collaborative learning settings. Computers & Education, 87, 309–325. https://doi.org/10.1016/j.
compedu.2015.07.014
Teja, S. V. (2020). Chatbot using deep learning. Academic Leadership-Online Journal, 21(6), 428–438.
Veletsianos, G., & Miller, C. (2008). Conversing with pedagogical agents: A phenomenological exploration of interacting
with digital entities. British Journal of Educational Technology, 39(6), 969–986. https://doi.org/10.1111/j.1467-8535.
2007.00797.x
Wang, Y. F., Petrina, S., & Feng, F. (2017). VILLAGE—virtual immersive language learning and gaming environment:
Immersion and presence. British Journal of Educational Technology, 48(2), 431–450. https://doi.org/10.1111/bjet.
12388
Weizenbaum, J. (1966). ELIZA—a computer program for the study of natural language communication between man
and machine. Communications of the ACM, 9(1), 36–45. https://doi.org/10.1145/365153.365168
Wu, E. H. K., Lin, C. H., Ou, Y. Y., Liu, C. Z., Wang, W. K., & Chao, C. Y. (2020). Advantages and constraints of a hybrid model
K-12 e-learning assistant chatbot. IEEE Access, 8, 77788–77801. https://doi.org/10.1109/ACCESS.2020.2988252
Yamada, M., Goda, Y., Matsukawa, H., Hata, K., & Yasunami, S. (2016). A computer-supported collaborative learning
design for quality interaction. IEEE Annals of the History of Computing, 23(1), 48–59. https://doi.org/10.1109/MMUL.
2015.95
Zhou, L., Gao, J., Li, D., & Shum, H. Y. (2020). The design and implementation of xiaoice, an empathetic social chatbot.
Computational Linguistics, 46(1), 53–93. https://doi.org/10.1162/coli_a_00368
 Proceeding of International Conference on Emerging Technologies in Engineering, Biomedical, Management and
Science [ETEBMS-2016], 5-6 March 2016

Applications of Artificial Intelligence & Associated Technologies

Jatin Borana
Btech IV year, Department of Electrical Engineering, Jodhpur National University
[email protected]

Abstract—This paper reviews the meaning of artificial Strong AI

intelligence and its various advantages and disadvantages The principle behind Strong AI is that the machines could be
including its applications. It also considers the current progress made to think or in other words could represent human
of this technology in the real world and discusses the applications minds in the future. Thus Strong AI claims that in near
of AI in the fields of heavy industries, gaming, aviation, weather future we will be surrounded by such kinds of machine
forecasting, expert systems with the focus being on expert which can completely works like human being and machine
systems. The paper concludes by analyzing the future potential of
could have human level intelligence. If that is the case, those
Artificial Intelligence.
machines will have the ability to reason, think and do all
Keywords- Turing Test, Gaming Industry, Weather functions that a human is capable of doing. Current research
Predictions, Expert System is nowhere near creating strong AI, and a lively debate is
ongoing as to whether this is even possible .
I. INTRODUCTION
Weak AI

A RTIFICIAL intelligence (AI) is defined as

intelligence exhibited by an artificial entity to solve
complex problems and such a system is generally assumed
The principle behind Weak AI is simply the fact that
machines can be made to act as if they are intelligent. Weak
AI simply states that thinking like features can be easily
to be a computer or machine. Artificial Intelligence is an added to computer to make them more useful tools and this
integration of computer science and physiology Intelligence already started to happen. For example, when a human
in simple language is the computational part of the ability to player plays chess against a computer, the human player
achieve goals in the world. Intelligence is the ability to think may feel as if the computer is actually making impressive
to imagine creating memorizing and understanding, moves. But the chess application is not thinking and
recognizing patterns, making choices adapting to change and planning at all. All the moves it makes are previously fed in
learn from experience. Artificial intelligence concerned with to the computer by a human and that is how it is ensured that
making computers behave like humans more human like the software will make the right moves at the right times.
fashion and in much less time then a human takes. Hence it More examples of Weak AI are witness expert systems,
is called as Artificial Intelligence. Artificial intelligence can drive by wires cars and speech recognisation systems
be divided into parts according to philosophy of AI. Artificial Intelligence (abbreviated as AI) is the capability
a) Strong AI b) Weak AI of a device to perform activities, which would otherwise
only be expected of the human brain. These activities
include the capacity for knowledge and the ability to acquire
it. It also comprises of the ability to judge, understand
relationships and last but not least produce original thoughts.

Intelligence = perceive + Analyse + React

Also, there is a huge different between short term

memory and RAM. Short-term memory holds pointers to the
long-term memory where all the information is actually
stored while RAM stores data that is isomorphic to data
being held on a hard disk. Also, RAM has a memory limit
while there seems to be no capacity limit when it comes to
short-term memory.

Fig 1. Overview of Artificial Intelligence

64
Proceeding of International Conference on Emerging Technologies in Engineering, Biomedical, Management and
Science [ETEBMS-2016], 5-6 March 2016

computation. By looking at each of these in turn, we can

gain a better understanding of their role in AI, and how these
underlying disciplines have developed to play that role.

II. ADVANTAGES AND DISADVANTAGES

 One of the major advantages of artificial intelligence is
that its decisions are based on facts rather than
emotions. Even after our utmost efforts, it is a well-
known fact that human decisions are always affected in
a negative way by our emotions
 Unlike humans, machines with artificial intelligence do
Fig 2. Areas of Artificial Intelligence not need any sleep, thus overcoming the inherent
disadvantage of tiredness in humans
Turing Test: The Turing test is a test of a machine's
 Easier spreading of knowledge. Once an artificial mind
ability to exhibit intelligent behavior. The test was
is trained for something, it can be very easily copied to
introduced by Alan Turing in his 1950 paper Computing
the others reducing the time wasted in otherwise
Machinery and Intelligence. The original question behind
passing on knowledge to other humans through
this test was “Can machines think? “. The test proceeds as
training .
follows a human judge engages in a natural language
conversation with one human and one machine, each of  Lack of creativity in responses
which tries to appear human. All participants are placed in  Inability to explain the logic and reasoning behind a
isolated locations. If the judge cannot reliably tell the certain decision
machine from the human, the machine is said to have passed  Current development is at a stage where the AI cannot
the test. In order to test the machine's intelligence rather than know when there is no solution to a particular problem
its ability to render words into audio, the conversation is  Any malfunctioning can lead to the AI producing
limited to a text-only channel such as a computer keyboard wrong solutions and since it cannot explain the
and screen.” Sufficiently many interrogators are unable to reasoning behind its answer, blind reliance on AI can
distinguish the computer from the human being then it is to lead to problems
be concluded that the computer thinks.  Lack of common sense in reasoning can also cause
major problems
 It can be used to cause mass scale destruction if given
in the wrong hands

All this being said, one of the most concerning problem

with the development of AI is that it will soon start
substituting humans in every field thus causing a high rate of
unemployment, which would lead to depression, crime and
poverty. Also, there are some fields that require the human
touch and there is a growing sense of belief that machines
will quite possibly never be able to replace humans. The
caring behavior of nurses in hospitals is one example of a
job that humans feel machines will never be able to do
justice to.

Fig 3 Turing test for checking machine intelligence

III. CURRENT PROGRESS
Artificial Intelligence was created with the sole aim of
Roots of AI mimicking or even outperforming human minds. Thus it is
Artificial Intelligence has identifiable roots in a number of very important we question the fact whether it has actually
older disciplines, particularly been able to do so.
 Philosophy It cannot be ignored that the fact of AI is being used all
 Logic/Mathematics around us especially in the fields of medicine, robotics, law,
 Computation stock trading etc. It is being used in homes and big
establishments. such as military bases and the NASA space
 Psychology/Cognitive Science
station. NASA has sent out artificially intelligent robots to
 Biology/Neuroscience
planets so as to learn more about their habitat and
There is inevitably much overlap Example, between
atmosphere, with the intention of investigating if there is a
philosophy and logic, or between mathematics and
possibility of humans living on these planets. Expert systems

65
Proceeding of International Conference on Emerging Technologies in Engineering, Biomedical, Management and
Science [ETEBMS-2016], 5-6 March 2016

have been used by Mercedes Benz and other auto Artificial Intelligence robots have become very common in
manufacturers in the design of vehicle components, subway heavy industries and are employed in jobs that are otherwise
systems in Washington, D.C. use expert system software considered dangerous for humans. These robots also
controllers to cause subway trains to stop within 3 inches of increase the efficiency, as they do not need any break while
the right spot on the platform. These trains have motormen working thus overcoming the inherent disadvantage of
primarily to reassure passengers. AI has filtered into general tiredness in humans.
applications in these fields and has become so common that
it is not referred to as Artificial Intelligence anymore. Blind
supporters of AI would point to the time when AI Deep Blue
II defeated chess master Garry Kasparov to prove that
Artificial Intelligence can in fact be smarter than humans.
Though there is no doubt that the AI Deep Blue II won that
game, it is still probably one of the dumbest software alive.
The operators were programming the AI in every round
depending on the opposition’s last move. Also, the Deep
Blue II had studied all of Kasparov‟ s previous games while
the latter wasn‟ t given the same benefit. One can safely say
that even though the Deep Blue II AI defeated Kasparov, it
was never a fair fight to begin with.
Latest technologies like Xbox 360‟ s Kinect and
iPhone‟ s Siri use algorithms based on Artificial
Intelligence, but it is a well-known fact that these
technologies are a long way from being perfect. Thus we can
safely conclude that though Artificial Intelligence has made
a lot of progress in the past few decades, it is not at a level Fig 4 AI Applications
where in one can confidently state that it is now ready to
completely replace the human mind. That being said, large- C. Weather Forecasting-
scale research is now being conducted into the field of Neural networks are nowadays being used for predicting
proper simulation of the human brain. Cortex is a project by weather conditions. Past data is provided to the neural
Artificial Development Inc. and Swiss government's IBM network, which then analyses the data for patterns and
sponsored Blue Brain Project, are two main ventures, whose predicts the future weather conditions.
goal is to simulate the human brain.
D. Expert Systems-
Expert Systems are machines that are trained to have total
IV. APPLICATIONS expertise in specific areas of interest. They are developed to
Artificial Intelligence in the form of neural networks and solve the problems in niche areas. These systems use
expert systems has applications in almost all human statistical analysis and data mining to solve these problems
activities. The combination of high precision and low by deducing the solutions through a logical flow of yes-no
computation time makes AI a cutting edge technology. questions. An expert system is made up of 3 parts-
Robot ES‟ s are already taking over workshop level jobs in Knowledge base- It stores all the information, rules,
large industries, thus side lining humans into a more data and relationships that are needed by the expert system
supervisory role. Stock brokerage firms are now using to have total expertise in its area of interest
Artificial Intelligence to analyze data, make analysis and buy Inference engine- It seeks information from the
or sell stocks without the interference of any human beings. knowledge base on being presented with a query, analyses it
Some of the applications of Artificial Intelligence are as and responds with a solution or recommendation in the way
follows- a human expert would
Rule- It is a conditional statement that links the given
conditions to the final solution
A. Gaming Industry-
One of the most commonly known applications of AI in the
E. Data Mining or Knowledge Extraction:
gaming industry is its use in chess. Even though these
Data mining is a fast-growing area. Data mining is a part of
machines are not as intelligent as humans, they use brute
a process called KDD knowledge discovery in databases.
force algorithms and scan 100‟ s of positions every second
This process consists basically of steps that are performed
so as to determine the next move. As stated earlier, AI is
before carrying out data mining such as data selection, data
also being used in Microsoft Xbox 360‟ s Kinect for body
cleaning, pre-processing of data, and data transformation.
motion detection. But it is still in its infancy and requires a
"Data Mining is the use of computer algorithms to discover
lot more advancement for it to be used in day-to-day
hidden patterns and unsuspected relationships among
applications.
elements in a large data set. AI is a broader area than
B. Heavy industries-
machine learning. AI systems are knowledge processing

66
Proceeding of International Conference on Emerging Technologies in Engineering, Biomedical, Management and
Science [ETEBMS-2016], 5-6 March 2016

systems. Knowledge representation, knowledge acquisition, VI. CONCLUSION

and inference including search and control, are three
The computing world has a lot to gain or benefits from
fundamental techniques in AI
various AI approaches. Their ability to learn by example
makes them very flexible and powerful. Furthermore there is
no need to devise an algorithm in order to perform a specific
task i.e. there is no need to understand the internal
mechanisms of that task. They are also very well suited for
real time systems because of their fast response and
computational times which are due to their parallel
architecture. The goal of artificial intelligence is to create
computers whose intelligence equals or surpasses humans.
Achieving this goal is the famous “AI problem from last
decade researchers are trying to close the gap between
human intelligence and artificial intelligence.

REFERENCES
[1] George F Ludger “Artificial Intelligence - Structures
and strategies for complex problem solving” 5th Edition,
Pearson, 2009.
Fig 5 Pattern Matching AI Applications [2] Girish Kumar jha, "Artificial Neural Networks and its
applications" international journal of computer science and
F. Knowledge representation: issues 2005.
Data mining seeks to discover interesting patterns from large [3] Nils J Nilsson American Association for Artificial
volumes of Data. These patterns can take various forms, Intelligence" AI magazine 2005.
such as association rules, classification rules, and decision [4] Xindong Wu, Senior Member, IEEE "Data Mining: An
trees, and therefore, knowledge representation becomes an AI Perspective" vol.4 no 2 (2004)
issue of interest in data mining. [5] Satvika Khanna et al. “Expert Systems Advances in
Education” NCCI 2010 -National Conference on
Computational Instrumentation CSIO Chandigarh, INDIA,
V. FUTURE ASPECTS 19-20 March 2010
The use of artificial intelligence will lead to production of [6] Kaijun Xu.” Dynamic neuro-fuzzy control design for
machines and computers, which are much more advanced civil aviation aircraft in intelligent landing system. Dept. of
than what we have today. Speech recognition systems will Air Navig. Civil Aviation Flight Univ. of China 2011.
reach much higher levels of performance and will be able to [7] Eike.F Anderson.,”Playing smart artificial intelligence in
communicate with humans, using both text and voice, in computer games” The National Centre for Computer
unstructured English. There will be a great future some day Animation (NCCA) Bournemouth University UK.
for expert system applications in all aspects of health care, in [8] K.R. Chaudhary “Goals, Roots and Sub-fields of
both clinical and administrative areas, in improving patient Artificial Intelligence. MBM Engineering College, Jodhpur,
care and in allocation of financial, social, and other India 2012
resources. But when it comes to the question of Artificial
Intelligence creating machines, which are more intelligent
than human beings, no one seems to have the answer. Also,
even if it is possible, the amount of time it will take cannot
be predicted. It is also expected to have human brain features
like learning from experience, cognition and perception.
Whether human consciousness will be incorporated in these
machines is still not known. Robots in the future will be able
to do everybody’s work and will be faster and more efficient
as compared to human beings in doing it. If one is ill, they
can hire a robot nurse that will provide them with medicines
at proper intervals. Thus it can be safely said that Artificial
Intelligence is still in its embryonic stage and its future
depends only and only upon the scientists solving the
mystery of the human brain. Till that is done, no one can
make a conclusion of whether our future will be affected
positively or negatively by Artificial Intelligence.

67
Concept paper: Efficiency of Artificial Intelligence (AI) tools For STEM Education In
Malaysia
Mohammad Aniq Bin Amdan *, Naldo Janius and Mohd Aidil Hazidi Bin Kasdiah

Faculty of Education and Humanities, UNITAR International University, lot129, Alam Mesra, Plaza Utama (phase 3),
Sulaman, 88400 Kota Kinabalu, Sabah, Malaysia.

International Journal of Science and Research Archive, 2024, 12(02), 553–559

Publication history: Received on 01 June 2024; revised on 09 July 2024; accepted on 12 July 2024

Article DOI: https://doi.org/10.30574/ijsra.2024.12.2.1273

Abstract
The concept paper identifies the relationship of Artificial Intelligence (AI) towards teaching and learning in STEM
education. AI can really revolutionize STEM education if AI-powered tools are in place to ensure that each of the students
receives personalized instructions, intelligent tutoring, and interactive simulations. Not only this, but they even grade
assignments or quizzes that are submitted automatically and prove predictions with analytics to create efficiency and
effectiveness in STEM pedagogy. However, there is a limited quantity of primary research regarding the actual impacts
of such AI technologies. The paper will hence fill this gap by making an in-depth assessment of the application of AI tools
in STEM classrooms. If strategically deployed, AI has huge potential to improve student mastery in STEM, increase
learner motivation and autonomy, and allow teachers to become more personalized in their support. However, it also
identifies challenges of equitable access, bias in algorithms, and wishing that the teachers have robust training
programs. It thus proposes, based on the results, key recommendations that include developing ethical guidelines,
investing in professional development, and designing AI systems accommodating diverse learning needs. To be precise,
this research provides relevant empirical evidence and some practical guidance for education stakeholders to harness
the transformative power of AI for STEM learning

Keywords: Artificial intelligence; STEM education; Teacher; Pedagogy; Teaching and learning

1. Introduction
STEM competencies are crucial for Malaysia's economic growth and advancement as a country. STEM skills are relevant
not just to 'core STEM' jobs but also to many other careers in fact, Malaysia t targets to have 493,830 scientists and
engineers by the year 2020 and become globally competitive. (Shahali, E. H. M., 2016) There is an acute shortage of
engineers and professional STEM workers in Malaysia, which hinders economic growth and precipitates delays in
projects, related cost escalation, and outsourcing of engineering tasks. Indeed, this supply issue is to be reduced at the
very outset at the level of STEM education itself. STEM education gives rise to curiosity, inquisitiveness, critical thinking,
and problem-solving skills, every one of them valuable for any STEM and non-STEM career. (Haston, A. L.,2020) This
will help to bridge the gender gap because, as of now, women are less likely than men to graduate with STEM degrees.
Encouraging girls in STEM right from their girl-child age is important for economic and social progress. One of the most
important aspects of today's learning is STEM, its education that integrates the subjects of Science, Technology,
Engineering, and Mathematics on how one is to think critically by solving problems and innovating. This has over the
past few years received appreciable attention that integrating AI tools into STEM education is likely to enhance the
learning experience and improve student outcomes.

* Corresponding author: Mohammad Aniq Bin Amdan

Copyright © 2024 Author(s) retain the copyright of this article. This article is published under the terms of the Creative Commons Attribution Liscense 4.0.
 International Journal of Science and Research Archive, 2024, 12(02), 553–559

1.1. What is STEM Education?

STEM education is interdisciplinary in nature and focuses on the application of knowledge and acquisition of skills in
the disciplines of science, technology, engineering, and mathematics, which will be applied in mechanisms of problem-
solving. This will further enhance a hands-on, project-based learning environment and increase the understanding of
the interrelationship of these fields with their real-life applications (International Conference of the Learning
Sciences,2018). STEM education calls for students to form higher-order thinking skills and can implement what they
are learning so that they create, evaluate, and analyze per Bloom's taxonomy.

1.2. What is Artificial-Intelligence (AI)?

Artificial intelligence is the part of computer science that develops systems and algorithms that perform tasks that are
done by humans, such as decision making and problem solving. Artificial Intelligence has the potential to transform
many areas in education, including STEM, by enabling personalization of learning experiences, intelligent tutoring
systems, and adaptive learning environments. AI tools integrated into STEM education efficiently improve the process
of learning, entailing individualized learning experiences for students through intelligent tutoring systems and adaptive
learning environments. These tools can enable any student to learn at his or her pace, receive immediate feedback, and
make use of resources tailored to his or her needs and preferences when it comes to learning. AI transforms and
enhances most of the aspects of learning and teaching in STEM. Here is keyways AI is impacting STEM education:

1.3. Personalized Learning Experiences

AI algorithms can analyze student performance data to identify areas in which they need extra support and provide
targeted, adaptive learning experiences that are tailored to each student's needs and pace. This makes it easier for
students to learn efficiently and effectively. Kumar (2023)

1.4. Intelligent Tutoring Systems

Intelligent tutoring systems can provide extra support, much like a human tutor, to the conceptually challenged students
enabling them to grasp concepts difficult for them. This would then enable pupils to stay at grade level for their class.

1.5. Adaptive Assessments

AI can provide students with personalized assessments that progress at their level, challenging them with questions
that are neither too easy nor too hard. This builds student confidence and motivation while providing teachers with
more accurate data on performance.

1.6. Virtual and Augmented Reality

AI empowers the creation of virtual and augmented reality interactive experiences that really help students engage
better with, and understand, difficult STEM concepts. For example, through VR, they can walk through the human body
or explore historical sites; through AR they will see mathematical concepts.

1.7. Automated Grading and Lesson Planning

AI can support teachers by auto-grading assignments and exams, saving time while providing uniform and accurate
feedback. AI algorithms can also analyze student data to pinpoint exactly where they need the most extra help, providing
teachers with ideas for targeted lessons. Kumar (2023). However, several other issues must be sorted out regarding
integrating AI into STEM education, such as possible biases in AI algorithms, keeping a check on privacy, and training of
teachers. Altogether, this relationship between STEM education and AI epitomizes a very promising outlook for AI in
opening new opportunities to improve student learning and engagement at the initiation stage of STEM studies. While
AI continues to further developments, its role in STEM education is only going to increase to prepare students for the
future they will inherit, driven by AI.

1.8. Problem statement

 Evaluation of Effectiveness: On the effectiveness of AI-enabled STEM learning experiences, there is an exacting
lack of full empirical studies to estimate the actual effectiveness of such learning experiences over traditional
teaching techniques. Further rigorous studies should be informed to analyze the impact of AI on learner subject
outcomes, learner engagement, and long-term knowledge retention in STEM fields.
 Deepening AI Methods for Adaptive and Inclusive STEM Education: Few studies have examined how more
advanced AI techniques deep learning and natural language processing, especially can be combined with
theories of learning to produce more efficient and inclusive learning experiences in STEM. It, therefore, opens

554
 International Journal of Science and Research Archive, 2024, 12(02), 553–559

avenues for extending the application of AI in STEM education to further make it adaptive to a wide plethora of
diversity in needs and backgrounds.
 Empower Teachers to Harness AI in STEM Classrooms: Most of the teaching population lacks the knowledge
and resources for the proper use of AI as a teaching tool in the STEM classroom. Professional development
programs need to be created that train teachers in both ways of integrating AI into instruction and issues of
fairness, responsibility, and ethics in AI uses for STEM learning.

1.9. Research objective

Identify the Effectiveness of AI-Enabled STEM Learning Experiences: Conduct rigorous empirical studies to identify the
impact of AI-enabled STEM learning experiences on student learning outcomes, engagement, and long-term knowledge
retention, in comparison to traditional teaching methods.

Develop Advanced AI Techniques for Adaptive and Inclusive STEM Education: Identify how sophisticated AI methods
like deep learning and natural language processing can be integrated with learning theories to create STEM learning
experiences that are adaptive and inclusive to diverse student needs and backgrounds.

Identify ways of empowering teacher using the AI-Tools in STEM education.

2. Literature review
Within the past decade, artificial intelligence technologies have been incorporated into education at rapid speed as a
promising way to improve teaching and learning outcomes in these critically needed fields of science, technology,
engineering, and mathematics. (Idris, M.D.B.2018).AI can enable true individualization of learning experiences through
which instructional strategies are to become more efficient and effective by offering real-time insights to students and
teachers, especially in Malaysia where STEM education assumes a national importance, the efficiency and effectiveness
of AI tools in enhancing STEM learning assume very importance. Several research studies have been conducted on the
applications of AI in education. These AI-powered adaptive learning systems allow changes in content and pace based
on the performance of each student, partially improving the learning outcomes. For example, Razzaq et al. (2005)
demonstrated that students receiving mathematics through an AI-based tutoring system scored significantly higher in
tests as compared to those taught by conventional instructional strategies. On the other hand, Nye et al. (2018) claimed
that an AI-powered adaptive learning platform helped students achieve higher grades and lower failure rates within an
introductory computer science course.

Beyond personalization, AI can further engage students and motivate their interest in STEM topics. It will be able to
output interactive learning experiences that can make complex STEM topics more open to students through immersive
technologies like virtual and augmented realties powered by AI. According to Ibáñez et al., in a study on the use of an
AR-based learning system, the level of conceptual understanding and motivating learning in the electronics course had
increased. A study conducted by Mohamad et al. in 2018 demonstrated that AI-driven educational gaming is very
promising regarding enhancing STEM learning and its engagement among primary school students within the
Malaysian context. Nevertheless, even as AI shows so much potential for STEM education, equally critical challenges
and limitations must be sorted out. One critical concern in this area is the bias present within AI algorithms, which can
bring about unfair or discriminatory effects if not properly designed and subsequently monitored for such issues.
Further, the integration of AI into education brings certain highly relevant ethical concerns to the forefront, specifically
regarding data privacy, transparency, and teachers' roles within an AI-augmented learning environment.

AI adoption in STEM education within Malaysia is at a very nascent stage, with extremely limited empirical studies on
its effectiveness. For example, a study by Nordin et al. in 2016, examining Malaysian teachers' attitudes towards the
integration of AI in education, reported that although the teachers are generally aware of the potentials and benefits of
using AI, the number of means to train for and acquire resources on how to effectively teach through AI are few. A study
carried out by Mohamad et al. in 2019 focused on the use of AI-powered chatbots for language learning in English at
Malaysian secondary schools. The effects observed on student engagement and motivation were overall positive.

AI has further promise in STEM education through the growing recognition of individualized and adaptive approaches
to learning. By way of adopting personalized learning that changes instruction based on the needs and preference of the
student, it is probable to improve learning outcomes, particularly in areas of science, technology, engineering, and math
by Amdan, M. A. B., Janius, N., Jasman, M. N. B., & Bin, M. A. H. (2024). By tapping into AI, adaptive learning systems can
monitor data in real-time on students' performance, automatically changing the content, pacing, and instructional
strategy for individual students with a view to attaining maximum learning outcomes.

555
 International Journal of Science and Research Archive, 2024, 12(02), 553–559

Notably, in the Malaysian context, there is an acute need for personalized and adaptive STEM learning in view of the
very diverse socio-economic and cultural backgrounds of students. According to Kamaruddin et al., 2019, it was found
that Malaysian students from poor socioeconomic families, usually from rural areas, perform below average in the
acquisition of science, technology, engineering, and mathematics due to insufficiency and a lack of resources and
support. AI-enabled adaptive learning platforms can potentially close these gaps by providing individual support and
intervention if something is going wrong with the student. Applications of AI in STEM education might be an innovative
way to play out the strengthening of the teaching role and instructional practices in teachers. AI-powered tools can
automate routine tasks, freeing up time for teachers from activities like grading and lesson planning to focus on more
meaningful interactions with learners. Moreover, real-time insights about student performances are provided by AI-
driven learning analytics to the teachers, by which teachers can identify the difficulty areas and work on their teaching
strategies accordingly in this context, Zawacki-Richter et al. (2019). get Line However, there are several challenges to
be addressed which will finally make AI embedding successful within the Malaysian STEM classroom setting. The first
and most important is building the capacity and confidence of teachers in using AI technologies effectively. A study
conducted in 2016 by Nordin et al. has revealed that, very few Malaysian teachers have received some training or are
ready with the resources to integrate technology, let alone artificial intelligence, within pedagogical practice. Such a
chasm will now need to be filled with thorough professional development programs purposed at empowering teachers
to empower AI in the service of STEM education.

One more challenge is that implementation of AI in STEM education addresses the equity and inclusiveness concerns of
Malaysian students' various needs and backgrounds. As Suresh and Guttag, 2019, raise if not designed and deployed
with fairness and ethical considerations as integral priorities, AI algorithms could perpetuate biases and discrimination.
thriving cooperation directly with educators, AI researchers, and the community stakeholders in the process of creating
AI-empowered learning solutions within the gymnasium of STEM, which is accessible, inclusive, and responsive to the
needs of all students. (Janius, N. ,2022)

In a nutshell, the literature suggests that AI has huge potential for improving STEM education in several ways through
personalization of learning, increasing students' engagement, and deriving useful insights for teachers. In a bid to arrive
at better understanding, further rigorous research is required with the efficiency and effectiveness of AI tools in
Malaysian settings, especially regarding learning outcome enhancement, redressing equity and inclusion, and making
teachers better placed at harnessing AI responsibly and effectively in their classrooms. It shall hence be seen that steps
taken to deal with challenges in AI deployment and its limitations to STEM education will lead to an equitably
distributed dividend of the technology toward overall improvement in STEM learning across Malaysia.

3. Expected outcomes of the studies

Table 1 Table of outcome (1.0)

Identify the Effectiveness of A.I. Enabled STEM Learning Experiences in Malaysia:

Establish through rigorous empirical studies whether and how AI-enabled STEM learning experiences work to bring
about better student learning outcomes, enhance student engagement, and promote long-term knowledge retention
in Malaysia, compared to traditional modes of teaching.
Determine how AI tools and technologies benefit or constrain STEM education within the Malaysian context.
Make evidence-based recommendations to optimize the integration of AI in Malaysian STEM classrooms for maximal
efficacy.
To develop advanced AI techniques for adaptive and inclusive STEM education in Malaysia
Research how latest AI techniques, deep learning, and natural language processing can combine with theories of
learning into the design of adaptive and responsive STEM learning experiences for diverse Malaysian learners.
Design and develop AI-driven, tailored STEM learning platforms that can individualize content, pace, or instructional
strategies per student, depending on their performance and preference.
Examine how these more advanced, AI-powered STEM learning experiences may improve access, boost engagement,
and drive further learning outcomes for currently underrepresented or disadvantaged groups of students within
Malaysia.

556
 International Journal of Science and Research Archive, 2024, 12(02), 553–559

4. Methodology
Table 2 Table of research methodology (2.0)

Research The study expected to be qualitative research design will include in-depth interviews with thematic
design data analysis. The qualitative approach has been chosen to gain deep understanding regarding what
educators, students, and other stakeholders feel about AI tools in STEM education (user experiences,
perceptions, and attitude).
Data In depth interview with participants.
collection Expected 10 participant from:
STEM educator in all levels (primary, secondary and tertiary)
Student enrolled in STEM education
Educational Administrator
60 minutes – 90 minutes Interview session
The interview will be recorded using audio recording
Data Thematic data analysis: Themes will be developed based on recurring patterns and significant insights
analysis related to the research questions.

4.1. Significance of the research

Integration of artificial intelligence into STEM can become a game-changing opportunity for improving learning
outcomes and preparing the student for future technological advancement. The research aims to explore the efficiency
of AI tools within Malaysian STEM classrooms, hence bridging an existing empirical literature gap on its overall impact.
Knowing the effectiveness of AI in this context gives valuable insights to educators, policy makers, and stakeholders.
Firstly, in finding a Challenges in Education, such resource constraints, variations of teacher expertise, and personalized
learning experiences are many of the different challenges characterizing STEM education in Malaysia. Traditional
teaching modes rarely consider these needs, which results in imbalanced student performance. AI tools shall be able to
address these challenges through the provision of personal learning experience and intelligent tutoring systems to
evaluate the adaptive characteristics of students. After that, Enhancing the teaching and learning using AI in STEM
education. AI tools can really help drive change in STEM education, automate administrative tasks so teachers can focus
on individualized instruction and mentorship. AI-powered platforms would aid teachers in grading, progress
monitoring and reporting, and failure detection in students' knowledge to make it easier for them to formulate
meaningful and targeted instructional strategies. In addition, innovative forms of teaching through virtual labs and
interactive simulations will make complex concepts easier and more compelling to study. Besides, this integration of AI
tools into STEM education is one significant move toward bringing educational equity. In Malaysia, there are differences
in the extent of the educational resources and opportunities available both at urban and rural levels. AI tools can fill this
gap with quality educating content and resources for students, regardless of their location. AI-powered learning
platforms provide rich, dynamic, interactive experiences for students at remote locations, amounting to the reduction
of inequalities in education. Furthermore, the Malaysian government strongly recognizes how STEM education
contributes to economic growth and innovation. The Malaysian Education Blueprint 2013-2025 is one such initiative,
focusing on improving STEM education for the surging demands of the workforce in the future. Application of AI tools
in STEM education serves both steps at the national level by generating a technologically efficient and innovative
generation. This study may enlighten the policymakers and can lead toward strategizing and developing policies for
effective AI adoption in the education sector.

Moreover, the future workforce will be characterized by rapid technological advancement and a need to acquire skills
in AI, Machine Learning, Data Science, and related areas of expertise in STEM. AI tools in STEM education can help
position Malaysian students better in these growing job markets. Most AI tools allow practical projects, preferably
learning concepts that incorporate the latest technologies at the highest possible level to enrich technical skills and
digital literacy for employability. This contribution gives the enhancement needed for better employability and
contributes to Malaysia's continuous economic growth with technological innovation. Then, AI could foster a culture of
innovation and research within students and educators in STEM education. AI makes possible tools to realize
collaborative learning and research projects utilizing students in real-world solving, innovation, and optimization.
Educators could apply AI to pedagogical research, data analysis of student performance structures, and bringing about
evidence-based teaching practices. This kind of continuous improvement will enhance STEM education and add new
knowledge and technologies. In addition, AI can provide flexible and personalized learning opportunities throughout

557
 International Journal of Science and Research Archive, 2024, 12(02), 553–559

an individual's life. Continuous learning and upskilling are necessary for survival in the job market due to the rapidly
changing world with respect to technological advancement. AI-powered learning platforms can offer learners a
personalized pathway of learning including adaptive content and real-time feedback. This allows a learner who wants
to gain new skills and knowledge to acquire them at any time in their lives. This would foster a workforce of skilled,
adaptable labor that will meet evolving economic demands in a better way. Then, his study has a host of implications
for the EdTech industry in Malaysia. The identification of the AI tools that are most effective in impacting STEM
education will help in guiding the development or improvement of offered EdTech products and services. It will harness
EdTech players for deep insights into the development of innovative, AI-driven solutions for the Malaysian educational
system, atop specific needs and challenges. This can engender an EdTech sector and new business opportunities. Lastly,
while this study was conducted in Malaysia, it can draw further implications for other countries that are currently facing
related challenges in STEM education. Informed by the insights of this study, best practices at the global level on the
integration of AI in education would also benefit from it, therefore feeding into the debate at the international level. The
sharing of knowledge and experiences among countries enables learning from each other and collaborative strategy
development to enhance STEM through AI. This study, foreseen to empower STEM education in Malaysia with AI tools.
The importance of the study can also be affirmed by its impact on resolving educational challenges, promoting equity,
adhering to national policies, preparing students for the future workforce, supporting innovation, enabling lifelong
learning, and finally providing information to the EdTech industry. It’s expected to help in the development of effective
AI-driven educational strategies and policies toward enhancing the quality of STEM education in Malaysia and
remaining competitive in the global knowledge economy.

5. Conclusion
In conclusion the promise of AI tools in the integration with STEM education in Malaysia is very bright, with an
improvement in learning outcomes, promotion of educational equity, and preparing students for technological
innovation. Therefore, Malaysia can explore the transformative potentials of AI to improve the quality of STEM
education by addressing these challenges and with monetary investment in professional development, further research
in this area and ethical deployment of AI. The paper provides some useful insights and practical recommendations for
educators, policymakers, and stakeholders to ensure the fullest possible utilization of AI for STEM education in
promoting economic growth and innovation for the nation. The results are relevant not only at the national level for
Malaysia but also for other countries that struggling with similar challenges in STEM education-related issues.

Compliance with ethical standards

Acknowledgement
The author would like to take this opportunity to extend our heartfelt appreciation to UNITAR INTERNATIONAL
UNIVERSITY for the publication of this research and who has contributed to the success of this publication, whether it
be directly or indirectly via their contributions. Without their assistance, this paper would not have been able to be
carried out in a perfect manner, I am extremely grateful for the assistance. In conclusion, I would like to express my
gratitude to the reviewers who remained anonymous for their insightful suggestion, which significantly enhanced the
quality of this paper.

Disclosure of conflict of interest

No conflict of interest to be disclosed.

References
[1] Amdan, M. A. B., Janius, N., Jasman, M. N. B., & Bin, M. A. H. (2024). Advancement of ai-tools in learning for technical
vocational education and training (TVET) in Malaysia (empowering students and tutor).
[2] Cheng, K. H., & Tsai, C. C. (2019). A case study of immersive virtual field trips in an elementary classroom:
Students' learning experience and teacher-student interaction. Computers & Education, 140, 103600
[3] Fadel, N. S. M., Ishar, M. I. M., Jabor, M. K., Ahyan, N. A. M., & Janius, N. (2022). Application of soft skills among
prospective TVET teachers to face the industrial revolution 4.0. Malaysian Journal of Social Sciences and
Humanities (MJSSH), 7(6), e001562-e001562.
[4] Haston, A. L. (2020). Critical Thinking in Higher Education STEM: A Qualitative Faculty Perspective (Doctoral
dissertation, Franklin University).

558
 International Journal of Science and Research Archive, 2024, 12(02), 553–559

[5] Hassan, Z. B., Janius, N., Atan, N. A., & Idris, M. D. B. (2018). Assessment of Service Learning in Higher Education
at Universiti Teknologi Malaysia. Advanced Science Letters, 24(1), 30-33
[6] Ibáñez, M. B., Di-Serio, A., Villarán, D., & Delgado-Kloos, C. (2020). Experimenting with electromagnetism using
augmented reality: Impact on flow student experience and educational outcomes. Computers & Education, 109,
154-165.
[7] Ismawi, S. N. M., Ishar, M. I. M., & Janius, N. (2022). Workability Elements ff Post-Diploma Students in Construction
Technology From Vocational Colleges in Malaysia. Journal Teknikal & Kajian Sosial (JUTEKS), 21(1).
[8] International Conference of the Learning Sciences. (2018). ICLS 2018: Proceedings of the 13th International
Conference of the Learning Sciences, Volume 1 [Conference proceedings]. University of Twente
[9] Janius, N., Ishar, M. I. M., Bang, P., Sid, R., & Wong, G. (2023). The Effects of Music towards the Mathematical
Language Development of Children. Malaysian Journal of Social Sciences and Humanities (MJSSH), 8(4), e002249-
e002249.
[10] Janius, N., Hassan, Z. B., Atan, N. A., & Idris, M. D. B. (2018). Planning in Service Learning Project at High
University. Advanced Science Letters, 24(1), 34-37.
[11] Janius, N., Ishar, M. I. M., Yusof, Y., Bang, P., Sid, R., & Wong, G. (2023). Belajar Sambil Bermain di dalam Kelas Pada
Peringkat Pendidikan Awal Kanak-Kanak. Malaysian Journal of Social Sciences and Humanities (MJSSH), 8(4),
e002248-e002248.
[12] Janius, N., Jahadi, N. E. H. B., Abdullah, S. N. L. B., & Ling, M. S. (2023). Kesedaran Pendidikan Keusahawanan
Terhadap Kerjaya Kanak-Kanak di Masa Hadapan: Satu Tinjauan Literatur. Malaysian Journal of Social Sciences
and Humanities (MJSSH), 8(5), e002286-e002286.
[13] Kulik, J. A., & Fletcher, J. D. (2016). Effectiveness of intelligent tutoring systems: a meta-analytic review. Review
of educational research, 86(1), 42-78.
[14] Kumar, A. (2023). The revolutionary computer technology: Artificial intelligence that is changing our world & its
impact on different jobs. International Journal for Research in Applied Science and Engineering Technology
(IJRASET), 11(6), 1234-1245. https://www.ijraset.com/best-journal/the-revolutionary-computer-technology-
artificial-intelligence-that-is-changing-our-world--its-impact-on-different-jobs
[15] Mohamad, M., Arif, F. K. M., Shariff, N. M., & Halim, N. D. A. (2018). Integrating game-based learning in STEM
education. In Proceedings of the 2nd International Conference on Applied Science and Technology (ICAST 2017)
(Vol. 2030, No. 1, p. 020116). AIP Publishing LLC.
[16] Mohamad, M., Hamzah, H., Halim, N. D. A., & Rashid, R. A. (2019). Chatbot: A Conversational Agent to Improve
English Language Proficiency among Secondary School Students. International Journal of Emerging Technologies
in Learning, 14(18).
[17] Nordin, N., Hamzah, M. I., Yunus, M. M., & Embi, M. A. (2016). Challenges of Malaysian Teachers in Adapting to the
Use of Technology. In 5th International Conference on Information and Communication Technology for The
Muslim World (ICT4M) (pp. 1-4). IEEE.
[18] Nye, B. D., Graesser, A. C., & Hu, X. (2018). AutoTutor and family: A review of 17 years of natural language tutoring.
International Journal of Artificial Intelligence in Education, 29(3), 393-399.
[19] Razzaq, L., Heffernan, N., Koedinger, K. R., Feng, M., Nuzzo-Jones, G., Junker, B., ... & Rasmussen, K. (2005). Blending
assessment and instructional assisting. In Proceedings of the 12th International Conference on Artificial
Intelligence in Education (pp. 555-562).
[20] Shahali, E. H. M., Halim, L., Rasul, M. S., Osman, K., & Zulkifeli, M. A. (2016). STEM learning through engineering
design: Impact on middle secondary students’ interest towards STEM. EURASIA Journal of Mathematics, Science
and Technology Education, 13(5), 1189-1211.
[21] Suresh, H., & Guttag, J. V. (2019). A framework for understanding unintended consequences of machine learning.
arXiv preprint arXiv:1901.10002.
[22] Yakop, A., Ishar, M. I. M., & Janius, N. (2024). PERSEPSI PELAJAR SEKOLAH MENENGAH PEREMPUAN DI JOHOR
BAHRU DALAM PEKERJAAN 3D. Journal of Modern Education, 6(20), 193-206.
[23] Zulkafli, S. S. A., Ishar, M. I. M., & Janius, N. (2024). PENGARUH BIDANG EKONOMI RUMAH TANGGA (ERT) DALAM
KALANGANPELAJAR SEKOLAH MENENGAH LELAKI DI JOHOR BAHRU. Journal of Modern Education, 6, 20.
[24] Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review of research on artificial
intelligence applications in higher education–where are the educators?. International Journal of Educational
Technology in Higher Education, 16(1), 1-27.

559
 Chatbot for University Related FAQs

Bhavika R. Ranoliya∗ , Nidhi Raghuwanshi∗ and Sanjay Singh∗†

∗
Department of Information and Communication Technology
Manipal Institute of Technology, Manipal University, Karnataka-576104, India
† Centre for Artificial and Machine Intelligence

Manipal University, Karnataka-576104, India

[email protected], [email protected], [email protected]

Abstract—Chatbots are programs that mimic human conver- AIML and LSA is used for creating chatbots. AIML is
sation using Artificial Intelligence (AI). It is designed to be the Artificial Intelligence Markup Language (AIML) and Latent
ultimate virtual assistant, entertainment purpose, helping one to Semantic Analysis (LSA) are used for developing chatbots,
complete tasks ranging from answering questions, getting driving which are used to define general queries like how do you
directions, turning up the thermostat in smart home, to playing do?, how can I help you etc. This pattern can also be used
one’s favorite tunes etc. Chatbot has become more popular in
business groups right now as they can reduce customer service
to give random responses for same query. LSA [3] is a Latent
cost and handles multiple users at a time. But yet to accomplish Semantic Analysis,which is utilized to discover likenesses be-
many tasks there is need to make chatbots as efficient as possible. tween words as vector representation. So that the unanswered
To address this problem, in this paper we provide the design of a queries by AIML will be viewed as a reply by LSA. Most
chatbot, which provides an efficient and accurate answer for any chatbots basically search for keywords, phrases, and examples
query based on the dataset of FAQs using Artificial Intelligence that have been customized into their databases, yet some utilize
Markup Language (AIML) and Latent Semantic Analysis (LSA). more propelled strategies. So far no chatbot has possessed the
Template based and general questions like welcome/ greetings capacity to totally trick people into trusting it as one of them
and general questions will be responded using AIML and other through its information of regular dialect [4]. In this paper
service based questions uses LSA to provide responses at any time the need for chatbot in education domain is highlighted and
that will serve user satisfaction. This chatbot can be used by any
University to answer FAQs to curious students in an interactive
designed to provide visitor satisfaction.
fashion. Rest of the paper is organized as follows. Section II reviews
Keywords—Artificial Intelligence Markup Language(AIML), the related work, which is not exhaustive but only the most
Latent Semantic Analysis(LSA), Pattern Matching, Chatbot, Human relevant ones. Section III briefs about existing applications of
Computer Interaction(HCI) chatbots particularly ELIZA and ALICE. Section IV discusses
about AIML. Section V presents the proposed chatbot design.
Section VI gives some screen shots of the chatbot response
I. INTRODUCTION and finally section VII concludes the paper.
Todays era is having many web based services like E-
business, Entertainment, Virtual assistance and many more. II. R ELATED W ORK
There is drastic increase in the world of web service, where
every thing is now getting associated with web. It is very user Weizenbaum J shows psychological issues related to the
friendly approach to avail everything to doorstep. There are ELIZA [1]. Thomas T [3] provided the way by which the
different types of customer service available like live chat chatbot is planned in a manner that for single template, it
support service, phone(telephone) services. But for all such gives irregular responses. LSA based inquiries are giving right
support services provided by human to human takes time to reactions for random responses.
answer customers query. As the number of clients increases Rashmi S and Kannan Balakrishnan [5] has provided
the waiting time increases as well, which results in poor client the execution of a curious chatbot, which finds the missing
satisfaction. information in inquiry and tests the queries to clients to
One of the important goals in the field of Human Computer gather information that are required to answer the question.
Interaction (HCI) is the outline of normal and instinctive Identification of missing data and querying same to provide
connection modalities. Specifically, numerous endeavors have accurate response. D. Aimless and S. Umatani [6] illustrated
been committed to the improvement of frameworks to com- the usage of AIML tags that will describes the functionality
municate with the client in a characteristic language [1]. and features of each tags associated to build AIML based
Computer based chatbots are getting to be distinctly famous chatbot.
as an intuitive and successful open framework between human
and machines. Chatbot is a manufactured substance that is III. E XISTING A PPLICATIONS
intended to reproduce a clever discussion with human ac-
complices through their regular language. Currently, chatbots ”Chatbots are program that interact with humans using nat-
are utilized by a great many web clients to intercede access ural language” [5]. Chatbots are used in many organizational
to information or learning bases and furthermore to do non domains where it can replace humans. Such systems are based
specific discussions [2]. on the ways ELIZA or ALICE communicates.

978-1-5090-6367-3/17/$31.00 ©2017 IEEE 1525

 1) ELIZA: ELIZA [5] is the primary chatbot created by class of data object called an AIML object that describes the
Joseph Weizenbaum utilizing a keyword coordinating strategy. behavior of computer programs. It consist of units or tag called
The thought was to persuade the client info and look for certain topics and categories. In AIML, categories are basic unit of
keywords, if a catchphrase was discovered then the appropriate knowledge. Each category consists of pattern which contains
response was recovered. In the event that a catchphrase is not input and template which contain answer of chatbot. Besides,
present then ELIZA would proceed, as per indicated principles, there are some optional context called “that” and “topic”. Tag,
to get more data from the client to keep the discussion < that > contain chatbot’s last utterance and < topic >
going. Consider the accompanying case to see how this would contain a collection of categories together. AIML consists of
function. words, underscore, symbols and wild card symbol like and
∗.
User: How would I be able to get good result?
There are three types of AIML classes:
At that point the chatbot would recognize the catch-
phrase result and would discover an association 1) Atomic categories
amongst result and subject and it would react with 2) Default categories
a question. 3) Recursive categories
ELIZA: Please provide more information about your subject. Atomic category : It is a sort of AIML classification
The chatbot does not comprehend what it is stating. It where there is a exact match.
just creates about as indicated by guidelines and for < category >
this situation there was a control connecting family < pattern >How are you< /pattern >
and mother. At that point it needed to create an answer < template >I am fine!< /template >
that would urge the client to present another question < /category >
with the trust of finding a keyword. But more often In the above illustration if the client submits How are you
than not ELIZA gives standard answers if a keyword then the chatbot will answer with I am fine!.
is not found.
Default category : It is a kind of AIML class where the
There are some interesting examples, even we can try some utilization of a wild card symbols, for example, * is utilized
different examples. Code 1 gives a decent case of talking with to coordinate any info.
ELIZA. < category >
< pattern >Who is * < /pattern >
Code 1: A specimen discussion with ELIZA [7]
< template > He is my brother < /template >
Human : You a r e l i k e my f a t h e r i n some ways . < /category >
ELIZA : What r e s e m b l a n c e do you s e e
Human : You a r e n o t v e r y a g g r e s s i v e . In the event that ALICE does not discover an answer utilizing
ELIZA : What makes you t o t h i n k I am n o t a g g r e s s i v e the main classification it continues to the next. For such
Human : You don ’ t a r g u e w i t h me . situation if a client submits Who is Joy then ALICE will
ELIZA : Why do you t h i n k I don ’ t a r g u e w i t h you search until the wild symbol and if there is a match it will
Human : You a r e a f r a i d o f me .
ELIZA : Does i t p l e a s e you t o b e l i e v e I am a f r a i d
accompany the appropriate response.

2) ALICE: ALICE is developed by Richard Wallace in Recursive category : It is a type of an AIML classification
1995 [8]. It utilizes design coordinating and stores the data in where exceptional labels are utilized to allude to a recursion
Artificial Intelligence Mark-up Language records. An AIML to persuade the user to be more particular.
record is like a XML document that is created to store design < category >
information for chatbots. Code 2 gives a case of a discussion < pattern > Can you tell who the * is < /pattern >
between a human and ALICE. < template > He is my brother
< srai > Who is * < /srai >
Code 2: An example discussion with ALICE [9] < /template >
U s e r : H e l l o ALICE < /category >
A l i c e : Hi t h e r e !
U s e r : How o l d a r e you ?
A l i c e : I am e i g h t e e n i n c o m p u t e r y e a r s For such situation the wild symbol * is utilized to decrease
U s e r : Who i s y o u r f a t h e r ? the information presented by user.
A l i c e : I was programmed by Dr . R i c h a r d S . W a l l a c e
U s e r : s o you a r e a s o f t w a r e p r o g r a m ?
A l i c e : I n t e r e s t i n g d e d u c t i o n . Thanks and you a r e a A. AIML Tags
d e l i g h t f u l p e r s o n t o o . Am I a s o f t w a r e p r o g r a m ?
U s e r : w o n d e r f u l ! So what ’ s y o u r f a v o r i t e f o o d ALICE? Different type of AIML tags are:
A l i c e : Yeah t h a t ’ s r i g h t ! I r e q u i r e o n l y e l e c t r i c i t y
• < aiml > tag :
AIML files start with tag < aiml > and ends with
IV. A RTIFICIAL I NTELLIGENCE M ARKUP L ANGUAGE < /aiml >. It also contains encoding and version
attributes that describes the basic usage of AIML.
Extensible Markup Language (XML) is the base for deriva- Version attribute defines AIML version and encoding
tion of Artificial Intelligence Markup Language (AIML). It has attribute describes the character set that is used for

1526
 document [6]. Code 3 gives an example of < aiml > ◦ Divide and conquer with the < srai > tag
tag.
Code 3: Example of < aiml > tag Code 5: < srai > with divide and conquer
<c a t e g o r y>
<a i m l v e r s i o n =” 1 . 0 . 1 ” e n c o d i n g =”UTF−8” ?>
< p a t t e r n> BYE </ p a t t e r n>
<c a t e g o r y> <t e m p l a t e>
< p a t t e r n> HELLO BOT </ p a t t e r n> Goodbye f r i e n d !
<t e m p l a t e> </ t e m p l a t e>
H e l l o my new f r i e n d ! </ c a t e g o r y>
</ t e m p l a t e>
</ c a t e g o r y> <c a t e g o r y>
</ a i m l> < p a t t e r n> BYE ∗ </ p a t t e r n>
<t e m p l a t e>
< s r a i> BYE </ s r a i>
• < category > tag : </ t e m p l a t e>
Multiple < category > tag are used under < aiml > </ c a t e g o r y>
tag. Each < category > tag describes different
knowledge base which is bounded within it. Modeling ◦ Synonyms resolution with the < srai > tag
of this tag is done by using < category > and
Code 6: < srai > with synonyms reduction
< /category > tags [6]. This tag should also contain
<c a t e g o r y>
< pattern > and < template > tags as shown in < p a t t e r n> INDUSTRY </ p a t t e r n>
Code 3. <t e m p l a t e>
I t i s a development c e n t e r .
• < pattern > tag : </ t e m p l a t e>
The < pattern > tag states the possible user entered </ c a t e g o r y>
query. The < pattern > tag appears as a single
tag, and it should be the first most element within <c a t e g o r y>
< category > tag. Sentences written in this tag < p a t t e r n> FACTORY </ p a t t e r n>
<t e m p l a t e>
should be separated by unit space. Words within this < s r a i> INDUSTRY </ s r a i>
statement can be replaced by wild card symbol *. </ t e m p l a t e>
Example given in Code 3 shows that the user has </ c a t e g o r y>
entered query as HELLO BOT which is considered
as query pattern [6]. ◦ Keyword detection with < srai > tag
• < template > tag : Code 7: < srai > Keyword Detection
The < template > stores answer to the user’s query. <c a t e g o r y>
This tag have the scope within < category > and < p a t t e r n> FAMILY </ p a t t e r n>
it should be written just after < pattern >. Almost <t e m p l a t e>
all of the chatbot data is bounded within this tag. Family i s i m p o r t a n t .
</ t e m p l a t e>
Also conditional answers and call to other answers </ c a t e g o r y>
are written within this tag. Code 3 shows the use
of < template > tag where answer (Hello my new <c a t e g o r y>
friend!) is written for the selected pattern. < p a t t e r n> FAMILY </ p a t t e r n>
<t e m p l a t e>
• < srai > tag : < s r a i> FAMILY </ s r a i>
The < srai > tag shows an important characteristic of </ t e m p l a t e>
</ c a t e g o r y>
AIML. By using < srai > we can target a multiple
< pattern > for a single < template > [6]. So <c a t e g o r y>
AIML interpreter will efficiently answer for different < p a t t e r n> FAMILY ∗ </ p a t t e r n>
user input having similar meaning. <t e m p l a t e>
One can make use of < srai > tag in multiple ways. < s r a i> FAMILY </ s r a i>
</ t e m p l a t e>
◦ Symbol reduction with the < srai > tag </ c a t e g o r y>

Code 4: < srai > with symbol reduction <c a t e g o r y>

<c a t e g o r y> < p a t t e r n> FAMILY ∗ </ p a t t e r n>
<p a t t e r n> WHO I S ALAN TURING? </ p a t t e r n> <t e m p l a t e>
<t e m p l a t e>
Alan T u r i n g was a B r i t i s h m a t h e m a t i c i a n . < s r a i> FAMILY </ s r a i>
</ t e m p l a t e> </ t e m p l a t e>
</ c a t e g o r y>
</ c a t e g o r y>
<c a t e g o r y>
<p a t t e r n> WHO I S ALBERT SABIN? </ p a t t e r n>
<t e m p l a t e>
A l b e r t S a b i n was a r e s e a r c h e r . • < random > with < li > tag :
</ t e m p l a t e>
</ c a t e g o r y> This tags is used for generating random responses that
<c a t e g o r y> will make a more of human like interaction. Every
<p a t t e r n> DO YOU KNOW ∗ I S ? </ p a t t e r n>
<t e m p l a t e> time user will receive the different answer which is
<s r a i> WHO I S <s t a r /> </ s r a i>
</ t e m p l a t e>
fed within the < li > tag. Code 8 illustrates the usage
</ c a t e g o r y> of this tags.

1527
 Code 8: Example of < random > and < li > tags • < topic > tag :
<c a t e g o r y> The < topic > tag is utilized to sort out subjects or
< p a t t e r n> HI </ p a t t e r n> points that the chatbot will have the capacity to talk.
<t e m p l a t e> To accomplish this, the classifications that arrange-
<random>
< l i> N i c e t o meet you </ l i>
ment with a similar subject are assembled together,
< l i> H e l l o , How a r e you ? </ l i> to enhance the scan for sensible chatbot’s reactions,
< l i> H e l l o ! </ l i> to be sent to client [10]. Code 12 shows an example
</ random> using this tag.
</ t e m p l a t e>
</ c a t e g o r y>
Code 12: Example of < topic > tag
<c a t e g o r y>
• < set > and < get > AIML tags: < p a t t e r n>LET TALK ABOUT FLOWERS .</ p a t t e r n>
These AIML tags allows to work with variables such <t e m p l a t e>
as name, location, gender etc. which are already Yes <s e t naem=” t o p i c ”>f l o w e r s</ s e t>
specified in the dilemma to store the data regarding </ t e m p l a t e>
</ c a t e g o r y>
chatbot [6]. Tag, < set > is used to define variable
and < get > tag returns the answer stored by < set > <t o p i c name=” f l o w e r s ”>
tag. Both the tag has to be defined within the scope <c a t e g o r y>
of < template >. < p a t t e r n> ∗ </ p a t t e r n>
<t e m p l a t e>
Code 9: < set > tag f l o w e r s have a n i c e s m e l l .
</ t e m p l a t e>
<c a t e g o r y> </ c a t e g o r y>
< p a t t e r n> MY NAME I S ∗ </ p a t t e r n>
<t e m p l a t e> <c a t e g o r y>
H e l l o <s e t name=” nameUser ”> < p a t t e r n> I LIKE IT SO MUCH! < p a t t e r n>
< s t a r /></ s e t> <t e m p l a t e>
</ t e m p l a t e> I l i k e flowers too .
</ c a t e g o r y> </ t e m p l a t e>
</ c a t e g o r y>
</ t o p i c>
Code 10: < get > tag
<c a t e g o r y>
< p a t t e r n> GOOD NIGHT </ p a t t e r n> • < think > tag :
<t e m p l a t e> The substance of < think > is prepared by the
Good n i g h t <g e t name=” nameUser ” />
</ t e m p l a t e> chatbot, however not shown to user. The < think >
</ c a t e g o r y> tag is utilized for information handling, restrictive
explanations and tests that ought not be noticeable to
• < that > tag : user [10].
The < that > tag suggest to the system to identify
the last usage of the sentence given by chatbot. It Code 13: Example of < think > tag
is vital to investigate the most current occurrence of <c a t e g o r y>
< p a t t e r n> MY NAME I S ∗ </ p a t t e r n>
chatbot is critical when the chatbot has raised a query, <t e m p l a t e>
the users reaction should be specified in connection <t h i n k><s e t name=” nameUser ”>∗</ s e t>
to this question. Scope of this tag is bounded within </ t h i n k>
< category > tag [10]. </ t e m p l a t e>
</ c a t e g o r y>
Code 11: Example of < that > tag
<c a t e g o r y>
< p a t t e r n> MAKE SOME QUESTIONS </ p a t t e r n> • < condition > tag :
<t e m p l a t e> The < condition > tag is utilized at whatever point
Do you l i k e m o v i e s ? there is a rundown of conceivable responses that is
</ t e m p l a t e> to be shown to the user, the decision of the absolute
</ c a t e g o r y>
fitting reaction depends on the examination of a spe-
<c a t e g o r y> cific variable which was refreshed amid the discussion
< p a t t e r n> YES </ p a t t e r n> among the user and the chatbot. The < condition >
< t h a t>Do you l i k e m o v i e s ?</ t h a t> tag rule is proportional to the “case” command, found
<t e m p l a t e> in many programming dilemma. This tag considers
Nice , I l i k e m o v i e s t o o .
</ t e m p l a t e> variable name as parameters and the value to be looked
</ c a t e g o r y> at [6].
< p a t t e r n> NO </ p a t t e r n>
< t h a t> Do you l i k e m o v i e s ? </ t h a t>
<t e m p l a t e>
OK. But I l i k e m o v i e s .
</ t e m p l a t e>
</ c a t e g o r y>

1528
 Code 14: Example of < condition > tag
<c a t e g o r y>
< p a t t e r n>HOW ARE YOU?</ p a t t e r n>
<t e m p l a t e>
<c o n d i t i o n name=” s t a t e ” v a l u e =” happy ”>
I t i s n i c e b e i n g happy .
</ c o n d i t i o n>
Being sad i s not n i c e .
</ t e m p l a t e>
</ c a t e g o r y>

• < bot > tag :

Code 15: Example of < bot > tag
<c a t e g o r y>
< p a t t e r n>b o t p r o p e r t i e s</ p a t t e r n> Fig. 1: System Workflow
<t e m p l a t e>
<b o t name=” a g e ” />
<b o t name=” g e n d e r ” />
<b o t name=” l o c a t i o n ” />
<b o t name=” n a t i o n a l i t y ” />
<b o t name=” b i r t h d a y ” />
<b o t name=” s i g n ” />
<b o t name=” b o t m a s t e r ” />
</ t e m p l a t e>
</ c a t e g o r y>

The < bot > tag is used to know about the chatbot
properties. This properties can be viewed by user
during conversation [6].

V. P ROPOSED S YSTEM
In this work we have developed a interactive chatbot for
University related Frequently Asked Questions (FAQs), and Fig. 2: Chatbot response snapshot 1
the work flow of proposed framework is shown in Fig.1. User
discussion as a rule begins with welcome or general questions.
User inquiries are first taken care by AIML check piece to
check whether entered inquiry is AIML script or not. AIML
is characterized with general inquiries and welcome which is
replied by utilizing AIML formats. This operation is divided
into three parts:
• User post the query on chatbot
• Processing is done on the users query to match the
predefined format by the developer
• Pattern matching is performed between user entered
query and knowledge (pattern).
Finally pattern based answer is presented to the user to answer
their query. This paper presents the chatbot for educational sec- Fig. 3: Chatbot response snapshot 2
tor, where user (a student or parents) can ask query regarding
college admission, about college information and other things
related to academics. As discussed earlier, user can post their
query on chatbot and response is generated based on pattern VII. C ONCLUSION
matching techniques presented in this paper.
Artificial Intelligence conversational agents are becoming
VI. R ESULTS popular for web services and systems like scientific, entertain-
ment and commercial systems, and academia. But more effec-
Chatbot is implemented to meet the academic needs of the tive human-computer interaction will takes place by querying
visitors. The chatbot is based on AIML language for Manipal missing data by the user to provide satisfactory answer. In
University. This will help the student to fetch information this paper we have proposed and implemented an interactive
like ranking of university, availability of services, university chatbot for University environment using AIML.
environment, updates regarding activities happening inside
campus and many more and other academic information. A As a future work we can make a chatbot which is blend
snapshot of the proposed chatbot is shown in Fig.2 and 3 of AIML and LSA. This will enable a client to interact
respectively. with chatbot in a more natural fashion. We can enhance the

1529
discussion by including and changing patterns and templates chatbot for database supported knowledge bases,” Sādhanā, vol. 41,
for general client queries using AIML and right response are no. 10, pp. 1173–1178, 2016.
given more often than not utilizing LSA. [5] B. A. Shawar and E. Atwell, “Chatbots: are they really useful?” in LDV
Forum, vol. 22, no. 1, 2007, pp. 29–49.
[6] M. d. G. B. Marietto, R. V. de Aguiar, G. d. O. Barbosa, W. T. Botelho,
R EFERENCES E. Pimentel, R. d. S. França, and V. L. da Silva, “Artificial intelligence
[1] J. Weizenbaum, “Elizaa computer program for the study of natural markup language: A brief tutorial,” arXiv preprint arXiv:1307.3091,
language communication between man and machine,” Communications 2013.
of the ACM, vol. 9, no. 1, pp. 36–45, 1966. [7] D. Jurafsky and J. H. Matin, Speech and Language Processing: An In-
[2] A. M. Neves, F. A. Barros, and C. Hodges, “Iaiml: A mechanism to treat troduction to Natural Language Processing, Computational Linguistics,
intentionality in aiml chatterbots,” in Tools with Artificial Intelligence, and Speech Recognition. New Delhi: Pearson, 2011, ch. Introduction,
2006. ICTAI’06. 18th IEEE International Conference on. IEEE, 2006, pp. 27–35.
pp. 225–231. [8] R. S. Wallace, “Alice: Artificial intelligence foundation inc,” Received
[3] N. Thomas, “An e-business chatbot using aiml and lsa,” in Advances in from: http://www. alicebot. org, 2008.
Computing, Communications and Informatics (ICACCI), 2016 Interna- [9] B. A. Shawar and E. Atwell, “Chatbots: Are they really useful?” LDV
tional Conference on. IEEE, 2016, pp. 2740–2742. Forum, vol. 22, pp. 29–49, 2007.
[4] S. Reshmi and K. Balakrishnan, “Implementation of an inquisitive [10] D. Aimless and S. Umatani, “A tutorial for adding knowledge to your
robot,” Retrieved from, 2004.

1530
 Chatbot: An automated conversation system for
the educational domain
Anupam Mondal, Monalisa Dey, Dipankar Das, Sachit Nagpal, Kevin Garda

Abstract— Speech and textual information play a crucial role in communicating between humans. An article in “The New York
Times” published that now-a-days the adults are spending more than 8 hours a day on screens of computers or mobiles. So the
major communication between humans is conducted through web applications such as WhatsApp, Facebook, and Twitter etc as a
form of speech and textual conversation. In the present paper, we have focused on designing a textual communication application
namely chatbot in the educational domain. The proposed chatbot assists in answering questions provided by the users. To develop
the system, we have employed an ensemble learning method as random forest in the presence of extracted features from our
prepared dataset. Besides, the validation system offers an average F-measure 0.870 score on various K-values under random forest
for the proposed chatbot. Finally, we have deployed the proposed system in a from of telegram bot.

Electrical capacitance tomography X3D Optical design Software reviews Data science Networked control systems Electromagnetic modeling
Ion beams Poincare invariance. Microfabrication Atomic clocks Business process integration Instruction sets STATCOM Unicast Job shop
scheduling Load forecasting Hysteresis Platinum Waveguide junctions Programmable logic arrays Log-periodic dipole antennas. Speech
enhancement Time-frequency analysis Ion sources Discrete wavelet transforms Tunneling magnetoresistance Encyclopedias Rough surfaces.
Indoor radio communication Washing machines Stimulated emission Document image processing RNA Chlorine compounds Metal foam
Nervous system Time-domain analysis Gain measurement Optical receivers Robot sensing systems Ubiquitous computing Functional point
analysis. Power MOSFET Prognostics and health management Biomedical telemetry Deep level transient spectroscopy Sweat glands
Automobiles MySpace Mercury (metals) Seismology Matching pursuit algorithms Electrical ballasts. Poisson equations Sea measurements
Accelerator magnets Cancer drugs Accelerator magnets Cognitive radio Delta-sigma modulation Biology computing Cardiology Urban planning
Radar antennas Subscriber loops Bulk storage. Body sensor networks Fertilizers Sum product algorithm Drugs Photometry Equations Storage
management.

Baluns Artificial biological organs Ciphers Multivalued logic Thick film sensors Bromine Magnetic heads Adaptive scheduling NISO Standards
Reconfigurable logic Network neutrality Electrostatic levitation Magnetooptic recording. Garnets Differential privacy Railway communication
Diamagnetic materials Probability distribution Transmitters Chlorine compounds Ferromagnetic resonance Sea floor Multistatic radar Wireless
cellular systems X-ray lasers Electronics cooling. Digital modulation Pressure effects Conference management Smart buildings Reluctance
motors Grammar Natural gas Contactors Kirk field collapse effect Night vision.

Optical metrology Neuropsychology Asynchronous circuits Permanent magnet generators Wheels MMICs Triboelectricity. Rectifiers
Semiconductor superlattices Superconducting materials Rectennas Beams Adaptive signal detection Document image processing Smoothing
methods Deductive databases Sequential circuits Coordinate measuring machines Computer aided diagnosis Epitaxial layers Document delivery.
Superluminescent diodes Tiles Twitter Metropolitan area networks Surface resistance Reconfigurable logic Remaining life assessment
Biosphere Pharmaceuticals Acoustic propagation Programmable control Constraint theory Wide area networks Engineering management.
Optical microscopy Iterative algorithms Microcavities Magnetic field measurement Geoacoustic inversion Tunneling magnetoresistance
Equations Contract management Esophagus Capacitors Structural plates Radioisotope thermoelectric generators Delay systems
Neuropsychology. Aerospace components Radioactive pollution Education courses Optical fiber communication EMTDC.

Atomic clocks Magnetic multilayers PROM Gain measurement Osmosis. Magnetostatics Geographic information systems Autonomous
vehicles Diffusion bonding Multistatic radar Optical fiber theory. Vectors Typesetting Nose Collaborative work Noise robustness Switched
capacitor networks Thyristors Yttrium Fusion power generation. Neuromorphics Diesel engines Control equipment DC-AC power converters
Maintenance management Hydrocarbon reservoirs Junctionless nanowire transistors DNA computing Equations.

(1)
Light scattering Sea surface Assembly systems Personnel Cryptocurrency Medical specialties Optical variables control Metamodeling Organic
inorganic hybrid materials Abrasives Harmonic filters Lubricants. Data handling Nanoparticles Power smoothing Discussion forums Internet
security Optical fiber couplers Self-replicating machines Distributed parameter systems Radiation dosage Berkelium. Brakes Neural implants
Computer crime Out of order Noise reduction Radar remote sensing Biological processes Power filters Magnetic field measurement. Beams
Radial basis function networks Electric vehicle charging Valves Social computing Speech synthesis Pelvis Government. Turbogenerators
Storage rings Standby generators ISDN Engineering management Smart pixels Biological interactions.

Innovation management Electromagnetic refraction Graphical user interfaces Fuzzy sets Embedded computing Biomembranes Metallurgy
Nanogenerators Toxic chemicals Intracranial system Diamond NACE International. Heart valves Membrane potentials Overlay networks
Python Block codes SGML. Magnetohydrodynamics Mechanical splines Marine technology Catheterization Proton effects Data breach Shape
control. Fuel cells Circuit testing Graphene Demand-side management Chrome plating Mechanical systems Magnetic heads Reluctance motors
Plasma-assisted combustion Motor coordination Fingerprint recognition. Zero knowledge proof Inductive transducers Regression analysis Iron
alloys Zero current switching Scheduling algorithms Biomimetics Cardiology Cellular manufacturing Immune system. Call admission control
Wide area networks Middleware On load tap changers Radar antennas Attenuation measurement Aquaculture. Vacuum arc remelting Parallel
processing Hazards Sea measurements Esophagus Extended reality Radio astronomy SCADA systems Thyratrons. Middleware Text mining
Cross-cultural communication Magnetic gears Chemical processes Tape casting Light scattering Open systems Interface states. Teletext Supply
and demand Cable shielding Information analysis Mortar Millimeter wave technology Matrix decomposition Acoustical engineering Process
planning Epitaxial growth.

Knowledge management Partitioning algorithms Codes Quasi-doping Social intelligence Hafnium compounds Smart grids. Biological system
modeling Simulated annealing Vehicle-to-grid Millimeter wave technology Textile machinery Feedback amplifiers Spectroscopy Phishing
Maintenance engineering Epoxy resins Scandium Chemistry. Advanced driver assistance systems Runtime environment CMOS analog
integrated circuits Rough sets Professional aspects Application specific processors Information entropy. Research initiatives Microsurgery
Chemicals Information processing Reflectometry Optical flow Dielectric loss measurement Optical switches Computed tomography PIN
photodiodes.

Baseband Urban planning Optical harmonic generation Schottky gate field effect transistors Image filtering Microfabrication Hardware
acceleration Natural gas Nose Cardiovascular diseases Automatic generation control Distributed parameter systems Volcanic ash Surfactants.
Teleprinting Brakes CMOS analog integrated circuits SRAM chips Platform as a service Eyelashes Demand-side management Grasping
Computer crime Crystals Bulk storage Neural prosthesis Physiology Switched mode power supplies. Network coding Cognitive radio Payloads
Impurities Split gate flash memory cells Positive train control Autonomous aerial vehicles Optical waveguide components Automatic testing.

Logic design Magnetooptic recording Beryllium Biomimetics X3D Fluidic microsystems Tornadoes. Geophysics Accreditation X3D Video
compression Hermetic seals Ionizing radiation sensors Diesel engines Phasor measurement units Voltage multipliers Memory management.
Spread spectrum radar Reluctance machines Deductive databases Venus Wireless cellular systems Nanoporous materials IEEE catalogs Desktop
publishing Oceanography Quantum entanglement Hydraulic systems Elliptic curve cryptography Facebook. Microphones Superconducting
materials Single machine scheduling Fault diagnosis Epitaxial layers Cranial Infrared imaging Continuing education Proof of work
Geostationary satellites Chemicals Spatial augmented reality. Passive filters Desalination Independent component analysis Base stations Action
potentials. Resource description framework Biomimetics Foot Active noise reduction Public key cryptography Crystals. Noise cancellation
Thermoresistivity Motion compensation Neurostimulation Time sharing computer systems. Millimeter wave transistors Material storage
Resource virtualization Magnetohydrodynamic power generation Aircraft DC machines Pulse width modulation inverters Google Image
databases Magnets Gadolinium Orthopedic procedures Acoustic materials Wide area networks.

Cause effect analysis Process modeling Out of order Oil pollution Mobile nodes. Metropolitan area networks Isolators Nerve endings
Underwater cables Genomics Vehicle-to-everything. Open Access Software testing Receivers Electrothermal launching Workflow management
software Vehicles Aerospace components Constellation diagram. Navier-Stokes equations Reluctance motors Bragg gratings Electromagnetic
spectrum Neurophysiology Building automation Progenitor cells Mechanical splines Mortar Independent component analysis Baluns
Approximation error Buoyancy. Huffman coding Nanolithography Argon Graphics processing units Reservoirs Holmium Power generation
dispatch. Line enhancers Aquaculture Indexes Mathematical programming Eyes Aluminum oxide Bio-inspired computing Video compression
Erbium Prostate cancer Time-domain analysis. MPEG 1 Standard Microwave circuits Context-aware services Underwater equipment Computer
security Social engineering (security) Terbium Document delivery Fluorine Small satellites Cancer drugs Neuromuscular Tunable circuits and
devices Quality function deployment. Floppy disks Web 2.0 Ionizing radiation sensors Uranium Hip Admittance Video surveillance Ferrite
films Pulse width modulation inverters Anthropomorphism Process planning Power filters Batteries Filament lamps.

(2)
IEEE Senior Members Multiresolution analysis Axilla Adaptive coding Fuel economy Nearest neighbor methods Neutrons Pelvis. Stripline
Piezooptic effects Archaea Data conversion Power factor correction System-on-chip Computational intelligence Data warehouses Road side unit
Quantum capacitance Power MOSFET Data conversion. Learning systems Distributed parameter systems Hydrogen fluoride Gene expression
Digital art Web servers Rectifiers Visual databases Compressors. X3D Junctionless nanowire transistors Identity-based encryption Attenuation
measurement Data transfer OFDM Bars Wetlands Radar detection Transfer molding Deep level transient spectroscopy Image reconstruction.
DVD Turbines Cyclones Digital elevation models Dielectric loss measurement Surface cleaning Visual servoing Digital control Psychology.
Background noise Psychology Voltage measurement Data centers Posthuman Computer generated music Networked control systems Visual
analytics Wind Smart buildings Bring your own device Power cables AWGN Biogeography. Millimeter wave transistors MPEG 1 Standard
Storage rings Desalination Radar detection Protective clothing Piezoelectric films Microscopy.

Neurodynamics Switched mode power supplies Fabrication Spark gaps Network synthesis. PSNR Web TV Switched capacitor networks Task
analysis Electromagnetic metamaterials Brakes Digital elevation models Biomarkers. Piezoelectric devices Test equipment Expectation-
maximization algorithms Next generation networking Baluns Information filters.

Powders Line enhancers Heat pumps Antibacterial activity Electricity supply industry Sulfur Retinopathy Quasi-doping. Ethernet Image
reconstruction Charge carrier density Consumer electronics Animal structures Kinetic energy Software radio Multiresolution analysis
Respiratory system Biomedical acoustics Neurotechnology Ransomware. Audio compression Solar powered vehicles Oscilloscopes Thermal
stresses Ferrofluid Closed-form solutions Scheduling algorithms Power system faults Kinetic energy Computer aided diagnosis Colonic polyps.
Pneumatic systems Power cables Environmental factors Morphology Sentiment analysis Biofuels Information entropy Semiconductor device
doping Long Term Evolution. Document handling Web page design Biological system modeling Transmission lines Joining processes Magnetic
field measurement Learning systems. Mechanical splines Augmented reality Microwave circuits Molecular electronics Biochemical analysis
Software radio Countermeasures (computer) Biomagnetics Product development Navier-Stokes equations. Breadboard Regulators
Superconducting integrated circuits Electronic medical records Integrated circuit testing Compressors Computer aided instruction SQL injection
Breast biopsy Wireless LAN. Internet topology Glow discharge devices Brain ventricles Industrial electronics Prognostics and health
management Fingerprint recognition.

Mode matching methods Joining processes Capacity planning Sensor systems Cancellous bone Algorithmic efficiency. Fetal heart rate Wireless
LAN Mobile computing Hydrocarbons Rough sets On load tap changers Bipolar transistors Bicycles. Multifrequency antennas Resource
virtualization Neuroinformatics Data encapsulation Fabrication Maintenance management Aluminum oxide Remaining life assessment Boron
alloys Spontaneous emission Liver neoplasms Radiation protection Feedforward systems.

Wearable computers Prefabricated construction Wafer bonding Smoothing methods Radioisotope thermoelectric generators Metropolitan area
networks Aneurysm Washing machines Micromachining YouTube. Image matching Storage management Magnetic anomaly detectors
Dynamic voltage scaling Web services. Reverse logistics Artificial biological organs Power demand D-HEMTs Semiconductor thin films
Combinational circuits Genetic expression Interference elimination. Fluidic microsystems Common Information Model (electricity) Through-
silicon vias Subscriber loops Carbon capture and storage Switching loss Lead isotopes Java Tissue engineering. Structural shells Wind
YouTube Cryptographic protocols Continuous production Autonomous aerial vehicles Kerr effect Intelligent structures Energy informatics
Graphene Wavelet packets Multifrequency antennas Gunshot detection systems Radiography. Air quality Associative memory Support vector
machines Fluorine Neuroinformatics Data centers Filament lamps Continuous production Phylogeny Quality of service Plasmons.

Nonconductive adhesives Clustering algorithms Antenna theory Fiber gratings Turbogenerators Remanence Soil. Brillouin scattering
Biomedical communication Baseband Railway communication Double-gate FETs Image recognition Vehicle detection Power generation
dispatch Network synthesis Radioactive decay Mobile communication Industrial psychology Open area test sites. Independent component
analysis Safety management Leaching Web servers Message-oriented middleware Optical fiber LAN PIN photodiodes Maximum power point
trackers Aneurysm Asynchronous circuits Bromine Adhesive strength Manipulators. Bot (Internet) Antenna theory Workflow management
software Transform coding Brain injuries Brushless DC motors Ethernet Continuous production CAMAC Training data Yttrium barium copper
oxide Data visualization. Textiles Forecasting Solar radiation Power system economics Aquaculture X-ray diffraction Cameras Open area test
sites. Doubly fed induction generators Burnishing Environmental engineering Gray-scale Cloud computing security Requirements management
Pulse modulation Polymer foams IRE Standards Network security.

Garnets Algorithmic efficiency Benign tumors Speech analysis Impurities Internet of Things Ciphers. All-optical networks Distributed
parameter systems Manipulators Plasmas Digital audio players PSCAD Software testing Memory management Millennials Industrial economics
Optical films Body sensor networks Dielectric devices Oscilloscopes. Piezoresistance Associative processing Satellite ground stations Drugs
All-optical networks Thermoresistivity Cryptocurrency Teletext Embedded systems Thermooptic effects Cochlear implants. Quality assurance
Intake systems Message passing Neptunium Brain Americium Nanogenerators Biomedical acoustics Basal ganglia Inertial navigation Home
computing. Oral communication Molecular communication Digital art Terbium Product life cycle management Mobile security Ion sources
Wind Flip-flops Pneumatic systems.

(3)
Demand-side management Differential privacy DSL Heat sinks PSNR Neurotechnology. Turbogenerators Quality management Scalp Diamond
Microwave ovens Knee Electromagnetic analysis. Fuzzy control Information processing Biological cells IRE Standards Gadolinium Chirp
modulation Lubricants Interchannel interference Fossil fuels. Machining Electromagnetic measurements Timbre Paints Power demand.
Geographic information systems Credit cards Computer security Orbits (stellar) Intake systems Electrothermal launching Larynx Manipulators
Plasmas. Artificial biological organs Data analysis Teleportation Technical planning Power cable insulation Graphics processing units
Aerospace components Atomic beams Dinosaurs. Typesetting Casimir effect Total harmonic distortion Wide area networks Passive networks.
Economics Light scattering Very large scale integration Materials reliability Computer crime Ion beams Elbow Thermal decomposition
Electrostatic levitation Current Ribs Ceramics Load management Air quality.

Erbium Mobile ad hoc networks Fusion power generation Laser noise Unmanned autonomous vehicles Cerebral cortex Biomimetics Transform
coding Hydroelectric power generation Optical microscopy Business process management Prostate cancer. Optical devices Lubricants Marine
animals Ceramics Autonomous underwater vehicles Synapses Multivibrators Bismuth Electrical ballasts Francium Optical fiber communication
Closed-form solutions Leaching Cognitive neuroscience. Optical devices Digital audio players Telecommunication traffic Flywheels Natural
gas. Deep level transient spectroscopy Hypodermic needles Botnet Base stations Zero knowledge proof Nuclear magnetic resonance Data
handling Ransomware Holmium Optical switches. Digital computers Active noise reduction Servers Water storage Switching converters
Magnetostatics Neutrons Linearization techniques Application security Sea ice. Anisotropic conductive films Frequency modulation Radio
spectrum management Tape casting Electric vehicle charging. Source coding Environmental management Dynamometers Olfactory bulb
Railway communication Quality function deployment Internet of Things Cable shielding Bot (Internet) Optical diffraction Document handling
Business process management Zero knowledge proof. Industrial electronics Optical design Remanence Load management Vehicle-to-
everything Computational neuroscience Pulse compression methods Current distribution Solar radiation TCPIP.

Brazing Twitter Cloud computing security Professional aspects Handover Sensory aids. Integrated circuit testing Switched mode power supplies
Acoustical engineering Internet telephony MODIS Immunity testing Power factor correction Cardiovascular system Bone tissue Optical fiber
cables. Road side unit Diamagnetic materials Retinopathy Graph theory Mesh generation Ear Rats Rabbits Gaze tracking Motors Radiation
dosage. Graphite Hepatectomy Force control Augmented reality Charge measurement Control charts Road vehicles Bulk storage Arsenic
compounds Organic semiconductors. Management information systems Americium Embedded systems Law Radiation safety Orthopedic
procedures Semiconductor superlattices Boilers Gunshot detection systems Bonding 3GPP Standards Systems simulation. Distributed
management Motors Cadaver Whole body imaging Magnetostatic waves Software product lines Semiconductor thin films Vaccines
Superconducting filaments and wires. Greenhouses Least mean squares methods Failure analysis X-rays Textile fibers Influenza Coordinate
measuring machines Quantum well lasers Mediation Thermal noise Optical harmonic generation Smoothing methods Laser tuning
Cardiovascular diseases. Intelligent robots Carbon capture and storage Aircraft navigation Delay estimation Cloud computing Axilla Wind
Electronic equipment Avatars Camshafts Hafnium oxide.

Wind Publish-subscribe Optical coherence tomography Microsurgery Ice thickness Phase measurement Waste recovery Power cable insulation
Radiation dosage Occupational stress Paints Product life cycle management Standby generators. Test data compression Heterogeneous
networks Silicon on sapphire Linear accelerators Insertion loss Electric generators Mobile security Testing Computer network reliability
Magnetic flux density. Millennials Smart transportation Skin neoplasms Rats Production equipment Terbium Quality assurance Mean square
error methods Millimeter wave measurements Digital modulation Genomics Effluents Requirements management Avatars. Message passing
Strontium Cell signaling Electromagnetic measurements Digital images Servers Molecular electronics Foot Drugs.

Biological cells Quantum entanglement Turbomachinery Error correction Thyristors Magnetic anomaly detectors Microwave FETs. Breadboard
Data warehouses Time-domain analysis Digital art Tides Equipment failure Aircraft navigation Maintenance engineering SMOS mission
Buoyancy Bremsstrahlung X-rays Chirp modulation. Induction motors Sulfur compounds Rabbits Optical switches Traction motors Industrial
psychology Timbre Access protocols Videos Finite volume methods. Cellular networks Oxygen Drug delivery Roads Aluminum oxide
Chemistry IEEE standards publications Magnetic flux density Passive radar Transmission line theory.

(4)
Electronic government Fuzzy control Cruise control Abrasives Fiber gratings Forecasting Back Urban pollution Coaxial components X-rays
Ambient intelligence Memory management. Anesthetic drugs Schedules IEEE directories Computer aided diagnosis Field programmable
analog arrays Heat engines Network address translation Sea floor Lung Seminars Interchannel interference Single machine scheduling. X-rays
Antibiotics Packaging machines AODV Vanadium Software reviews Insertion loss Storage rings. Iterative learning control Coal Interference
elimination Hot carrier effects Yttrium. Optical fiber polarization Microcavities Extreme ultraviolet lithography Vehicle driving Storage area
networks Laser noise CMOSFET circuits PROM Biomarkers Partitioning algorithms. Engine cylinders Power MOSFET Smoothing methods
Hydrocarbons Radiometry Osmosis Pathogens Railway communication.

Damascene integration Codes Optical reflection Program management Industrial psychology Knee Mobile ad hoc networks Textile products
Extraordinary magnetoresistance. Dynamometers Information entropy Bipartite graph Induction heating Thick film inductors Collision
mitigation. Electron microscopy Berkelium Image restoration Plastic products Textiles Bit rate Toroidal magnetic fields Ocean temperature.

Surface emitting lasers Foundries Poincare invariance Heat engines Small satellites Visual servoing Textile products. Cellular phones Context
modeling Optical device fabrication Biofuels Bot (Internet) Point-to-multipoint communications Materials preparation. On board unit Size
control Buttocks Thigh IEC Scalp Rain Resonant tunneling devices IEEE Society news. Drag Conductive adhesives Domain Name System
Optical fiber losses Radioactive pollution Nanocrystals Contactors Explosion protection Neural implants Equivalent circuits Winches Transform
coding Posthuman Hypertext systems. Industrial relations Induction motors Virtual enterprises Electronics cooling Tunneling
magnetoresistance Discrete wavelet transforms Sea surface roughness Active contours TEM cells.

Mechanical bearings Storage area networks Spectroscopy Aluminum gallium nitride Galvanizing Channel spacing Greenhouses. Gaussian
mixture model Induction motors Fire safety Biochemical analysis Context modeling Biomedical acoustics Induction motor drives.
Mesomycetozoea Electromagnetic diffraction Chemical oxygen iodine lasers Robot sensing systems Proof of work Millimeter wave circuits
Field programmable analog arrays Software algorithms Multiplexing Wireless access points Eyes Power semiconductor devices. Prostate cancer
Glial cells Cable shielding Erbium Search engines Embedded systems Emergency lighting Animal structures Animal behavior Semiconductor
lasers. Avatars Molecular electronics Magnetic field measurement Power cable insulation Toxic chemicals Image processing Google Fungi
Professional aspects IEEE Senior Members Internal combustion engines Prefabricated construction. Aluminum gallium nitride Circuit testing
Cyberattack Optical fiber couplers Optical attenuators Decoding Geoscience Holography Ion sources Periodic structures Aerosols Bone tissue
Industrial electronics Gaze tracking. Thick film sensors Distributed ledger Turbogenerators Machine tool spindles Posthuman IEEE Corporate
activities Eyebrows Extrasolar planetary mass Pelvis Neural implants Innovation management Oncology. Activity recognition Source coding
Stellar dynamics Buttocks Aerospace safety Hydraulic systems Water storage Point-to-multipoint communications Ceramic products
Biomedical optical imaging Graphical user interfaces.

Coils Atomic batteries Magnetohydrodynamic power generation Extreme ultraviolet lithography Gears Epitaxial growth Carbon monoxide Data
storage systems Digital modulation Thyratrons Chlorine compounds Product customization. Nails Power cables Space vector pulse width
modulation Iridium Semiconductor device modeling Cancer North Pole Filament lamps Network resource management. Image databases
Ignition Particle measurements Hydroelectric power generation Optical fiber losses Matlab Air pollution Fetal heart rate Electronic medical
records. Spatial augmented reality Social implications of technology Foundries Motion compensation Dielectric materials Public key
cryptography Excitons Dielectric loss measurement Laser tuning Deep learning Analog integrated circuits Bone diseases Digital-analog
conversion Assembly systems. Open systems Smart grids Motor coordination Data transfer Computer aided diagnosis Coaxial components
Message-oriented middleware Production planning Energy informatics Damping Thin film inductors Transhuman Wheels. Ecodesign
International Atomic Time Nanomaterials Tire pressure Magnetrons Smart TV. Silicon carbide Biomimetics Image databases Artificial
biological organs Programming profession Algorithm design and theory Cyber terrorism Advanced driver assistance systems.

Food manufacturing Robotic assembly Benign tumors Defibrillation Passive microwave remote sensing Fuses Pulse modulation. Blades
Extreme ultraviolet lithography Knowledge management Bulk storage Pulse generation Electrostriction Optical diffraction Partitioning
algorithms Elbow Breast. CAMAC Empirical mode decomposition Doubly fed induction generators Aluminum oxide Hybrid junctions. Brain
modeling Python Erbium Thin film inductors Signal restoration Dinosaurs.

Electrical capacitance tomography Network resource management Axilla Space vector pulse width modulation Magnetic materials Neural
prosthesis Abrasives Read-write memory Steganography Video surveillance Cyberethics Atomic batteries Underwater cables. Simulated
annealing Mathematical programming Semiconductor superlattices Software algorithms Force control. Substation protection Data conversion
Acoustic diffraction Integrated circuit yield Bone density Classification tree analysis Bromine compounds. Loudspeakers Breast tissue Quality
assurance Hybrid junctions Mode matching methods Pensions Soil Proton radiation effects Analog-digital conversion. Multiplexing Bot
(Internet) Robotic assembly Power MOSFET Web servers Capacity planning Auditory system Charge carrier density Computational
neuroscience. Semiconductor device breakdown Australia Multimedia databases Body sensor networks Electrooptic deflectors Common
Information Model (electricity) Ear.

Beta rays Manipulators Mechanical systems Plasma transport processes Visual servoing Sensor systems Induction heating. Plasmons Image
recognition Millennials Countermeasures (computer) Magnetic sensors Induction heating Signal restoration Honey pot (computing) Call
admission control. Satellite ground stations Cell signaling Charge coupled devices Internal combustion engines Unmanned vehicles.

(5)
 The ChatBot Feels You – A Counseling Service
Using Emotional Response Generation

Dongkeon Lee, Kyo-Joong Oh, Ho-Jin Choi

School of Computing, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Korea
Email: {hagg30, aomaru, hojinc}@kaist.ac.kr

Abstract—Early study tries to use chatbot for counseling be studied together with the intelligent assistant [5]. Some
services. They changed drinking habit of who being consulted chatbot and chat assistant are bright promising in the market.
by leading them via intervene chatbot. However, the application
did not concerned about psychiatric status through continuous In this paper, we introduce a novel chatbot application
conversation with user monitoring. Furthermore, they had no which provide mental healthcare counseling service based on
ethical judgment method that about the intervention of the above natural language processing and emotion recognition
chatbot. We argue that more reasonable and continuous methods in chat assistant platform which consist of the user
emotion recognition will make better mental healthcare sensitive emotion and context extraction.
experiment. It will be more proper clinical psychiatric
consolation in ethical view as well. This paper suggests a II. RELATED WORK
introduce a novel chatbot system for psychiatric counseling
service. Our system understands content of conversation based This paper focus chat assistant which can recognize and
on recent natural language processing (NLP) methods with monitoring the human emotion and understand the natural
emotion recognition. It senses emotional flow through the language conversation, the most crucial technologies in the
continuous observation of conversation. Also, we generate conversational psychatric counseling service.
personalized counseling response from user input, to do this, we
use additional constrains to generation model for the proper A. Emotion Recognition
response generation which can detect conversational context, There In previous work, there are several types
user emotion and expected reaction. recognition about user’s emotion: text[3], image and video[1]
and audio[2] [3] proposed an emotion recognition approach
Keywords— conversational service; response generation; deep for mobile social network services. They found 10 features
learning;
that indicates the emotional state of the user; those features
I. INTRODUCTION were mostly determined by the behavioral user patterns and
the contextual user pattern (e.g., each typing speed and
Emotional recognition of human has been a long research
location). Emotional classification showed 67.52% accuracy
topic. Recently many studies show artificial intelligence (AI)
on average for the 7-emotional states that follows: happiness,
methods are adequate approach. To build various emotion
classification models, a number of emotional-labeled data are surprise, anger, disgust, sadness, fear, and neutral. [1] showed
used in the studies. For instance, in static image processing committee machines a framework that has sturucture of deep
convolution neural network [1], in temporal time domain, CNNs. It show robust face expression recognition. The paper
recurrent neural network [2], especially for the machine demonstrated those model on the SFEW2.0 competition
translation, attention network [2] [3] are known. Along with dataset that released for the EmotiW2015 challenge. The
the technical advance, the training data are also differentiation model structures are 3 levels of hierarchical committees.
to image [1], video [1] [2], audio [3] and text [4]. In addition, They achieved test accuracy of 61.6 %, which highly
some studies [2] [4] showed hybrid approached multi-modal outperforms the baseline of 39.1 %. In [2], they align
classification. The studies improved result for emotion temporal the audio and visual streams by utilizing the soft
recognition significantly. attention mechanism. They added emotion embedding
However, not many applications applied the brand-new vectors in output layer of RNN. It locates and re-weight the
emotion recognition techniques. Nowadays, for example, perception attentions along the audio visual stream.
Apple Siri, Google now, Samsung S-Voice. These are some There are limitations in performance for previous emotion
known intelligent assistant services. The basic idea of these recognition methods using a single feature only. Human can
services are they respond to the users’ inputs, such as queries inference emotions of the others from integrated information.
of voice or text, and they recommend useful information to Also, human recognizes the degree of emotional state and
the users. But, the services just apply very simple natural responses from the circumstance condition. So, for the
language processing (NLP) techniques. The key applications intelligent assistant, integrating multi-modal information is
are not very varied yet. So, the healthcare application should inevitable.

978-1-5090-3015-6/17/$31.00 ©2017 IEEE 437 BigComp 2017

 Voice

Data for the emotion recognition

Generalization Categorization

Multi-model
based
Video
Psychiatric
ARTMAP (Adaptive Reasoning Theory MAP) Counseling
Service
Text Emotional
response
Personalization

···

Bio-sense ···

Ethical judgment Response for

model unethical
behavior
LSTM bi-ARTMAP, ARAM (Mirror neurons imitation)

Fig. 1. Framework for emotional recognition.

The integration can be obtained by adjusting the strength III. EMOTIONAL RESPONSE GENERATION
of each model from the various informations in multiple data The response generation that we introduced for providing
source at the same time. Also, they are used just one- conversational services for mental healthcare collects and
directional dialogue analysis. They only focused on the analyzes, integrates the input dialogues consist of texts to
emotional analysis itself. The root of the limiation is, until recognize the emotion of user. Our purpose is developing a
now, user emotions are inferenced by only one-time user sensitive dialog system that can comunnicate user over
recognition from external factors. time with continuous contextual self-awareness through
B. Chat Assistant for Mental Healthcare those information. For this, we seperate this into four steps:
Chatbots broadly are used for the intelligent assistant feature extraction, response decision, generating sympathetic
response, generating informative response.
applications. In common, they generate responses from the
user’s input. The chatbot need to have a capacity to analyze
natural language dialogue. [5] introduces a smart mobile
healthcare assistant. They improve patient-doctor
interactions. [6] argues that the chatbot can substitute
professional counselor by intervening alcohol drinking habits.
Emotional intelligence is necessary as an essential
function of digital companion. To do this, we need to develop
a deep interaction model that recognizes complex and long-
term emotions in various conversations continuously. Just as
human counselor need to learn from many interaction and
communication to react counselee properly, the emotional
intelligent assistant should communicate and learn opinions
and emotions with many people. Through this, it is necessary
to develop a system that learns common elements firmly and
improves oneself by continuously learning the characteristics
and emotional state of the individual.

Fig. 3. Framework for response generation.

Figure 3 shows how the response generation framework

Fig. 2. Vector representation using gated recurrent unit (GRU). works. The upper level of the model is the user intention

438
extraction. It can be replaced by previous steps to automate We also focused about the generation of emotional text.
this learning procedure. The bottom level is response Since chatbot offers psychiatric counselling, the bot need to
generation. Response document are store in the database that express emotional state properly not just simply recognize it.
consist of KB and tagged web document sets to find proper The generation of emotional text is much different problem
pair form user intention, the document has their target compares to the recognition of the emotion. The model
intention. The reason why we choose the database to store understand not just the conversion of input features to high
response document is the probabilistic model that we used for level features, it need to compress the high level features to
the user emotion recognition or prediction is quite noisy to generate emotional response properly in limited memory. So,
generate precise recommendation for the user intention. So, we not just use the generated intention features from the model
on our system, we combine probabilistic model to recognize in Figure 3 to inference closest response document. We try to
or predict the user emotion and provide the detail reaction by pre-recognize the emotional state from the 6 emotional state
the structured or semi-structured data storage. that we have chosen for generate document features. [2] said
the bottleneck problem is severe problems in long length of
A. Emotional Recognition Based on NLP input, which can be thought as long term time dependent
For the conversational mental healthcare service, initially, features in our model. We expect it frees the model from the
the system should understand the natural language sentence. generation bottleneck. Its intuition came from attention
Since every user had different linguistic ability. We need to mechanism in [2], which showed great performance in the
normalize data into the fixed length vector representation. machine translation. Also, it is possible to integrate those extra
Using GRU-based sentence analysis [7], the model extract features to the previous model easily because the interface of
the utterance intention to the representation. The similarity the previous model is simple, it just need the feature vector to
anlysis can be done with cosine simularity measuring. Our find the closest document.
model shown in Figure 2, which can measure distance among Figure 4 shows that model which generate emotional
the sentences along the domain information. features from the model should collect the data from the SNS
The method that we used for recognize and inference the which contains various information in it. It relative to the user
user emotions from fusing mutual utterance information such emotion via the social network, for example, text, picture, and
as text. Additionally, user information such as location, sex, social activities. We use transformed data in previous steps
age, facial expression or any other personally generated and put this to our emotional attention based model to generate
signals are can be collected through the wearable devices for contextual sensitive generation for proper response.
more accurate inference. In addition, the study of the criteria for the human moral
We should choose 8-kind of emotions from a judgement on this system is needed. The psychiatric
representative emotional which originally more than 8 counseling can affect to the user’s life directly. Before the
emotions to simlify the problem [8]. We collect the data form model generates the respond reation to providing
the public media which drama and radio scripts and SNS intervention, the respond must be carefully taken from an
dialogues which contains manually descripted emotions. ethical standpoint. Aside the technology advance, the system
Figure 1 shows the framework for the emotional recognition can face the unethical situations through the learning process
from various sample inputs. in ethical aspect of the human. For these ethical achievement,
The system also track the user emotion based on life log even form the data collection, to the response generation and
system. The maintenance system collects the bio information feedback, the whole process should be done by continous
continously to track the user context. It enhances emotional attention with ethical view.
collection which used to emotional recognition model. B. Personalized Response Generation
The user’s personal information like age and gender
improves personalized response generation method for
classification and recognition of emotions in probabilistic
sense. The service can generate appropriate response using
the user input features. In the same time, the user’s emotion
extraction be applied according to the user's personal
information like age and gender.
Also, for the proper response generation, system need to
collect pair dataset of user intention and proper response we
expect, the intention previous steps and the right response can
be obtained from both manually and automatically which
came from the structural data of target domain knowledge pair,
such as, knowledge base (KB) or Web document.
The user intention consists of two component briefly, one
is the document context and another is user emotion. The
possible issue of this system for the emotion extraction and
the document generation is that how to deal the user intention
and the document context features together. In the model
above can face the situation that only the user intention or the
Fig. 4. Framework for Emotional feature Extraction. document feature affects the model output. If one of those

439
feature dominates the feature affection, the effectiveness of levels of hierarchy. The system enables the natural language
the model drops severely. processing to understand user state and observation of
continuous user’s emotional changes sensitively.
Consequently, the experience of users who need mental
healthcare will be improved by our approaches. In future
works, we apply the emotional generation to natural language
reponse sentence not just feature extraction.
ACKNOWLEDGMENT
This work was supported by the Institute for Information
& Communications Technology Promotion (IITP) grant
funded by the Korea government (MSIP) (No. R0101-16-
0062, Development of Knowledge Evolutionary WiseQA
Platform Technology for Human Knowledge Augmented
Services).
REFERENCES
[1] B. K. Kim, J. Roh, S. Y. Dong, and S. Y. Lee, “Hierarchical
committee of deep convolutional neural networks for robust facial
(A) Sympathetic response expression recognition,” Journal on Multimodal User Interfaces, pp.
1-17, 2016.
[2] L. Chao, J. Tao, M. Yang, Y. Li, and Z. Wen, “Audio Visual Emotion
Recognition with Temporal Alignment and Perception Attention,”
arXiv preprint arXiv:1603.08321, 2016.
[3] H. Lee, Y. S. Choi, S. Lee, and I. P. Park, “Towards unobtrusive
emotion recognition for affective social communication,” In proc. of
2012 IEEE Consumer Communications and Networking Conference,
pp. 260-264, 2012.
[4] M. Wöllmer, F. Weninger, T. Knaup, B. Schuller, C. Sun, K. Sagae,
and L. P. Morency, “Youtube movie reviews: Sentiment analysis in
an audio-visual context,” IEEE Intelligent Systems 28(3), pp. 46-53,
2013.
[5] A. Hommersom, P. J. Lucas, M. Velikova, G. Dal, J. Bastos, J.
Rodriguez, M. Germs, and H. Schwietert, “Moshca-my mobile and
smart healthcare assistant,” In proc. of e-Health Networking,
Applications & Services (Healthcom), pp. 188-192, 2013.
[6] D. Elmasri, A. Maeder, “A Conversational Agent for an Online
Mental Health Intervention,” In proc. of International Conference on
Brain and Health Informatics, pp. 243-251, 2016.
(B) Informative response
[7] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares,
H. Schwenk, and Y. Bengio, “Learning phrase representations using
Fig. 5. Detailed feature selection mechanism. In (A), the response RNN encoder-decoder for statistical machine translation,” arXiv
generated from probabilistic model that deciced the input sentence expect preprint arXiv:1406.1078, 2014.
sympathetic reaction. In costrast, (B), expect informative reaction.
[8] R. Plutchik, “Emotions and Life: Perspectives from Psychology,
Biology, and Evolution,” Washington, DC: American Psychological
So, we planned detailed feature selection model, we put Association, 2002.
constraint to the user intention feature by putting expected
reaction feature. It helps recognize the user intention whether
he or she wants emotional sympathetic response, for instance,
sympathize, sorry, be astonished, which generated from
probabilistic model or more informative response that gives
user precise information from the (semi-)structured data
storage as we explained above.
Figure 5 said how those feature selection works, the
feature selection done automatically from the one of labeled
user intention that indicates what user wants from the machine,
in view of sympathetic or informative.
IV. CONCLUSION
We introduced an novel chatbot application which
provide conversational mental healthcare service based on
emotion recognition methods and chat assistant platform
which consist of the context sensitive advanced natural
language-based technique to provide personalized response
generation continously. To understand the natural language
input sentences and recognize user’s emotions, system got 4

440
 The potential of conversational agents to provide a rapid HIV counseling and
testing services

Alastair van Heerden, Xolani Ntinga, Khanya Vilakazi

Human and Social Development
Human Sciences Research Council
Pietermaritzburg, South Africa

Abstract—In low-and middle-income countries where demand AIDS is, therefore, a critical first step in the process of seeking
for health services outstrips the available supply of skilled labor, care.
advances in information and communication technologies have
already been shown to hold promise. While much of the mHealth A number of issues exist that prevent people from testing
literature continues to explore mature technologies such as text including fear of stigmatization and discrimination, insufficient
message and web portals, continual advancement in machine knowledge about HIV, long waiting times at health facilities,
learning opens innovative new areas of exploration for public poor relationships with health staff and, for men particularly,
health practitioners. This paper explores one such possibility, a issues of privacy when testing [6], [7]. The development,
conversational agent, able to guide users through an HIV availability, and legalization of HIV self-testing provides one
counseling and testing session. Using commercially available potential solution to some of these issues by offering people the
software (http://api.ai), an agent was designed and built chance to test in a private space at their convenience [8].
according to the Center for Disease Control’s guidelines for the However, not all are in favor of self-testing. The chairman of
provision of HIV counseling and testing in a non-clinical setting. the South African Medical Association has voiced concern
The agent was linked to the Telegram chat client over the potential mental health risks associated with taking an
(http://telegram.org) and 10 testers were invited to participate in unmonitored HIV test [9].
a simulated HIV counseling interaction. Six testers found that
talking to the agent felt natural, and equivalent to chatting to a This paper aims to explore the feasibility of addressing
human. Seven said they would feel comfortable taking a real HIV these self-testing concerns, health facility-related HIV testing
test with the agent. Key concerns with the current agent were the barriers, and human resource shortages, by designing a
use of overly formal language, the speed at which the agent conversational agent capable of guiding someone through the
responded (too fast) and the agent either misunderstanding or pre-test counseling phase of an HIV test. While counseling
not understanding the tester. Positive sentiment towards the seems a very human skill, recent evidence suggests that with a
agent included the fact that testers felt like the session was more large enough set of data, it may be possible to model
private and anonymous, and avoided the need for them to visit a conversations without the need for the model to be restricted to
public health facility and stand in a long queue to get tested. a specific domain (content area), or crafted with predefined
rules [10]. Using an online counseling dataset consisting of 3.2
Keywords—conversational agents; digital assistants; HIV;
public health; pre-test counseling; HIV testing; low-and middle-
million messages sent by 408 counselors, [11] were able to
income countries construct a model able to predict, with an accuracy of 59%, the
likelihood of a counseling session ending positively, after
reviewing the first 20% of a conversation between a client and
I. INTRODUCTION counselor. Check-questions, message length, similarity to last
The potential impact of rapidly advancing technology on message and hedges (softening words like “sometimes” and
the future of employment has some scholars worried and others “maybe”) were all significantly, positively correlated to a
not [1], [2]. In low-and middle-income countries where successful counseling outcome. Further, the potential of
demand for health services outstrips the available supply of technology-based self-service channels and digital health
skilled labor, these advancements in technology may have the interventions to support patients in their everyday life is
potential to bring positive change. As an example, the South beginning to be documented [12], [13]. Exploration of
African public health sector is currently strained due to the lack conversational agents in a health care setting suggest that users
of appropriately trained and skilled health care workers that are are accepting [14] of, and can form a working alliance with
required to deal with the largest human immunodeficiency [15], embodied conversational agents. Examples of such agents
virus (HIV) epidemic in the world[3]–[5]. With treatment, HIV include Florence (http://getflorence.co.uk), Molly
and acquired immune deficiency syndrome (AIDS) can now be (http://sense.ly), and koko (http://itskoko.com). While there is
considered a chronic infectious disease with those infected able increasing interest in the use of conversational agents in a
to live long and healthy lives. However, without antiretroviral health care context, the extent to which patients find them
therapy, people living with HIV remain infectious to others and acceptable for different use cases needs further evaluation [16].
susceptible to opportunistic infections. Testing for HIV and

978-1-5386-3148-5/17/$31.00 © 2017 IEEE 80

 The aim of this study was to assess whether it is possible quick response buttons (e.g. yes and no as clickable buttons
for public health practitioners to use commercially available above the onscreen keyboard) and emojis, (3) an incoming and
hardware and software to create a simple HIV counseling and outgoing context, which is used to limit when certain intents
testing, conversational agent that was acceptable for use. More can be triggered i.e. it holds the conversational pieces together
than the technical feasibility, the experiment aimed to assess and helps maintain the thread of the conversation. For example,
people’s reaction to the use of this type of technology as a if the user says “yes” and the context is “ready-to-test”, then
support or replacement for traditional human led HIV the “ready to test” intent is triggered, rather than the intent that
counselling. The paper presents the tools, models, training and responds to a “yes” when the “ever-tested-for-HIV” context is
initial reactions of people to Nolwazi, an HIV pre-test active and (4) entities - which were not used in building this
counseling agent. particular agent. When finalized, the agent was duplicated to
create a male counselor called “Lwazi” and a female counselor
II. METHODS called “Nolwazi”. These are common names in South Africa
where the study took place.
A. Modeling an HIV and AIDS counseling session
A number of guidelines exist that outline the process of C. User feedback
providing voluntary HIV counseling and testing services. This Ten testers were approached to try out the conversational
agent was designed around the Center for Disease Control’s agent, without being required to take an actual rapid HIV self-
(CDC) guidelines for the provision of HIV counseling and test during the process. Each tester was temporarily provided
testing in a non-clinical setting [17]. The basic CDC model is with a Samsung J1 mobile phone running Android 4.1 and the
divided into 6 stages. These are, (1) introduce and orient client Telegram messaging app. They were then offered the
to session; (2) prepare for and conduct rapid HIV test; (3) opportunity to talk to either Lwazi or Nolwazi. At the end of
conduct brief risk screening; (4) provide initial results and the interaction, basic feedback was elicited about their
follow protocol for confirmatory testing; (5) develop care, experience with the agent. All ten sessions took place on one
treatment, and prevention plan based on results; (6) refer and day at the offices of the Human Sciences Research Council,
link with medical care, social and behavioral services. For the Pietermaritzburg, South Africa.
purpose of this research we selected only some of the stages as
a proof of concept, namely, (1) introduce and orient client to III. RESULTS
session, (2) prepare for and conduct rapid HIV test, (3) conduct
brief risk screening and (6) refer and link with medical care, A. Modeling an HIV and AIDS counseling session
social and behavioral services. Fig. 1 outlines the basic counseling model used for the
Suggested counseling scripts were extracted from the CDC design of the agent, based on the CDC guidelines for the
guidelines and populated into a non-linear dialog flow model. provision of HIV counseling and testing in a non-clinical
This basic model was then reviewed by a registered clinical setting. The model was condensed to an introduction and
psychologist and a trained HIV and AIDS counselor who has orientation, followed by a brief risk assessment, before
10 years of experience offering community based HIV concluding with the rapid HIV test. While traditional
counseling and testing services. Their suggestions were counseling is typically non-linear, the HIV pre-test counseling
incorporated into the flow. In particular, it was suggested that session was found to be a fairly linear process with a clear
at all times a person should be able to 1) request to talk to a beginning, middle, and end.
trained counselor and 2) any time mention was made of
thoughts of self-harm or suicidal ideation that the agent should B. Agent design and training
respond by linking the tester to a trained professional. Fig. 3 presents a portion of the final non-linear dialog flow
model that was implemented in api.ai. The agent code for
B. Agent design and training Nolwazi is available on request. Two key conversational points
A number of commercially available online sites that were found to be critical with a number of potential answers
offered Natural Language Processing as a service were needing to be handled by the agent. The first was “Why did
reviewed for suitability. Criteria used in the evaluation of the you come for a test today?” and the second “What worries do
offerings were (1) integration with chat clients (such as you have about testing for HIV?” Using the Natural Language
Whatsapp, WeChat and/or Telegram), (2) graphical user Processing abilities of api.ai the agent was designed to match
interface for bot design and training, (3) ability to export model any utterance by the tester relating in some way to 1) Having
for publication and (4) offers webhook integration to interface had unprotected sex and 2) Having cheated on a partner (or
with other services and allow the addition of anticipated future being cheated on) with the question “why did you come for a
functionality. api.ai11 was selected and user – agent test today?”. With respect to “what worries do you have about
interactions were entered into the system as an “intent”. api.ai testing for HIV?”, the agent was able to respond intelligently to
intents consist of four components, (1) phrases a user can say issues of 1) worry about being HIV positive and 2) having the
to trigger the intent, (2) one or more responses the agent can tester’s partner find out that they had taken an HIV test. In all
use to reply, including the possibility of images, video, audio, cases, and at any point in the conversation, the agent was able
to respond to comments about thoughts of self-harm and
suicidal ideation by asking whether the tester wanted to be
1
http://www.api.ai linked to a real human counselor. At the end of testing (if

978-1-5386-3148-5/17/$31.00 © 2017 IEEE 81

 1. Introduce and orient Introduce yourself, describe role
client to session and give brief session overview:
• How long the session will take.
• Process for conducting the test.
• How results are returned

Ask how the client decided to be

2. Conduct brief risk tested; listen and probe for
previous testing history and
screening
indicators of increased risk:
• Potential exposure in last 24-72
hours
• Potential exposure in last 3
months
• Symptoms
3. Prepare for and
• Ongoing risk
conduct rapid HIV test

Address indicators of increased risk and

tailor prevention messages to client’s
situation:
• If potential exposure in last 24-72
hours, discuss PEP.
• If potential exposure in last three
Process of conducting the
months, discuss testing for acute HIV
rapid HIV test, including:
infection, condoms, and re-testing.
• Test being used
• If symptoms, discuss importance of
• Sample collected
accessing medical care
• Time for results to get
• If ongoing risk, discuss PrEP, condoms,
ready
re-testing, partner and couples testing.

Fig 1. Adapted HIV counseling flow model used for the design on
the conversational agent.

positive or having an indeterminate result) the tester was

provided with a link to a map showing the location of the
nearest health facility.

C. User feedback Fig 2. Example of interaction between tester and Lwazi (the male agent)
in the Telegram messaging client.
The average age of testers was 30.2 years (s.d. 2.3) with 5
of the 10 being female. During an average session length of 26 with the testers. Other feedback given by the three testers who
minutes (s.d. 9.3), the agent was able to guide 6 of the 10 did not find the simulation realistic was that the language was
testers to a point in the counseling progress where the next step too formal and not congruent with how real people talk to each
would have been to complete an HIV rapid self-test. An other on chat apps.
example of the interaction between agent and client using the Seven of the ten testers said that they would be open to
Telegram messaging platform can be seen in Fig. 2. being offered an opportunity to undertake a real HIV test with
There were mixed reactions to the quick reply buttons, with the agent. Among the reasons offered for their willingness were
more people (6) preferring them to having to type out that the interaction felt private and anonymous, they did not
responses. Reasons included the fact that it was easier than have to worry about being judged by the counselor and it could
typing, for others that it ensured that the agent understood them save time not having to wait at a clinic for a counselor. For the
– which didn’t always happen when they typed their own three who would not consider taking a real HIV test only one
answers. One tester stated that they were preferable when the clear reason was offered and that was that there may arise a
conversation lent itself to a discrete set of responses, with the situation during the counseling session that the agent could not
remaining three preferring to type all their answers. A clear address.
majority of testers (6) thought that the familiar chat interface, Testers were not particularly concerned with the gender of
avatar image, and appropriate responses, made the interaction the agent, with six stating that any gender would be fine. Of the
feel substantially like talking with a human. Some testers (2) remaining testers, four stated that they would prefer a female
report that the conversation flowed easily and that they felt safe agent, such as Nolwazi, over a male version of the agent. Only
and unjudged by the agent. For the four testers who did not one tester said they wanted to speak specifically with Lwazi,
find the experience to be realistic, a weakness in the experience the male agent. While the overall experience and feedback did
was the fact that text appeared unrealistically fast. Large blocks not vary much by gender there were some indications that age
of text were delivered to the tester within a half a second of the may be a factor in the acceptability of such agents, with
tester submitting their response. Some testers described this as younger individuals appearing to be more open to such
making the interaction uncomfortable as the responsibility for innovation. With such a limited sample, no real conclusions
keeping the conversation flowing was perceived to always rest can be drawn beyond this simple observation.

978-1-5386-3148-5/17/$31.00 © 2017 IEEE 82

 Fig 3. Extract of the conversational agent’s model.

IV. DISCUSSION with traditional face to face HIV counseling and testing
This study is clearly beset by a number of limitations and services (HTS). Anonymity, privacy, speed and easy access,
shortcomings. Despite these, the enquiry suggests that people which are all issues that currently limit the effectiveness of
are open to the idea of being guided through a health screening regular HTS, were noted by testers to be the strengths of the
process by an anonymous non-human conversational agent. agent. This could be a particularly useful model for men who
The data also point towards improvements and future are well known to be less frequent users of public health
directions of enquiry that may see an agent that is truly capable facilities due to issues of privacy, clinics not being male
of offering public health screening in low resource settings, for friendly spaces and the waiting times required during working
both chronic infectious disease (such as HIV) as well as non- hours [18].
communicable diseases (such as diabetes and hypertension).
B. Study Limitations
A. Reviewing the current conversational agent The largest of the limitations of this research is the
Misunderstanding and not understanding user input, type of simplicity of the agent. A fairly linear yes / no conversation
language used, narrowness of knowledge base, and speed of emerges from the current design and prevents the level of depth
agent response, all detracted from the user experience. Most of and understanding that a realistic counselor would display. The
these issues do not seem to pose insurmountable technical second major limitation is the very small number of testers and
challenge. They also seem to hint at an opportunity to design the limited feedback that was obtained. Third, the study has
different types of agents that could be tested for effectiveness limited theoretical, technical and methodological rigor. Fourth,
with users of different demographic profiles. For example, for only one experiment was conducted using the agent. Finally,
an 18 year old girl, an agent with age-appropriate syntax, many of the technical details underlying the agent were taken
response speed and knowledge, could interact with the subject for granted rather than fully explored.
in a very different way than might be ideal for a 45 year old.
C. Future directions
Remarkably, despite these shortcoming, 7 out of the 10
A number of avenues of further exploration are opened by
testers said that they would consider taking and HIV self-test
this study. First and most importantly, a more robust agent
with the agent as a counseling guide. For such a rudimentary
capable of responding to a broader range of questions,
agent this is a surprising, yet encouraging finding. There
anxieties, and concerns of HIV testers needs to be designed.
appears to be an appetite for such an agent, particularly among
This agent should be more easily able to spend time discussing
younger more technologically savvy users. Reasons given for
issues raised by the tester, while at the same time slowly
why people would consider being guided through an HIV
guiding them towards the end goal of offering the user an
counseling and testing session by a conversational agent are
opportunity to test. Rather than the somewhat linear agent
interesting as they speak directly to the challenges observed
designed here, a more realistic agent would be able to move

978-1-5386-3148-5/17/$31.00 © 2017 IEEE 83

 forward and backward through the conversational flow, re- Boston, 2017.
addressing concerns that may already have been spoken about [3] N. Poku and A. Whiteside, The Political Economy of AIDS in Africa.
that re-emerge. The agent needs to further be fleshed out with Abingdon, United Kingdom: Routledge.
some of the counseling tools delineated by Althoff, Clark and
[4] A. Soucat and R. Scheffler, The Labor Market for Health Workers in
Leskovec (2016) that have been empirically shown to lead to Africa: A New Look at the Crisis. Washington, D.C.: The World
better online counseling outcomes [11]. This would include Bank, 2013.
counseling techniques such as check questions (statements that
[5] B. M. Mayosi and S. R. Benatar, “Health and Health Care in South Africa
make it clear the counselor understands), thanking the user for — 20 Years after Mandela,” N. Engl. J. Med., vol. 371, no. 14, pp.
trusting the counselor, and the use of hedges. Re-working the 1344–1353, Oct. 2014.
agent scripts to use language which is more appropriate to the
youth texting culture would also go a long way to improving [6] G. D. Kwapong, D. Boateng, P. Agyei-Baffour, and E. A. Addy, “Health
service barriers to HIV testing and counseling among pregnant
the user experience. A further improvement could be made to women attending Antenatal Clinic; a cross-sectional study,” BMC
the agent by more realistically simulating typing speed and Health Serv. Res., vol. 14, no. 1, p. 267, Dec. 2014.
variably delaying how quickly a response is sent. While hinted
[7] M. Strauss, B. Rhodes, and G. George, “A qualitative analysis of the
at in the current agent, other areas for exploration and barriers and facilitators of HIV counselling and testing perceived by
improvement are how the agent connects testers with people adolescents in South Africa,” BMC Health Serv. Res., vol. 15, no. 1,
and places in the outside world. By identifying distress through p. 250, Dec. 2015.
sentiment analysis performed on the user text, it seems possible
[8] T. Makusha et al., “HIV Self-Testing Could ‘Revolutionize Testing in
that a trained human counselor could be alerted to the situation South Africa, but It Has Got to Be Done Properly’: Perceptions of
and introduced, with the permission of the user, into the Key Stakeholders,” PLoS One, vol. 10, no. 3, p. e0122783, Mar.
conversation. 2015.

For those users who complete the test and are HIV positive, [9] M. Richter, W. D. F. Venter, and A. Gray, “Home self-testing for HIV:
careful thought will need to be given as to how best the user AIDS exceptionalism gone wrong.,” South African Med. J., vol. 100,
no. 10, pp. 636–642, 2010.
could be linked into a formal health care setting. In this agent, a
link to a map of the closest clinic is provided. Further thought [10] O. Vinyals and Q. Le, “A Neural Conversational Model,” Jun. 2015.
needs to be given to alternative approaches to linkage to care. [11] T. Althoff, K. Clark, and J. Leskovec, “Large-scale Analysis of Counseling
Home and community based HIV testing are well established Conversations: An Application of Natural Language Processing to
areas of research and best practices described in this literature Mental Health,” May 2016.
could provide a solid foundation for the hand-over to care [12] R. Agarwal, G. Gao, C. DesRoches, and A. Jha, “The Digital
component of the agent [19]. The current agent provides three Transformation of Healthcare: Current Status and the Road Ahead,”
quick response buttons namely “Positive”, “Negative”, “Invalid Inf. Syst. Res., vol. 21, no. 4, pp. 796–809, 2010.
Test”. For some users, there may be the possibility that, after
[13] L. Marsch, S. Lord, and J. Dallery, Behavioral Healthcare and
receiving a positive result, they feel unable to share the Technology: Using Science-Based Innovations to Transform
outcome, even with a non-human conversational agent, due to, Practice. New York, USA: Oxford University Press, 2014.
for example, anthropomorphization of the counseling bot.
[14] T. W. Bickmore, S. E. Mitchell, B. W. Jack, M. K. Paasche-Orlow, L. M.
Another future area of work could address how best to engage Pfeifer, and J. ODonnell, “Response to a Relational Agent by
with the user when it comes to disclosing the result. One Hospital Patients with Depressive Symptoms.,” Interact. Comput.,
possibility is the use of a trained convolutional neural net vol. 22, no. 4, pp. 289–298, 2010.
which would be able to detect when the rapid test is positive,
[15] T. Bickmore, A. Gruber, and R. Picard, “Establishing the computer-patient
negative or invalid. A user could be asked by the agent to take working alliance in automated health behavior change
a photo of the test result using the chat client. This image interventions.,” Patient Educ. Couns., vol. 59, no. 1, pp. 21–30, Oct.
would be classified by a convolutional neural net, and the user 2005.
then given the result. [16] J.A. Micoulaud-Franchi, P. Sagaspe, E. de Sevin, S. Bioulac, A. Sauteraud,
While much work is still required to understand the and P. Philip, “Acceptability of Embodied Conversational Agent in a
Health Care Context,” in Intelligent Virtual Agents: 16th
potential use, legal implications and impact of such International Conference, IVA 2016, Los Angeles, CA, USA,
conversational agents in health care, these findings suggest September 20--23, 2016, Proceedings, D. Traum, W. Swartout, P.
such enquiry may have merit. Khooshabeh, S. Kopp, S. Scherer, and A. Leuski, Eds. Cham:
Springer International Publishing, 2016, pp. 416–419.
ACKNOWLEDGMENT [17] Centers for Disease Control and Prevention, “Implementing HIV Testing
in Nonclinical Settings. A Guide for HIV Testing Providers,”
The authors would like to thank the ten testers who Atlanta, 2016.
volunteered to try out the conversational agent with no
compensation or incentive. [18] M. Sharma, R. Ying, G. Tarr, and R. Barnabas, “Systematic review and
meta-analysis of community and facility-based HIV testing to
address linkage to care gaps in sub-Saharan Africa,” Nature, vol.
REFERENCES 528, no. 7580, pp. S77–S85, Dec. 2015.
[1] C. B. Frey and M. A. Osborne, “The future of employment: How [19] A. van Heerden et al., “Perceived mHealth barriers and benefits for home-
susceptible are jobs to computerisation?,” Technol. Forecast. Soc. based HIV testing and counseling and other care: Qualitative findings
Change, vol. 114, pp. 254–280, Jan. 2017. from health officials, community health workers, and persons living
with HIV in South Africa,” Soc. Sci. Med., vol. 183, pp. 97–105, Jun.
[2] J. Bessen, “Automation and Jobs : Technology Boosts Employment,”
2017.

978-1-5386-3148-5/17/$31.00 © 2017 IEEE 84

978-1-5386-3148-5/17/$31.00 © 2017 IEEE 85
 &KDWERWDVDOHDUQLQJUHVRXUFH"&UHDWLQJ
FRQYHUVDWLRQDOERWVDVDVXSSOHPHQWIRUWHDFKLQJ
DVVLVWDQWWUDLQLQJFRXUVH
Donn Emmanuel Gonda Beatrice Chu
Center for Education Innovation Center for Education Innovation
The Hong Kong University of Science and Technology The Hong Kong University of Science and Technology
Hong Kong Hong Kong
[email protected] [email protected]

Abstract²One of the technologies that educators can learning based on their needs and behaviors. For example, a
leverage is the use of chatbots or conversational agent to chatbot is available 24/7, can open a communication line that
support teaching and learning. Moreover, with the recent is more responsive than the traditional means such as email
advancement in AI, companies like Google, IBM, and Amazon or forum.
provide platforms that are focused on building conversation
rather than the technicalities of computer programing, in However, creating a chatbot, like conversational agents,
particular, natural language processing. In this case study, we are challenging to build [10]. Chatbot usually requires a
will look into the application of a simple rule-based chatbot, specific programming knowledge such as Natural Language
based on Google DialogFlow, in a teaching assistant training Processing, and Machine Learning. These knowledge are
course run by a Center for Education Innovation at the Hong just some of the technicalities that one should know before
Kong University of Science and Technology. We will unpack building a chatbot. However, with recent advancement in AI
the details of how a chatbot can be integrated into both online and conversational agents, companies like Google, IBM, and
and face-to-face content to augment some of the challenges that Amazon provide platforms that focus on the flow of
can be encountered in teaching. conversation rather than the computer programing of [11].
These platforms democratize the technical know-how in
Keywords²Chatbot, Teaching Assistant Training, building these chatbots, which makes it easier for a non-
Conversational Agent, Chatbot for Education
technical person to pick it up and build customized chatbots
I. INTRODUCTION that will fit the learning context.
The application of technology inside the classroom, such In this case study, we will look into the application of a
as using teaching videos for flipped classroom [1] and for simple rule-based chatbot, based on Google DialogFlow, in a
MOOCs [2], embedding learning analytics to better teaching assistant training course run by Center for
understand students' online behavior [3], and producing Education Innovation (CEI) at the Hong Kong University of
gamified contents to engage students [4], [5] are some of the Science and Technology (HKUST). The course team developed
few trends in higher education. These trends, though they this bot as a supplement to the flipped classroom format
might not be entirely new, enables educators to see their created for the course. We will first look into the challenges
practice into the new light brought by the fast-changing and difficulties posed by the current course design such as
technology landscape. Advances in education technology
x providing scalable support for the Graduate Teaching
provided the tools and application to collect data extensively
Assistants (GTAs),
and at the very minute level. It also allowed educators to
scale up the delivery of their learning contents and provide a x addressing GTAs' diversity and
more personalized experience for the students.
x maximizing efficiency in daily task routines
These changes in education technology landscape call for
a more personalized experience to support individual In section 2, we will tackle these issues by looking into
learning patterns and to cater to the diverse needs of each the format of the current course design and the challenges
student in both online and face-to-face interactions [6]. met by the course team. Section 3 will focus on the chatbot
Educators are now faced with this challenge to keep up with design, using Google DialogFlow, how it will be used to
the needs of this generation of learners. Further, Rezende and address the concerns raised in this case study. Finally, the
de Souza Barros [7] find out that students who are using the last section will give us the overall recommendation based on
online component as a resource view the online materials in the lesson learned from this experience.
a non-linear fashion. This finding was based on their study of
II. CHALLENGES WITH THE CURRENT COURSE DESIGN
the different navigation patterns of students in an online
environment. This finding means that the students. Needs Every year, HKUST assigns roughly 550 full-time
support and feedback at various learning points. Also, this is research postgraduate (RPg) students to the role of Graduate
where technology should be maximized to ensure that Teaching Assistant (GTA) to support teaching and learning
learning support will be available for these students 24/7. in undergraduate courses. These GTAs join the teaching
support team in their respective departments to carry out
One of the technologies that educators can leverage is the various teaching functions such as teaching tutorials,
use of chatbots or conversational agents [8], [9]. Fadhil and conducting laboratory sessions, marking assessments, and
Villafiorita [4], in his paper on adaptive learning, noted that providing feedback as assigned by the department. However,
these chatbots enable the learners to personalize their for most of the time, these GTA students are coming from

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE

Authorized licensed use limited to: Newcastle University. Downloaded on October 25,2020 at 23:16:53 UTC from IEEE Xplore. Restrictions apply.
their discipline with little or no background in teaching. In a Further, in another study, Merrill [15] pointed out the
similar study with the engineering education, Sohoni, Cho, importance of guidance as to the students' learning progress.
and French [12] mentioned that the skills in teaching are In this case, there is a need for a mechanism where students
acquired during the on-the-job training, and the teachers can ask questions particular to the theories and concepts
practiced on their students, in this case, the undergraduate within their discipline. They also need a space where they
students. For this very reason, the CEI, the University's can bounce back their thinking from the things they learned
central unit dedicated to promote and develop pedagogical online or during the activities in the F2F sessions.
practices within the university, and designed a training
course to augment the gap in the skills needed by the GTAs C. Optimize daily task routines
as they carry out their teaching duties in the University. The This issue is tied with the scalability of support but
center aims to revamp the current GTA training program instead framed in classroom management thinking. As
using a more proactive approach by creating an online previously mentioned, each module is delivered at least 15
component for the flipped classroom and using active times a year, which means that the instructor will have to
learning strategies for the face-to-face sessions. repeat the same lesson multiple times. On top of that, these
sessions are delivered using active learning strategies where
This new format will provide more time for GTAs to GTAs need to work on a series of activities through the
reflect on their learning experience. It will also allow them to entire session. This issue was raised by the instructor when
discuss how the strategies observed can be further applied in she mentioned that "It is hard for the [her] to give feedback
their actual teaching. The new course design is consist of five to individuals or groups and to keep everyone moving or on
modules focusing on various aspects of teaching, such as the right track.´ This issue can also be seen in running
marking, presentation, and classroom management. Each flipped classroom approach where the focus of in-class
module is delivered through a 120-minute workshop where activity gears towards student-centered learning. It is also
GTAs receive goes through various activities followed by a assumed that in an active learning classroom, there is a shift
few minutes of consolidation in between. However, this new from information giving to facilitating a role for the
course format still poses multiple issues in the delivery and instructor [16]. Also, receiving prompt feedback will allow
facilitation, particularly in providing support, addressing the student to explore deeper into the topic. The prompt
diversity, and optimizing routines. feedback is essential, especially during the F2F session,
A. Providing scalable support for the GTAs where the students are expected to consolidate their learning
from the online materials and the activities.
In one of the conversations with the course instructor, she
mentioned that the "workshops are run in multiple sessions, III. RESEARCH METHOD
covering similar if not the same content.´ She further added
that the ³questions raised by GTAs are similar...". In the The design-based research [17] approach was used to
previous course design, each module is conducted for at least document the course design, chatbot's multiple iterations, the
15 times every year to make it available for more or less 600 interaction between the users, in this case, the GTA students
GTA students. Further, there are only three instructors in the and the course team [18]. This research methodology focuses
course team, making a ratio of one instructor to 200 students. on the. Principles underpinning the design process by
This ratio makes it difficult, if not impossible, for the identifying the problems, refining the solution, and
instructor to provide scalable support to their students. documenting the entire process. The research team collected
However, there is a silver lining in this issue since it is a 12 hours of lecture videos, recorded 4 hours of the
training program, most of the lessons are repeated, and the interview from the course team, documented 11 course
majority of the contents are shared among the 600 GTA development meetings, and aggregated 300 students'
students. This repetitive process of delivering the same feedback. The team collected the data from September 2018
materials and answering similar questions in every session till June 2019. This methodology fits well in this study since
can take a tremendous amount of time from the instructor it will allow the researchers to outline the details of the
and limit her from addressing GTAs' concerns or focusing development which will enable readers to understand the
her time on more meaningful activities with her students. On design process of the study entirely.
a similar note, Chickering and Gamson [13] suggested that IV. ADDRESSING THE COURSE DESIGN CHALLENGES USING
constant communication with the students and prompt CHATBOT
feedback are some of the good practices in teaching in higher
education. In the coming semester, Fall 2019, CEI redesigned course
will provide robust learning content and hands-on
B. Addressing GTAs¶ diversity experiences for the GTAs. In the new course design, the
Currently, a majority of the GTAs are coming from Hong program will have five modules which focus on active
Kong at 55%, followed by students from Mainland China learning, presentation and facilitation, feedback, micro-
with 37% while the distribution between MPhil and Ph.D. teaching, and Canvas Learning Management System (LMS).
students will be at 25% and 75% respectively. On top of that, These modules are offered in a flipped-classroom approach
the University is divided into four schools with 18 divisions where the learning theories are delivered online, and the
and departments. This structure of the university makes it face-to-face hands-on activities consolidate the theories
challenging for the course team to address the diverse needs learned. The flipped classroom approach allows the team to
of GTAs coming from diverse subject matter expertise and double the class size, which enabled them to maximize room
educational background. Furthermore, the instructors have to utilization and halved the number of sessions. Furthermore,
cater to department-specific needs in terms of the teaching moving the learning contents online creates more time for in-
materials, presentation skills, and content-specific feedback. class activities [19]. Given this situation, facilitation is
Nolan [14] defined it as tailoring to fit individual needs and needed more than ever for both online component and face-
giving the learners the ownership of their learning. to-face sessions.

Authorized licensed use limited to: Newcastle University. Downloaded on October 25,2020 at 23:16:53 UTC from IEEE Xplore. Restrictions apply.
 On the one hand, the course team realizes that when the
GTAs are online, their access for support and feedback relies
heavily on the automated feedback from the LMS. Further, if
they would like to ask open-ended questions, they can only
do it either using the discussion forum or through email.
However, getting feedback through the discussion forum or
email may take some time, and it depends on the availability
of the instructors. On the other hand, GTAs may have some
quick questions that they would like to ask or verify as they
go through the series of activities. Some of these questions
deal with administrative inquiries such as submission format
or grading criteria, or it can be about the learning content.
These questions can be repetitive, and dealing with these
may cause valuable time for the instructors, and the
Fig. 2. Chatbot embedded in Canvas LMS
execution of the in-class activity can be interrupted.

TABLE I.  MAPPING OF MODULE FLOW WITH LEARNING CONTENT

One solution that the team came up with is to create a
AND CHATBOT ROLE simple rule-based chatbot that can handle some of these
questions from the GTAs. In table 1, we have mapped out
Module Learning Chatbot Role
Flow Content
the general flow for each module with the main learning
Pre- Lecture For dictionary content and the role of the chatbot in supporting these
class videos or FAQ; learning contents.
Online Information
activities giving A. Creating a User-Led Chatbot
For formative In addressing the diversity and optimizing issues, a User-
assessment; led conversational chatbot was designed to serve as an FAQ
MCQ quiz
or a dictionary type chatbot where GTAs can ask a question,
In-class Group Administrative, and the chatbot will give out a definition (see image 1). This
activities Instruction type of chatbot is widely used in the field of finance and
Individual giving technology sector. Chatbots like Siri, Cortana, Bixby, and
activities Google assistants are just some of the few chatbots that are
common to us. However, unlike these commercial chatbots,
Post- Readings For dictionary
class Deep- or FAQ; the ones that are being built for this course are explicitly
dive Information designed to do a handful of tasks such as defining words that
materials giving are specific in each the module.
These bots can be embedded throughout the LMS, and it
can serve both as a pre-class and post-class of each module.
The goal of these chatbots is to encourage the students to
think and ask questions during the learning process. By
embedding these chatbots, GTAs can quickly raise their
question that they may have as they go through the learning
process. Moreover, since the chatbot can handle multiple
questions from various users at the same time, it can give the
GTAs a more personalized experience in learning the
content.
B. Creating a Bot-Led Chatbot
Another type of chatbot is a bot-led chatbot. These types
of chatbots use a reversed approach where the initiator of the
question is the bot itself (see image 2). These bots can be
designed to address the diversity of learners by creating a
branching scenario [10] initiated by a question. These
branching scenarios can adapt to the learners¶ ability and
knowledge. One common application is to convert the
typical multiple-choice question into a chatbot, and instead
 of static feedback, the chatbot can further ask a question that
Fig. 1. Sample questions can lead the students back on the correct learning path.
Though this process can be designed into a learning
management system or other learning packages authored in
Adobe Captivate or Articulate Storyline, one clear advantage
for these bots is the ability to discern simple conversation
language. Meaning that the MCQ can allow students to
respond in free or open reply.
For example, the chatbot can start with the question
regarding facilitation, and it can further ask whether the GTA

Authorized licensed use limited to: Newcastle University. Downloaded on October 25,2020 at 23:16:53 UTC from IEEE Xplore. Restrictions apply.
is interested in knowing more about laboratory facilitation or Once the content-specific (intents and entities) word have
tutorial facilitation. This can immediate help the GTAs who been placed, the next step is to design the conversation flow.
will focus on helping in the laboratory and vice-versa. This is where the commercial chatbot platform really shines.
Commercial chatbot platforms provide an intuitive and
C. Creating a Chatbot for Education straightforward process to create these conversations.
To build these chatbots, we have to rely on the Finally, what makes Google DialogFlow stand out from its
instructors' experience, notes, and reflection to list the initial competitors is that it provides a one-click integration
question needed for the training phases (see image 3). At this solution. Therefore, further cutting down the technicality of
point, Google DialogFlow only requires at least 3-5 implementing the chatbot. These allow the educator to easily
questions for it to be able to run its machine learning embed the chatbot in an LMS or course website the same
algorithms for natural language processing. Then, we map way as embedding a YouTube video.
out these questions with the learning content in order for the
chatbot to determine content-specific words or phrases that V. CONCLUSION
the GTAs may ask. These set of words are saved as entities This initiative, though there is still much work to do,
in the system (see image 4). shows that there is a gap in the teaching and learning
landscape that chatbot can fill. However, the current natural
language processing is still in its early stage, using a chatbot
in the field of education is promising. In particular, these
chatbots can address some of the logistical and diversity
issues that a classroom might have. These issues, from
simple Q&A and quick information to scenario-based
question, can be addressed by a chatbot if appropriately
designed.
A. Chatbot and Dataset
Provided with the current landscape of this technology,
developing a chatbot is resource-intensive. Those who wish
to develop a chatbot for a particular course would require at
least a few semesters of data and a tremendous amount of
time to clean and organize it. Hence, it is strongly advised to
 pick a course with at least a few hundred students. Using it in
Fig. 3. Sample questions for Intent
a large class will not only increase the impact of the chatbot
but also make the data collection process faster. The clean
and organized dataset is still the backbone of creating
educational chatbots.
It is also worth noting that a carefully designed rule-
based chatbot can augment the scarceness of data. Adjusting
the number of questions and branches can make the chatbot-
making process manageable and feasible. Another method to
address the scarcity in data is to create a chatbot specific to a
particular task. For example, creating multiple chatbots
across the LMS to answer a specific topic is a lot more
controllable in terms of the dialog flow and the collection of
data.
B. Need to Improve the Dataset for Educational Chatbot
We also think that there is a need for educators to
Fig. 4. Sample words for Entities continue to push for the limits of this tool in education. This
will steer awareness and curiosity among the members of the
Moreover, these content-specific words or phrases is the teaching community. Moreover, by doing so, we will be
key to personalizing the responses of the chatbot. Hence, improving the quality of the data set for which can be
address the diversity issue in the subject matter-specific beneficial for the wider community.
content. For example, if a GTA would like to get some
examples of procedural learning, the chatbot may further ask Another realization that we learn in building these bots is
the GTA to be specific or which field of study. Alternatively, that there is a drought in the resources from these
if a GTA an example of procedural learning in engineering, commercial platforms. Majority of the readily made chatbots
then the chatbot may give an example on using a formula to can be found in the field of business and medicine. However,
compute for the unknown. While if the GTA asked for in terms of educational purpose with the learning design
procedural learning in the field of science, then the chatbot embedded, the resources are close to zero.
may provide a laboratory procedure example. The seemingly REFERENCE
complicated questions that need to be addressed by the
instructor can now be transferred to the chatbot system. This [1] Y.-H. Chiang and H.-C. Wang, ³Effects of the in-flipped
makes the chatbot an appealing tool to address low-level classroom on the learning environment of database engineering,´
questions or repetitive question. Int. J. Eng. Educ., vol. 31, no. 2, pp. 454±460, 2015.

Authorized licensed use limited to: Newcastle University. Downloaded on October 25,2020 at 23:16:53 UTC from IEEE Xplore. Restrictions apply.
[2] K. F. Hew, ³Promoting engagement in online courses: What [18] T. Anderson and J. Shattuck, ³Design-based research,´ Educ.
strategies can we learn from three highly rated MOOCs,´ Br. J. Res., vol. 41, no. 1, pp. 16±25, 2012.
Educ. Technol., vol. 47, no. 2, pp. 320±341, Mar. 2016. [19] M. B. Gilboy, S. Heinerichs, and G. Pazzaglia, ³Enhancing
[3] T. Elias, ³Learning analytics: Definitions, processes and student engagement using the Flipped Classroom,´ J. Nutr. Educ.
potential,´ learning, pp. 1±23, 2011. Behav., vol. 47, no. 1, pp. 109±114, 2015.
[4] A. Fadhil and A. Villafiorita, ³An adaptive learning with
gamification & conversational UIs: The rise of CiboPoliBot,´ in
User Modeling, Adaptation and Personalization 2017
Conference, 2017, pp. 408±412.
[5] K. F. Hew, B. Huang, K. W. S. Chu, and D. K. W. Chiu,
³Engaging Asian students through game mechanics: Findings
from two experiment studies,´ Comput. Educ., vol. 92±93, pp.
221±236, Jan. 2016.
[6] X.-H. Wang, J.-P. Wang, F.-J. Wen, J. Wang, and J.-Q. Tao,
³Exploration and practice of blended teaching model based
flipped classroom and SPOC in higher university,´ J. Educ.
Pract., vol. 7, no. 10, pp. 99±104, 2016.
[7] F. Rezende and S. de Souza Barros, ³Students¶ navigation
patterns in the interaction with a mechanics hypermedia program,´
Comput. Educ., vol. 50, no. 4, pp. 1370±1382, 2008.
[8] J. Pereira, ³Leveraging chatbots to improve self-guided learning
through conversational quizzes,´ in Proceedings of the Fourth
International Conference on Technological Ecosystems for
Enhancing Multiculturality - TEEM ¶16, 2016, pp. 911±918.
[9] J. Ahn et al., ³Wizard¶s apprentice: Cognitive suggestion support
for Wizard-of-Oz question answering,´ in 18th International
Conference on Artificial Intelligence in Education, 2017, pp.
630±635.
[10] W. Huang, K. F. Hew, and D. E. Gonda, ³Designing and
Evaluating Three Chatbot- Enhanced Activities for a Flipped
Graduate Course,´ Int. J. Mech. Eng. Robot. Res., vol. 8, no. 5,
pp. 813±818, 2019.
[11] D. E. Gonda, J. Luo, Y.-L. Wong, and C.-U. Lei, ³Evaluation of
developing educational chatbots based on the Seven Principles
for Good Teaching,´ in 2018 IEEE International Conference on
Teaching, Assessment, and Learning for Engineering (TALE),
2018, pp. 446±453.
[12] S. Sohoni, Y. Cho, and D. French, ³A survey to capture needs
assessment for graduate teaching assistant training,´ Adv. Eng.
Educ., vol. 3, no. 3, pp. 1±20, 2013.
[13] A. Chickering and Z. Gamson, ³Seven principles for good
practice in undergraduate education.,´ Am. Assoc. High. Educ.
Bull., vol. 39, no. 7, pp. 2±6, 1987.
[14] S. Nolan, ³Physical metaphorical modelling with LEGO as a
technology for collaborative personalised learning,´ in
Technology-Supported Environments for Personalized Learning:
Methods and Case Studies, J. O¶Donoghue, Ed. 2010, pp. 364±
385.
[15] M. D. Merrill, ³First principles of instruction,´ Educ. Technol.
Res. Dev., vol. 50, no. 3, pp. 43±59, 2002.
[16] M. McPherson and L. Bacow, ³Online higher education: Beyond
the hype cycle,´ J. Econ. Perspect., vol. 29, no. 4, pp. 135±154,
2015.
[17] T. Reeves, ³Enhancing the worth of instructional technology
research through µdesign experiments¶ and other development
research strategies,´ in Annual Meeting of the American
Educational Research Association, 2000, pp. 1±15.

Authorized licensed use limited to: Newcastle University. Downloaded on October 25,2020 at 23:16:53 UTC from IEEE Xplore. Restrictions apply.
 CHATBOTS IN EDUCATION
A passing trend or a valuable pedagogical tool?
Department of Informatics and Media
Uppsala University

Author: Sofie Roos

Supervisor: Ruth Lochan
Date: June 3, 2018
 Abstract
Digitalizing education and reinventing the learning experience is one of the
big challenges in this age of information. In the field of E-learning, the
application of a chatbot as part of the education has shown interesting
potential, both as a teaching and administrative tool. Chatbots have
been ’trending’ for a few years and quite a few papers examining it in
the educational sector have been published, albeit very little interest
seems to have been given to the summation of this knowledge. In an
attempt to fill the knowledge gap this thesis performed a literature study
to examine the documented features and possible uses for chatbots in
an educational context. Since quite a few chatbot technologies have
been developed at this time and exhibit varied functions, this study was
limited to only examine bots based on the XML derived language AIML.

The results imply that chatbots in education have quite a few uses
and even more possible features. An AIML-based chatbot can be both
simple and complex to implement, all depending on the effort put into imple-
mentation. The tool is diverse and may be used for many different purposes
and aims, the only limitation being the creators creativity and imagination.

Keywords: Chatbot, Artificial Intelligence Markup Language (AIML),

Artificial intelligence (AI), Education, Learning, E-learning

i
List of Tables

3.1 Literature-search terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

3.2 Literature-search string . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.3 Final Themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
3.4 Role of Bot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
3.5 Bot Features sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
3.6 Encompassing system sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.7 Bot Augmenting Tech sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . 18
3.8 Platform sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

4.1 Analyzed Papers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4.2 Paper 1: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.3 Paper 2: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
4.4 Paper 3: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.5 Paper 4: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4.6 Paper 5: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.7 Paper 6: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.8 Paper 7: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.9 Paper 8: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.10 Paper 9: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.11 Paper 10: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.12 Paper 11: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.13 Paper 12: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.14 Paper 13: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
4.15 Paper 14: Chatbot sub-themes . . . . . . . . . . . . . . . . . . . . . . . . . . . 41

7.1 Chatbot features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

ii
List of Figures

1.1 Simple AIML object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

1.2 Using srai for symbolic reduction in AIML object . . . . . . . . . . . . . . . . . 2

4.1 Keywords in the papers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

4.2 Abbreviated Keywords paper 11 and 14 . . . . . . . . . . . . . . . . . . . . . . 24
4.3 Sub-Themes for Role of Chatbot . . . . . . . . . . . . . . . . . . . . . . . . . . 26

iii
Contents

1 Introduction 1
1.1 Defining the chatbot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 AIML - The brains of the chatbot . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.3 Common uses of chatbots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Aim of the study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.5 Research question . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.6 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.7 Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.8 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Background 6
2.1 Origins of the AIML-based Chatbot . . . . . . . . . . . . . . . . . . . . . . . . . 6
2.1.1 Thinking machines and weak AI . . . . . . . . . . . . . . . . . . . . . . 6
2.1.2 ELIZA and ALICE: The foundation of pattern matching chatbots . . . . 7
2.2 A brief explanation of educational systems . . . . . . . . . . . . . . . . . . . . . 7
2.2.1 E-learning systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3 Methodology 9
3.1 Research approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2 Research strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
3.2.1 Structure of the literature review . . . . . . . . . . . . . . . . . . . . . . 10
3.3 Coding and Analysis of the Literature . . . . . . . . . . . . . . . . . . . . . . . 11
3.3.1 Defining thematic analysis in relation to the research . . . . . . . . . . . 11
3.3.2 Performing the coding and analysis . . . . . . . . . . . . . . . . . . . . . 13
3.4 Limitations of the method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

4 Results 19
4.1 The papers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.2 Publication Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2.1 Author(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.2.2 Institution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2.3 Time Published . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2.4 Keywords in paper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.2.5 Journal/conference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.3 Educational Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.3.1 Level of education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.3.2 Student demographics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.3.3 Subjects & Courses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.3.4 Type of education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.4 ICT implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

iv
 4.4.1 The themes in the ICT implementation details group . . . . . . . . . . . 26
4.4.2 The papers and their chatbots . . . . . . . . . . . . . . . . . . . . . . . 34

5 Analysis 42
5.1 Publication details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.1.1 Author(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.1.2 Institution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.1.3 Time published . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
5.1.4 Keywords in paper . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.1.5 Journal/conference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.2 Educational context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.2.1 Level of education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.2.2 Student demographics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.2.3 Subject & Courses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.2.4 Type of Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.3 ICT implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.3.1 Role of Chatbot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.3.2 Bot Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.3.3 Encompassing System . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.3.4 Bot Augmenting Tech . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.3.5 Platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

6 Discussion 46

7 Conclusion 48
7.1 Future research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

v
Chapter 1

Introduction

Chatbots are trending and they can now be found in almost every industry from e-commerce
to travel. The increased use of late may be due to improved language processing or the more
accessible development tools for non-developers. It may also be that many chatbots are made
available through mainstream messaging applications, thus not forcing the user to download yet
another application and allowing them to keep using an application they are already comfortable
with. (Jennifer Lachs, 2017). But what is a chatbot?

1.1 Defining the chatbot

A chatbot is a computer program that is designed to simulate a conversation with its users, often
over the Internet. Furthermore the analogy that a chatbot often treats a conversation like a
game of tennis can be used to describe the conversation flow of a chatbot, i.e. get message,
reply, get message, reply, and so on (The Oxford Dictionary, 2018). Deryugina (2010) provides
almost the same definition, but adds the word ’intelligent’ before communication, specifying
the need for intelligent replies rather than just random ones. The chatbot technology has been
referred to by many names, the more established ones include: Chatbot (can be spelled chat bot,
chat-bot as well), Chatterbot, Conversational Agent, Conversational System and Pedagogical
Agent (or Intelligent Pedagogical Agent, IPA). The last is exclusively used in educational settings
or educational papers. (Deryugina, 2010; Doering, Veletsianos, and Yerasimou, 2008; Heller and
Procter, 2009). Chatbots are build in many ways, but a popular and quite simple way is through
the use of AIML, which is presented in the next section.

1.2 AIML - The brains of the chatbot

Figure 1.1: Simple AIML object

Artificial Intelligence Markup Language or AIML was built to work as the brain of the chatbot

1
by Dr. Wallace and an open source community (the Alice foundation) (Pandorabots, 2018b;
Kerly, Hall, and Bull, 2007). It is a derivative of XML that AIML enables the botmaster (the
person who manages the chatbot) to add knowledge to the chatbot, with the AIML files as the
chatbots’ knowledge base or brain. These files consist of AIML objects (see figure 1.1), which
are made up of sets called topics and categories. A category is the basic unit of knowledge
in AIML, and each category consist of an input question, an output answer, and an optional
context. The question is the pattern and the answer the template. The language used in the
AIML pattern or template is simple, consisting of words, spaces and wildcard symbols like or *.
The words can be letters or numbers, but no other characters. There are two primary types of
optional contexts that can contribute more advanced functionality to the chatbot, <that> and
<topic>. The <that> tag is inside the category and its pattern always match what was last
said by the chatbot, something that is important when chaining conversations. The <topic> tag
is found outside the category, and can be used to group categories together. (Pandorabots, 2018a)

AIML use recursion in order to connect patterns that have synonymous meaning to each other,
this way the botmaster can streamline the production of categories. The tag <srai> is used
to achieve recursion and it has many uses, some of these are 1) Symbolic reduction: Reducing
more complex terms to their simpler equivalents, 2) Divide and Conquer: Split an input into
subparts, then combine the answers again, 3) Synonyms: Map synonyms of a phrase to the same
reply, 4) Spelling or grammar corrections, 5) Detecting keywords anywhere in the input, and 6)
Conditionals: Branching patterns. An example AIML implementing technique 1 is shown in figure
1.2. (Pandorabots, 2018a)

Figure 1.2: Using srai for symbolic reduction in AIML object

1.3 Common uses of chatbots

Initially chatbots were developed to be entertaining and to mimic human conversation. This is
still a popular reason when developing chatbots, but since the popularity with the technology
has gone up, so has the different uses. The chatbot technology has then been used for many
things, like for retrieving information, answering questions, helping make fact based decision, as a
shopping assistant, as a museum guide, language partner, or in education, among other things.
(Shawar and Atwell, 2007)

As mentioned, a chatbot can be used as an information retrieval tool, in order to answer a

users questions on a specific topic. This kind of chatbot has a wide range of applications, from
interactive FAQ to helping a customer make decisions. An example of a bot like this is the YPA, a
chatbot used to provide users with information from the British Telecom’s Yellow pages. The user
might ask for a plumber with an emergency service, and the chatbot will search its knowledge
base and retrieve an adress, or if none was found, ask for additional details. (Shawar and Atwell,
2007). Of course this type of chatbot has applications in education as well, for example a chatbot

2
named ALEX, who can help students wanting to find basic legal information. (Deryugina, 2010)

The chatbot can be used as a tool to learn and practice a language. There are many perks of
having a chatbot as a conversational partner when learning language, for example a it could allow
a student to practice their language skills anytime, it would not mind repeating the same material
several times, it can incorporate both text, speech which might help a student with both their
reading, writing and listening skills. However, the chatbot as a language tool also has a few
disadvantages. The responses of a chatbot is often predictable, redundant or retain no memory of
previous responses. Most chatbots are also bad at detecting spelling or grammatical mistakes.
But the chatbot could still be useful for non-beginner language learners, and besides being used
as a teacher or learning partner with rich knowledge, it may be a friend who listens to the woes
of the user. (Shawar and Atwell, 2007)

When it comes to chatbots in education, Shawar and Atwell (2007) states that it is important to
remember that the teacher is the backbone of the teaching process and the learning technology
may act as an amplifier but not a replacement. One example is that the when a chatbot is
used to answer student questions, the teacher can use generated log files of the conversations
to see what the students are having issues with and what their weaknesses are. Thus, the
teacher can use a chatbot to look for problems as the students use it to solve them. (Shawar
and Atwell, 2007). Unfortunately not many chatbots can be found that are actually in contin-
uous use in education, a reason for this may be that their capabilities are still quite undocumented.

1.4 Aim of the study

Continuing with the problem introduced in the previous section that chatbots are still not com-
monly implemented in education, the aim of this thesis is set in context. In order to further the
development of educational chatbots it is important to find out what has been done, and to
summarize this knowledge in a concise and relevant manner. No such summarizing study was
found, thus, the aim of this study was to produce that summary to find out what pedagogical
uses and capabilities a chatbot has in an educational context by reviewing the literature in the
field, this means looking at what the chatbot can do and be used for in an educational context.
In this circumstance it is also relevant to look at if the chatbot can be useful on its own or if
additional technology is needed, such as any system the chatbot needs to be encompassed in or
that enhance its capabilities, e.g. e-learning system or text-to-speech technology. Hence, the work
in this study can be used as a stepping stone for coming researchers who might more efficiently
uncover further potential of the technology. This aim was developed with the focus on what
might be needed for teachers and researchers when continuing to develop the chatbot technology.
Of course there may be more that one thing that is important, for example which type of chatbot
to use, how to adapt it properly to a course, what kind of architecture the chatbot system might
need, how to achieve the greatest pedagogical value and so on. But the question of what role a
chatbot could take and what can it actually do in the context of education seemed more acutely
relevant, since its answer is a good starting point to answering other questions.

1.5 Research question

In order to fulfill the aim, two specific questions were developed. In this thesis the questions are
narrowed even more by limiting it them to only AIML-based chatbots. This decision was taken on

3
the base of the simple use of AIML and the open source nature of the system, making it a good
base for continued development. The question posed in the aim was divided into two parts for
clarity and they are thus:

• What roles can AIML-based chatbots take in an educational context?

• What features can AIML-based chatbots have in an educational context?

By answering these questions the knowledge gap regarding what has been done with chatbots in
education, and what capabilities the technology has shown will be filled.

1.6 Limitations
The limitations to this study are mainly concerned with the databases searched and the papers
chosen, these were Elsevier (Science Direct), ProQuest (ABI/INFORM), IEEE (Xplore) and ACM
(Digital Lib). The motivation of the choice for databases 3. Within these databases only papers
published in Journals or conferences were chosen. Considering the content of the papers, only the
ones concerning an educational use of an AIML based chatbot have been reviewed, to stay in
line with what the research question seeks to answer. This study was performed as a qualitative
literature review, which means a limitation to the result is always the researchers bias, specifically
when analyzing the results.

1.7 Ethics
Apart from correctly citing the reviewed works no ethical considerations was taken, considering all
the reviewed works was published and thus are already ethically scrutinized by the authors and
their peers.

1.8 Outline
Chapter 2 - Background
The background chapter addresses some important grounding knowledge that is needed for the
scope of this thesis. The first part addresses the definition of a chatbot, some relevant history,
examples of famous chatbots and their structure and some common uses of chatbots today. The
second part briefly explains e-learning systems and their relevance to education.

Chapter 3 - Methodology
An important piece in the thesis puzzle is the methodology, it is the frame the thesis was built on.
The work was conducted as a qualitative integrative literature review, with a thematic analysis
focused on finding themes in the different papers and summarizing the gathered knowledge into a
concise conclusion. These choices are motivated and the process explained in the methodology
chapter.

4
Chapter 4 - Results
The chapter presents the results of the literature review. Initially the papers that were analyses
are introduced. Then the found themes and sub-themes are described and validated with quotes
form the data, both in the order they were initially designed in within the methodology chapter
and in relation to each reviewed paper.

Chapter 5 - Analysis
The results are analyzed in this chapter in the same order they were presented in the results
chapter. Any interesting details of the data are examined and analyzed.

Chapter 6 - Discussion
The results and analysis are discussed and elaborated on.

Chapter 7 - Conclusion
A conclusion is made from the results, the analysis and the discussion and the thesis ends with
suggested further research.

5
Chapter 2

Background

The background contains two parts. The first part addresses the origins of the chatbot, present-
ing some relevant history with examples of famous chatbots. The second part briefly explains
e-learning systems and their relevance to education.

2.1 Origins of the AIML-based Chatbot

2.1.1 Thinking machines and weak AI
In 1950 Turing asked himself the groundbreaking question “Can machines think?”. He then
defined the now famous Imitation game (or Turing test), the game where the computer is supposed
to convince a human interrogator that the computer is in fact also human (Turing, 1950). Turing
questions his own reasoning and after defining both his meaning of a machine, and of the concept
of thinking, he goes on to revise the initial question to instead read: “Are there imaginable digital
computers which would do well in the imitation game?”. The imitation game is thus used as the
scale on which the machines ability to think would be measured.

No machine or digital computer has yet been able to beat Turing’s imitation game, and even the
best contestants seem to still be far off. As a way of furthering the development of intelligent
machines Hugh Loebner came up with a limited version of the Turing test in 1990, called the
Loebner Prize Competition. This is considered the first formal instantiation of a Turing test and
it is carried out annually. In the contest 4 judges take turn interacting with two dialogue partners
in a chat system. The interaction is limited to a total of 25 minutes and at the end the judge is
asked to declare which partner is the machine. If a program were to fool half the judges that it
is human a Silver Medal and $25,000 would be awarded as a prize. If the judges managed to
identify the machine, they were asked to score its “thinking” capabilities between 0 and 100 and
cash prizes are awarded the top 4 AI’s. The Grand Prize of $100,000 and the Gold Medal will be
awarded the program who can provide answers that are indistinguishable from a human, which
according to Turing’s definition would then be proof of a thinking machine. (Loebner.net, 2017;
Loebner, H., 1994; Floridi, Taddeo, and Turilli, 2009)

Turing’s ideas had a huge impact on the technological development in the field we today call
Artificial intelligence, or AI for short. AI is a branch of computer technology dedicated to making
digital computers of computer-controlled robots do tasks that are generally identified as requiring
intelligence (or thinking) like that of a human brain (Copeland, B.J, 2018). But like all things
revolutionary his ideas also received a lot of criticism.

6
One of Turing’s bigger critics was Searle (1980) who, based on two propositions and three conse-
quences of these propositions, argues that ”only a machine could think, and only very special kinds
of machines, namely brains and machines with internal causal power equivalent to those of brains.”.
With this argument in mind Searle (1980) wanted to differentiate between what he identified as two
types of AI, strong AI and weak AI. Strong AI the definition of an actual thinking machine, i.e. a
system that possesses these causal brain powers that Searle argued for. Weak AI, on the other hand,
is a system lacking the causal brain powers, thus not an actual thinking machine but rather a ma-
chine making a plausible imitation of human intellectual behavior. (Searle, 1980; Deryugina, 2010)

According to the previously given definition of a chatbot, and after presenting the concept of
weak AI from Searle (1980) a conclusion can be drawn. The chatbot technology fall under the
category of Weak AI (Deryugina, 2010). A chatbot is not an actual thinking machine, but rather
a machine feigning intelligence after imitating human conversation patterns. Non the less, it is
still considered AI and it has a lot of valuable and popular uses. Two such popular chatbots from
history are presented in the next section, the ELIZA and ALICE chatbots.

2.1.2 ELIZA and ALICE: The foundation of pattern matching chatbots

One of the first and most famous chatbots was ELIZA developed by Weizenbaum (1966). He
published an explanation of ELIZA and explains it as a computer program which enables a natural
language conversation between human and computer. ELIZA is programmed to imitate a Rogerian
pshychotherapist and works by identifying keywords and then mapping them to an associated
rule which transforms the input sentence and then outputs a response, this is called pattern
matching. If no keyword was found a content-free remark or an earlier rule transformation is
retrieved and printed (Weizenbaum, 1966). Since this was the first program of its kind it made
a great impression on specialists from the AI-field but also on non-technical people (Deryugina,
2010). One of the claimed reasons for Eliza’s success was the fact that she was developed to try
and replicate the conversation between a therapist and a patient, the success stemming from the
fact that the job of the therapist consists mainly of asking questions and listening to the answer.
This might be the case, but it only means that it is important to be aware of the limitations of
the technology and use it in a way that enhances its capabilities instead.(Weizenbaum, 1966).

2.2 A brief explanation of educational systems

E-learning
E-learning is simply learning making use of digital resources in order to get/give access to an
educational curriculum outside the classroom or to support students learning (Travis, 2008).
E-learning can be performed through many means, but a few common systems used for e-learning
will be explained in the following paragraphs.

2.2.1 E-learning systems

LMS
LMS is an abbreviation for Learning Management System which is a type of software often used
to launch and track online learning activities. Most commonly the teacher will upload resources
and thus make them accessible to learners remotely. A LMS is, in its core, a repository to store
and track information, most often related to learning of some sort (Pappas, 2018). Its aim is to

7
offer management functionality to its users (Mikic, Burguillo, Llamas, Rodrı́guez, and Rodrı́guez,
2009) who usually fall in to one of two categories, 1) Online learners who use the LMS to join
online training courses, and 2) an e-learning team who uses the system to distribute information
and training content. Having a LMS is very beneficial to educational institutions, the five main
benefits being: 1) It can organize and safely store big data. 2) It can monitor learner progress
and performance. 3) It improves resource allocation. 4) It can personalize the online learning
experience. 5) It improves accessibility. (Pappas, 2018)

LCMS
LCMS is an abbreviation for Learning Content Management System. It is used to create and
manage the contents of an educational program, typically a course. A great LCMS provides its
user with the tools to easily create content, add different types of learning material, store contents
in a repository, reuse and export contents (even to different systems, and, manage and maintain
tasks over the content. It is often confused with the LMS but it provides another dept in its tools.
The two are however often combined to give users the LMS’s management abilities as well as the
content creation abilities of the LCMS. (Mikic et al., 2009)

8
Chapter 3

Methodology

In this chapter the chosen methodological approach is presented and motivated along with the
different steps that are necessary to perform during the approach.

3.1 Research approach

Since the aim of the thesis was to find the capabilities of an AIML-based chatbot in an educational
context the approach to the entire research was a qualitative one. A qualitative approach is
preferable when examining everything from peoples lives, life experiences, behaviors, emotions
and feelings to organizational functions, social movements, cultural phenomena and interactions
between nations (Strauss and Corbin, 1998). The thesis examines the current literature on the
subject, trying to identify the capabilities, pros and cons of the chatbot implementation in an
educational context. This might be considered examining cultural phenomena, as the chatbots
are designed and implemented within certain cultures. This fact was the basis of the choice of
a qualitative approach and it was in turn reflected on the chosen data gathering and analysis
methods.

3.2 Research strategy

The same reason that dictated the use of a qualitative approach also dictated that the strategy
used was an exploratory one. The research questions reflect this choice as well, being more or
less open ended, without claiming a hypothesis or leaning towards any predetermined probable
conclusion. Since the aim was to look at what the current literature is saying about the capabilities
of chatbots the natural step was to perform a literature review. The literature review was not
focused on setting the context for the thesis or justifying the study as is, two goals Levy and Ellis
(2006) lists for literature review, instead it examined specifically what has been written about
using chatbots in education.

The type of literature review that was performed was an integrative review. This type of review
looks at the representative literature of the topic to review and synthesize it in a concise way.
The goal of an integrative review is generally to be generate new frameworks of perspectives on
the topics, This fit well with the aim of this study as it aimed to bring forth the perspectives of
the literature and review them. The literature set in an integrative review should include studies
related to the research question, in this case, literature about AIML-based chatbots in educational
settings. University of Southern California (2018) writes that: “A well-done integrative review
meets the same standards as primary research in regard to clarity, rigor, and replication. This is

9
the most common form of review in the social sciences.”

3.2.1 Structure of the literature review

The literature review was executed in three steps, the first was a number of ’wild searches’ using
the Google search engine and the second slightly more structured focused on academic databases
and the third and last a very structured search using databases containing both educational
research and IT-research.

The initial ’wild search’ had the aim of providing the researcher with the basic knowledge about
chatbots and some current trends regarding them, and subsequently to provide relevant search
terms to be used in the following steps, mainly in regards to what terms were commonly used to
denote a chatbot. The second search followed up on this wild theme but added slightly more
structure, using the found terms to figure out which where relevant to this research and what
research had actually been done. It resulted both in literature to build the theoretic background
with, by describing chatbots, and it confirmed the found terms as relevant (see table 3.1). This
search also highlighted that the area of research is quite young, i.e. a lot of articles may not
have been published in journals yet, but only at conferences. A decision to include both papers
published in journals and at conferences was therefore taken, to be able to analyze as much
relevant data as possible. These two initial searches resulted in the search strategy used to make
the third search as good as possible.

Main search strategy

After having identified key terms for the research area the next step in the search process was
to identify databases containing key journals/conferences. The work of Levy and Ellis (2006)
was utilized to find databases that cover as many of the 50 top-ranking journals in the field of
informatics, leading to the choice of Elsevier (Science Direct), ProQuest (ABI/INFORM), IEEE
(Xplore) and ACM (Digital Lib) thus covering approximately 80% of the list in Levy and Ellis (2006).

The identified search terms are listed in table 3.1. The terms in the table were combined into
one complex search string using logical operators AND and OR to delimit (see table 3.2). This
was used to run searches on the chosen databases. Other limitations applied to the search was
that the papers should be peer-reviewed (if this option was available on the chosen database) and
published after 1995 in either a journal or as a conference proceedings. The final sum from all 4
databases was 72 papers. The year 1995 was used since this is the year Dr. Wallace first published
ALICE and was thus the year AIML was invented. All of the found papers where exported into
EndNote reference system which could then be used to store and organize the papers. The next
step was to start reviewing the articles and removing those not deemed relevant for the scope
of the research defined in the introductory chapter. This was carried out by first reading only
the titles of articles and removing those clearly irrelevant right away. Secondly the abstracts was
reviewed and, if needed, the extended content of the papers as well. Then the same process was
repeated again, removing those considered off scope. Eventually this resulted in 14 remaining
papers.

Two criteria were used to determine if a paper was in the scope and they were both a direct
result of the research question. They were 1) The chatbot should be AIML based - If the
paper didn’t report what technology the bot was based on or it reported a different technol-
ogy, it was removed. 2) The chatbot should have been used or evaluated in an educational context.

10
 Concept 1 Concept 2 Concept 3
Chatbot* Education AIML
Chat-bot Pedagogy Alice
Conversational agent - Artificial intelligence markup language
Conversational system - Pandorabots
Pedagogical agent - -
Chatterbot* - -

Table 3.1: Literature-search terms

Search string
((Chatbot* OR ”Conversational agent” OR ”Conversational system”
OR ”Pedagogical agent” OR Chatterbot* OR Chat-bot*)
AND (Education OR Pedagogy) AND (AIML OR ALICE OR
”Artificial intelligence markup language” OR Pandorabots))

Table 3.2: Literature-search string

3.3 Coding and Analysis of the Literature

When coding and analyzing the found papers thematic analysis was used as a way to organize
and describe the data in a detailed way (Braun and Clarke, 2006). Thematic analysis, its purpose
and how it is done is explained briefly below, along with how it was applied in this particular
research case. After that the actual process of the coding analysis is described and the found
themes presented.

3.3.1 Defining thematic analysis in relation to the research

The purpose of thematic analysis is to identify, analyze and report themes within the data.
Thematic analysis is not the only method that strives to find themes in qualitative data, however
according to Braun and Clarke (2006) there are three main benefits to thematic analysis in
comparison to other methods. 1) Researchers using thematic analysis does not have to take on
the theoretical undertaking of grounded theory, something that is appropriate for the current
research, since grounded theory is not the goal. 2) It can offer a more accessible form of analysis,
in particular to a beginner researcher. 3) Thematic analysis is versatile enough to allow the
researcher to both reflect reality and to look beneath the surface of reality. In relation to point
three’s dual nature Braun and Clarke (2006) states that it is important to make the position
of a thematic analysis clear and transparent. The next few paragraphs tries to do just that by
considering the decisions Braun and Clarke (2006) find important and then motivating the made
choices in relation to the research goal.

11
The focus is on the details
Braun and Clarke (2006) lists two ways of conducting the analysis, 1) producing a rich narrative
of the data set, or 2) making a detailed account of one particular aspect. The first way can lend
the reader a sense of what the most important themes in the data set are and this demands that
the identified themes must be an holistic reflection of the whole data set. This in turn, inevitably
leads to a loss of detail and depth. The second way however may provide this precise and nuanced
rendering of one specific theme or group of themes, which of course is done at the cost of the
broader narrative. (Braun and Clarke, 2006)

Since the research question at hand was quite narrow, looking specifically to find out what has
been done with chatbots in education the choice here was quite straightforward. Like the headline
hints this research has gone for the details specific to the research question and thus the themes
and their creation followed the same path. In reality this means that, at every introduction of
a code, the question: Does this relate to the chatbot that the paper describes? If the answer
was no, the detail wasn’t coded at all, or else, it was coded with its relation to the chatbot in mind.

Identifying themes in the data

There are two main ways of finding the themes in the data, the inductive (bottom up) way or
the deductive (top down) way. The purpose of inductive analysis is to code the data without
trying to make it fit in any pre-determined frame of codes, deductive analysis on the other hand is
solely driven by the pre-determined interest of the research. This translates to coding either for a
specific research question to begin with (deductive approach) or that the coding process is a part
of the evolution of a specific research question (inductive approach). (Braun and Clarke, 2006).

In regard to the exploratory nature of this research the way of finding themes would always contain
a certain amount of exploration. It was also analyzed with the specific research question in mind,
but the specific formulation of the question was largely influenced by the coding process. Because
of this duality in the nature of the study, a sort of mix between the deductive and the inductive
analysis was performed. The deductive approach was the main part, used to find initial themes
which were independent of the actual data, in order to make sure all codes of initial interest were
identified. This resulted in overarching theme groups which incorporated several more detailed
themes illustrating different aspects of the analysis that were important. The inductive approach
came in to play when the coding of the themes started and new sub-themes and codes emerged
with this exploration. Since the focus was still on the details of the chatbot, all new themes could
be comfortably sorted in to one of the identified overarching theme groups and thus keep the
initial top down structure.

The level of the themes

This decision centered on the question of which level the themes should be identified at, this could
be either at the semantic level or at a latent level. If the themes are at a semantic level they are
found at the surface level of the data (explicit). The analytic process of semantic themes often
include going from a description, with the data organized to display the themes in a semantic
context and to then summarize the themes interpretively in an attempt to give them broader
meaning. In contrast, if the themes are at a latent level they go beyond the explicit meaning of
the data and strive to find the underlying ideas or assumptions that shape the semantic data
content. The interpretation is performed already at the identification of the the themes and the

12
subsequent analysis is not just a description but in fact already theorized. (Braun and Clarke, 2006).

Since this research was all about looking at other researchers work in a straightforward manner
and then to retell and summarize what was found, the themes are exclusively on the semantic
level, thus the process of analysis started with a description displaying the found themes in their
semantic context and then interprets them to give a broader meaning.

What is a theme?
If something in the data can be considered important relative to the research question it can be
considered a theme. A theme often show some level of patterned meaning within the data set.
According to Braun and Clarke (2006) there are not hard-and-fast rules to how encompassing or
grounded a theme needs to be in the data set to be considered valid. Thus the decision of what a
valid theme comprises depends a lot on the earlier analysis decisions.

Leaning on the previously explained decisions of the analysis, the definition of a theme in this
research is that a theme is somehow related to the precise research question and aim, so as to
stay in the details of the data. A theme that is related to the research interest can therefore be
sorted into one of the theme groups. Further, a theme is semantic, no underlying meaning is
interpreted at this stage. And lastly due to the exploratory nature, a theme doesn’t have to be
grounded in the entire set to be of interest, rather any singularity is an interesting find, as long as
it can still be related to the overarching groups.

3.3.2 Performing the coding and analysis

With the above analysis decisions in mind, the actual coding began. This work iterated over the
6 phases of thematic analysis listed by Braun and Clarke (2006), which were used in a recursive
fashion to achieve a thorough and reliable result. The steps are: 1) Getting to know the data, 2)
Setting up initial codes, 3) Looking for themes, 4) Reviewing the themes, 5) Establish and name
themes, and 6) Generate the report. How these steps were performed is outlined below, however,
due to the fact that the analysis was performed top-down, the steps were not done in the order
Braun and Clarke (2006) listed, and will thus be presented in the order they were used below.
Step 6 was omitted since that was the actual production of the report. The final theme groups
and sub-themes with accompanied descriptions can be found in table 3.3.

Looking for themes

As mentioned, due to the top down, deductive, nature of the analysis, the order of the steps
was shuffled to better fit the needed process. The analysis thus started with drawing out three
overarching themes that were important for the research aim and question. These were developed
before the initial read through (phase 1), to avoid getting sidetracked by details in the data and
were thus developed with the background information and the research question as the main
focus. The three themes where comprised of the details about the publications themselves, details
about the educational context described, and details about the ICT specifications, in this case,
the chatbot and its environment. The themes are described more closely in step 3 but rele-
vant here since they dictated what codes were selected, due to the top-down nature of the analysis.

13
 Theme Group Sub-Theme Description
Author(s) Who did the research?
Institution Origins of the authors
Publication details
Time published When was is published?
Journal/Conference Where was it published?
Keywords in Paper The authors’ chosen keywords
Level of education University, High School, etc.
Student demographics Age, gender, nationality, etc.
Educational Context
Subjects & Courses Course specific or general etc.
Type of education Course given on distance, campus or blended?
Role of chatbot What will the bot be used for?
Bot Features What can the bot do?
ICT Specifications
Encompassing system Is it stand-alone or in a larger system?
Bot Augmenting Tech Addition of technology to enhance bot
Platform Implemented in Web or some OS?

Table 3.3: Final Themes

The encompassing themes are both wide and narrow, in the way that they kept the codes inside
the scope of the research, but at the same time allow for exploration with discovered codes. The
themes where then used as the theoretical frame from which to locate interesting codes. Since this
research used published literature as its data it was important to code the details of the papers,
such as the authors, where and when the paper was published, the location of the authors and the
keywords used. The purpose of this was to be able to identify any patterns in this meta-data in
relation to itself or the content patterns. As for the educational details the focus was now to look
at the actual data, searching for the details about the educational context in which the chatbot
was developed or tested. Lastly the details about the ICT, maybe the most critical group since
this is where the actual answers to the research question could be found. This theme included all
the technical details surrounding the chatbot, like the purpose of the bot, what features were
implemented, if it was integrated into a larger LMS or used as a stand-alone application and so on.

Some initial sub-themes were also generated within the scope of the three overarching themes. The
publication theme included 5 sub-themes, the education theme 7 sub-themes and the ICT theme
4 sub-themes. These were later revised slightly to accommodate for the new knowledge gained
about the data during the next step and also when the actual coding started it naturally revised the
themes as well to better fit both the data and the theory. The revised themes are shown in table 3.3

Getting familiarized with the data

The purpose of this phase was, like the title entails, to get to know the data. This was done
through careful reading of all the data, several times, and noting interesting details and initial
patterns. Sometimes this phase includes transcribing the data, but this was not necessary in
this research since the papers were already in a good format. In this research all of the papers
where initially read straight through without a lot of thoughts or notes, to get a first idea of
the content and to be able to create an initial set of codes (phase 1 and 2 were very mixed).

14
The second read notes were taken on the data and an even better feel for the content was procured.

Setting up initial codes

When developing initial codes the analysis decisions was kept in mind, primarily the definition of
what a code is and on what level of meaning if should be. The code should always represent a
feature of the data that is interesting to the researcher (Braun and Clarke, 2006). In this research
the focus was to make a deep-dive in the details of the data, but to stay in the semantic meaning,
the codes should also be contained in the scope of one of three the overarching themes.

Once this step of coding was reached, it consisted of the initial themes or sub-themes which had
been coded in the data. For example, the authors of a paper was located and coded as Author.
This straightforward way of coding was done for all initial sub-themes, with the intent of later
working through these themes to search for patterns of more detailed codes. For several of the
themes a deeper specification of code was deemed unnecessary due to the fact that no specific
pattern could be found, thus the coding could be kept at the general level of the theme for the
most part. One example of this is the Subjects or courses theme, which were supposed to find
what subjects the chatbots had been tried in. This theme yielded such a varied result it was
necessary to keep it at the general level, as to not lose the variety, the same was true for all other
themes in the groups of Publication details and Educational context.

In the theme group of ICT specifications however, a more detailed view of some the themes was
identified and coded. For example, Role of bot, was deemed a way too diverse group and thus
needed more detail. This detail was initially just codes, to be refined into new sub-themes in the
following steps. The data was the key part in all these decisions, if a theme contained obvious
importance and support after the initial coding, it was reviewed again and given more detail with
generated codes.

Reviewing the themes

This step started with the initial themes and the generated codes and sought to refine those even
more. During this process the generated codes were streamlined into the themes, and in a few
cases, they helped identify themes missing from the initial core and were thus added instead.
This step mainly involved refining the codes generated as details of the themes, for example, the
codes of the theme Role of Bot were reviewed and could eventually be divided into 7 sub-themes.
The same process was repeated with the other themes of the ICT group as well. Braun and
Clarke (2006) writes that the step involves looking for patterns in the coded data extracts and to
also make sure that the themes fit the data, or else, revise or extract the ill-fitting codes into
another theme. The step is done when the thematic map of the data is satisfactory, and when
the researcher have a good idea about what the themes are and how they fit together. (Braun
and Clarke, 2006)

Establish and name themes

At this point in the analysis it was time to define and finalize the themes and then to analyze the
data within them. This was done by first trying to find the ’essence’ of each theme by itself and
the themes overall. In this stage a lot more sub-themes were defined (like the ones in figure 4.3,
which according to Braun and Clarke (2006) can be used in order to give a better structure to large

15
and complex themes. A theme still needs to be concise and clear, and (Braun and Clarke, 2006, p.
92) describes it with the following quote: “It is important that by the end of this phase you can
clearly define what your themes are and what they are not. One test for this is to see whether you
can describe the scope and content of each theme in a couple of sentences.”. This description is
presented in a very short version in the theme table at the start of the section (table 3.3) and
a longer description of each theme and its containing data will be presented in the results chapter.

The original idea with the initial themes were to be able to establish sub-themes within all of
them, but for the themes in the groups Publication details and Educational context the data was
so dispersed no particular patterns could be identified, thus no further coding was possible or
even necessary. The themes on those groups was thus kept as presented in the initial theme table
(3.3). In the group of ICT implementation details however, 5 out of the 6 themes were divided
into sub-themes. Not all sub-themes have equal grounding in the data, but every theme illustrate
a point of information in the data that was interesting to the research question. The example of
the ’Role of bot’ theme has already been illustrated as having 5 sub-themes which all describe a
role the chatbot has played in the research, see table 3.4 as well. Additionally, sub-themes were
identified in the other themes: ’Bot Features’ have 18 sub-themes (table 3.5); ’Encompassing
system’ have 4 sub-themes (table 3.6); ’Bot augmenting Tech’ have 3 sub-themes (table 3.7);
and ’Platform’ have 5 sub-themes (table 3.8). The Access to bot theme did not include any
sub-themes. The themes with their sub-themes are presented in the respective tables referenced.

Role of Bot
What will the bot be used for?
Sub-Theme Description
Tutor Provide individual didactic support
Student evaluation Evaluate student progress or performance
Questions & Answers Answer students topic specific questions
Teacher communication Medium for student to communicate with teacher
Natural conversation Chatbot is a conversational partner

Table 3.4: Role of Bot sub-themes

3.4 Limitations of the method

The method used when conducting this work was an integrative literature review, followed by a
thematic analysis. The main points of delimitation is the choice of databases and search strategy,
and the theme choices made within the analysis.

When gathering the data for the literature review 4 databases were selected based on their coverage
of popular journals and conferences in the field of informatics. Combined with a quite limited
search strategy only 14 papers were gathered for review, a set that is quite small. The reason for
these severe delimitations were the time limitations set on the thesis work, as finding and reading
papers as well as performing the analysis were very time consuming tasks, even with the small set.
The advantage of having this small dataset is that the papers could be more thoroughly analyzed,
most likely leading to better results.

16
 Bot Features
What can the bot do?
Sub-Theme Description (The chatbot can ... )
Demonstrate learning task demonstrate a task
Animated gestures supporting knowledge base use animated gestures
Allow input as speech accept input in speech format
Control e-learning platform’s activity monitor activity on platform
Provide means of contact with teacher contact teacher for learner
Point of interaction for learner act as a point of interaction
Provide output as speech provide spoken output
Help students 24/7 provide its services anytime of day and year
Provide content from other knowledge sources use other knowledge sources for its answers
Support feedback to students provide feedback to students or teachers
Give recommendations about learning material give recommendations about learning material
Hold topic specific conversation conduct a topic specific conversation
Hold general conversation conduct a general conversation
Teacher can edit bot content be edited by the teacher
Ask the student questions ask the student questions
Conversational strategies use conversational strategies
Answer student questions answer the students questions

Table 3.5: Bot Features sub-themes

By choosing initial themes to work with rather than keeping the analysis process completely
inductive the results were limited. This was a known fact taken into account while making the
decision, as the research question, even though open, is quite specific and targeted. However it is
an important fact to remember, as a researcher looking at the exact same papers but in a fully
inductive way will most likely discover other details in them.

17
 Encompassing system
Is it stand-alone or in a larger system?
Sub-Theme Description
Library system Chatbot is included in a library system
Virtual environment Chatbot is used in a virtual environment
E-learning system Chatbot is included in an E-learning system
Stand-alone Chatbot is not encompassed in any system

Table 3.6: Encompassing system sub-themes

Bot Augmenting Tech

Addition of technology to enhance bot
Sub-Theme Description
Linguistic tools Tools to enhance the chatbots language capabilities
Animation Chatbot was embodied and animated
Text-To-Speech Speech as input or output was implemented

Table 3.7: Bot Augmenting Tech sub-themes

Platform
Implemented in Web or some OS?
Sub-Theme Description
Web Chatbot was published on a Web platform
Mobile OS Chatbot was published on some mobile platform
Not-specified The paper did not specify a platform

Table 3.8: Platform sub-themes

18
Chapter 4

Results

This chapter presents the results of the literature review. Initially the papers reviewed are described

4.1 The papers

As mentioned in chapter 3, 14 papers were found in the search. These 14 papers where published
in different journals or conference proceedings and apart from 3 papers they were written by
different authors. The authors originated from different parts of the world and also from different
universities. The papers are presented in the table below, their authors, the paper title (sometimes
shortened slightly) and a short summary of the paper. The full references can be found in the
reference list.

Table 4.1: Analyzed Papers

Id Paper
Author(s) Orlando and Giovanni (2008)
Title An integrated system to monitor activities in e-learning environments
Summary The paper describes an integrated system with the purpose of monitoring and
supporting an e-learning systems activities and discovering the students issues.
1
The system alerts the teacher with a phone call if a learner poses a question that
cannot be answered, the question has then been vocalized using text-to-speech
technology. An AIML-based chatbot works as the front-end of the application
and is the instance the learner interacts with.
Author(s) Doering, Veletsianos, and Yerasimou (2008)
Title Conversational Agents and Their Longitudinal Affordances on Communication
Summary The study investigated the effect of conversational agents on communication
and interaction when used to assist participants in developing an online portfolio.
Data gathered during four weeks revealed that participants talked to the agents
2 on a range of subjects, not only related to the domain of e-portfolio. The
participants didn’t find the agents very helpful in completing class activities,
but used them as social companions. The researchers conclude that further
research is needed on three points: 1) Learner-developed conversational agents,
2) Improve the agents “intelligence” regarding the domain knowledge, and 3)
Agents that better meed users’ humanistic and utilitarian. expectations.

19
 Table 4.1: (continued)

Id Paper
Author(s) Mikic, Burguillo, Rodrı́guez, Rodrı́guez, and Llamas (2008)
Title T-BOT and Q-BOT: Two Bots for Tutoring Courses and Evaluating Students
Summary The authors have developed two AIML-based bots with the purpose of helping
students during the learning process and supporting teaching activities on an
e-learning platform like Claroline or Moodle. One bot has the role of a tutor
3
(T-bot) and communicates with the learner in natural language to give them
adequate and domain specific answers and guiding students to the right course
material. The other bot had the role of an evaluator (Q-bot) and can track and
supervise the student through personalized questionnaires. Both bots have been
developed as easily integrated modules for Claroline or Moodle.
Author(s) Heller and Procter (2009)
Title Animated pedagogical agents: Historical figure application
Summary The authors perform an experiment to explore how an historical figure applica-
tion of a pedagogical agent affected student engagement and motivation. 88
4
participants where randomly assigned to one of three groups; a no image agent,
a static image agent and an animated image agent. The results contradicted the
expectations and previous research on animated pedagogical agents in that the
no image agent received higher ratings for half the measures.
Author(s) Mikic, Burguillo, Llamas, Rodrı́guez, and Rodrı́guez (2009)
Title CHARLIE: An AIML-based Chatterbot as an Interface in INES
Summary INES (INtelligent Educational System) is an e-learning platform which comprises
a set of different technologies. One of these technologies is a chatbot which
communicates with the students in natural language and act as the “brain”
5
of the system. The paper focuses on the description of this chatbot called
CHARLIE (CHAtteR Learning Interface Entity). CHARLIE can communicate
with students in natural language and answer general or domain specific questions.
The student can also request questionnaires or free questions from the bot to
test their knowledge.
Author(s) Ibáñez, Rueda, Galán, Maroto, Morillo, and Kloos (2011)
Title A 3D Multi-User Virtual World for Language Learning
Summary The paper describes a 3D multi-user virtual world for language learning. It was
developed to test if it it would create an engaging learning experience and thus
foster communication skills with minimal teacher involvement. The open-source
6
development toolkit Open Wonderland was used as a base and then extended
to provide natural language conversation with Non-Player Character chatbots,
textual tagging of virtual objects, automatic reading of texts in learning sequences
and the arrangement of learning activities meant to promote collaboration. The
authors preliminary evaluation of the experience shows great promise.
Author(s) Rossi and Carletti (2011)
Title MAPIT: a pedagogical-relational ITS
Summary The researchers describe an open source Learning management system containing
a chatbot. The system is aimed at supporting online teachers work by implement-
7 ing a knowledge base specific to pedagogical aspects and not a specific subject.
The chatbot developed was based on the ALICE chatbot and should retrieve
information already coded in the courses or originated in student activity logs and
then answer the students most common questions in order to relieve teachers
from the task giving them time to answer more difficult questions better.

20
 Table 4.1: (continued)

Id Paper
Author(s) Allison (2012)
Title Chatbots in the library: is it time?
Summary The paper describes a chatbot pilot “Pixel” developed to answer questions
about the library and its resources at the University of Nebraska-Lincoln. The
chatbot could answer questions from a variety of users from around the world
and, surprisingly for the author, attracted a lot of social chatter. The majority
8
of questions where of a factual nature which the Pixel could handle and the
database was proved easily revised as library resources or personnel changed.
The chatbot provided a consistent service 24/7 and also a playful interface
which engaged users, relieving them from complicated navigation systems and
endless scrolling through search results. At the same time it could refer the more
complicated questions to the librarians.
Author(s) Masmuzidin and Wan (2012)
Title v-Penglipur Lara: A Pedagogical Agent in Malaysian Folktales Land
Summary Pedagogical agents have been praised as a tool to enhance learning and teaching,
only the authors identified a limited application with children as participants. The
9 research then investigate if a pedagogical agent can enhance children’s learning
as well. They developed an agent set in a virtual learning world called Hikayat
Land, with the purpose of teaching children Malaysian folktales and moral values.
The agent has the role of a traditional storyteller and the children can interact
with him using natural language.
Author(s) Soliman and Guetl (2013)
Title Implementing Intelligent Pedagogical Agents in virtual worlds
Summary Virtual worlds are providing an interesting medium for engineering education since
it can provide visual collaboration and authentic and active learning experiences.
However, they lack the important educational support provided by a teacher.
10
The authors solution is to include pedagogical agents in the virtual worlds
and thus adding the lacking learning support. The paper presents a prototype
implementation of an IPA which can interact with a learner in natural language
and support their understanding of the virtual experiments. Both the IPA and the
learning scenarios are implemented in the open source platform Open Wonderland.
Author(s) Gang, Bo, Chen, Yi, and Zi (2014)
Title Chinese Intelligent Chat Robot Based on AIML
Summary The paper shows different aspects of an implemented Chinese Intelligent Chatbot,
like the internal structure, the foundation, the final effect and most importantly
11 the problems met during construction and how they were solved. A Chinese
chatbot encounters unique problems due to the lack of a perfect corpus and the
Chinese word segmentation system. The paper presents a way to handle these
problems through the inclusion of a word segmentation system in the bot as well
as a means to handle synonymous sentences.

21
 Table 4.1: (continued)

Id Paper
Author(s) Kumar, Chandar, Prasad, and Sumangali (2016)
Title Android Based Educational Chatbot for Visually Impaired People
Summary The purpose of the paper is to present an Android application containing an
educational Chatbot designed for visually impaired people. The application can
12 be launched with Google Voice Search and is used by asking questions in spoken
natural language, the application will then convert it to text and run the query
against the AIML database or, if no pre-saved answer was found, against the
Wikipedia API. The answer is then returned to the application as both text and
voice.
Author(s) Fonte, Nistal, Rial, and Rodrı́guez (2016)
Title NLAST: A natural language assistant for students
Summary The researchers present a system consisting of two parts, an android application
and a server platform. The android application implements a chatbot which
interacts with both the student and the server. The objective for the system was
13
to enable the student to carry out several actions related to their studies like:
consult exams questions, receive recommendations about learning materials, ask
questions about a course, and check their assessed exams. The purpose of the
chatbot is thus to provide a friendly interface which can be interacted with using
natural language.
Author(s) Ranoliya, Raghuwanshi, and Singh (2017)
Title Chatbot for University Related FAQs
Summary The paper presents a chatbot design implemented to be able to answer FAQs
regarding university related questions in an efficient and accurate way. The
14
implemented chatbot is based on AIML and the researchers have the goal to
further improve the capabilities by eventually also using Latent Semantic Analysis.
The bot works well to provide 24/7 service to students or others interested in
the university.

4.2 Publication Details

This theme group contains the themes regarding the meta-details of the data, in this case
the details regarding the publications. The themes are Author(s), Institution, Time published,
Keywords in paper and Journal/Conference (see table 3.3). These were analyzed in order to find
patterns (or the absence of patterns) in the data and thus be able to draw conclusions regarding
who has an interest in the chosen field, or where and when this interest manifested itself.

4.2.1 Author(s)
In most of the dataset the authors differed between the papers, the only deviation was three
papers, paper 3, 5, and 13. Paper 3 and 5 shared the same four authors, and in paper 13 three of
these four authors appeared again, along with a new author. Other than that, no author appear
more than once in the set and the names of the authors can be seen in table 4.1

22
4.2.2 Institution
As with the authors, no institution stands out as having a greater representation in the dataset.
This fact implies that no institution has tried to specialize their work towards chatbots in education.
The sole exception of this was the three papers mentioned in the ’Author(s)’ section, in which
the institution was the same in all three papers as well (papers 3, 5, and 13). The authors of
these papers were all from the Telematics Engineering Department, at the University of Vigo.

4.2.3 Time Published

The papers have been published during a span of 9 years, from 2008 to 2017, with 3 papers
published in 2008, 2 each in 2009, 2011, 2012, and 2016, and 1 each in 2013, 2014, and 2017.
Ergo the papers were published quite evenly from 2008 until 2017. The search strategy restricted
the time of publication to after 1995, but no article was found in the span between 1995 and 2008.

4.2.4 Keywords in paper

Figure 4.1: Keywords in the papers

This theme encompassed the keywords the authors had tagged their papers with, most papers
defined these just below the abstract in order to make their paper more searchable. There were
50 keywords identified in the whole set (see fig 4.1), the majority used only once (43 words). The
most used keyword is AIML at 5 times, closely followed by Chatbot at 3 times. Five keywords
were used 2 times each; ALICE, Artificial Intelligence Markup Language, Chatterbots, Intelligent
Tutoring System and Patterns Matching. However a few of the words used as keywords in the
same or in different papers are synonymous, making the simple count more unrepresentative. For

23
example AIML and Artificial Intelligence Markup Language are counted as different keywords
but as they are synonymous could be counted as only one keyword. The issue at hand was that
the abbreviation and the full phrase were both used as keywords for the same paper, a choice
the authors made which was therefore respected in the thesis, thus keeping the words separate.
This phenomena is showed in figure 4.2 where the full phrase has been written first and then the
abbreviation is included as a parenthesis. If these duplications were to be disregarded however,
the list of tags would be shortened.

Figure 4.2: Abbreviated Keywords paper 11 and 14

4.2.5 Journal/conference
No Journal or conference was represented more than once i.e. no publication stands out as having
a greater representation in the dataset. The majority of the papers, 8 out of 14, were published
as conference proceedings.

4.3 Educational Context

This theme group examined what educational context the chatbots in the papers were developed
to target by looking at the predetermined themes. The themes were ’Level of education’, ’Student
demographics’, ’Subjects & Courses’ and ’Type of education’ (see table 3.3). They were analysed
in order to see if any educational context were more clearly represented in the set or to find if any
context stood out.

4.3.1 Level of education

This theme encompassed what level of education the bot was developed for or tried in, and was
relevant in order to see if anything differed in the approaches of the researchers or in the imple-
mentation of the chatbot depending on the targeted level. Not all papers clearly state that they
are directing their efforts towards a specific level, although in the non explicit ones it was implied
that the chatbot was developed toward university or at least, simply adults. 6 papers explicitly
stated their level, 5 out of these declared that their target level was university students in graduate
or post-graduate courses. The 6th paper, which was the only clear deviant in this theme was
targeting children, or Malaysian Children to be exact. This was paper 9 and the chatbots purpose
was to teach the Malaysian children about morals through the virtual telling of Malaysian folktales.

24
4.3.2 Student demographics
As with the level of education, not all papers contained information about who the chatbot system
was designed for or tested by, and not all researchers even described the testing of their chatbot,
if any testing was done at all. 6 papers did however mention their participants, papers 2, 4, 6,
7, 9, and 14. Again all except paper 9 had tested their chatbots on students or adults, in line
with the findings in the theme Level of education. Apart from this pattern no other similarities in
the demographics were found, the tested students ranged from 18 to around 200 in the different
papers. Two papers (2 and 4) reported the gender of their participants, and in both cases the
overwhelming majority was women.

4.3.3 Subjects & Courses

This theme had only 6 findings. Two papers directed their chatbots toward courses in education
or educational technology (2 and 7). One paper developed their bot to target Psychology students
(4) and one for courses in Computer Architecture (13), and again, the chatbot in paper 9 was
build with the subject of folktales and morals in mind. A slightly different subject was that of the
chatbot in paper 10, which was build to incorporate “remote and online labs and simulations”.
No specific course or subject is mentioned however, which means the implementation might be
intended to be flexible as to subject, as long as experiments or simulations are a part of the corpus.

4.3.4 Type of education

This theme was included among the initial theme since it was thought to embrace an important
aspect of the chatbots profitability, namely that it is beneficial for online courses. But unfortunately
only two references to this theme could be found in the literature. Paper 2 described the type
of their education in terms of how often the students would meet: “students met seven times
throughout the semester for an average of three hours”. The type of education seems to be in
part campus based, but online/distance based as well. In paper 7 the course stated as online
based and would serve up to 2000 students.

4.4 ICT implementation details

Due to the more complex nature of this theme-group the presentation of the results within was first
explained in relation to each theme, and after that according to the respective Chatbot described
in the papers. This is done in order to be able to both look at each individual theme on its own,
but to also be able to see more clearly how the themes fit together within the chatbots. Thus
the section starts with the codes, explaining them, validating their foundation in the dataset and
showing an overview of each theme and its sub-themes. Next the individual paper will be discussed
along with the present themes and sub-themes of that paper, thus a more detailed description of
that papers chatbot might be given as well as a view of how the themes and sub-themes relate
to one another. Whats important here is not any hidden meaning or inference made from the
findings, but the findings themselves, i.e. the presence of the themes and sub-themes.

25
4.4.1 The themes in the ICT implementation details group
The themes found in the ICT group are ’Role of Chatbot’, ’Bot Features’, Encompassing system’,
’Bot Augmenting Tech’, ’Platform’, and ’Access to bot’ (see table 3.3). The findings of these
themes will be presented and validated below. These themes were decided before the coding, as ini-
tial themes, and were coded in the papers. The results of each theme were then reviewed on order
to locate the sub-themes which are presented in this section and in the tables 3.4, 3.5, 3.6, 3.7, 3.8.

Role of chatbot
This theme encompass the roles of the different chatbots. A chatbot might have one or more
roles and the role of a chatbot might hint to what the chatbot does, but these facts are not
the main focus of the theme. The focus is instead on what purpose the chatbot was given and
thus which role it was supposed to play in the system at large. 5 sub-themes were extracted and
their relationships evaluated through their appearances together. The sub-themes were: ’Teacher
communication’, ’Natural conversation’, ’Student Evaluation’, ’Questions & Answers’, and ’Tutor’.

Figure 4.3: Sub-Themes for Role of Chatbot

As mentioned, the relationships between the themes were evaluated, as this seemed like both an
interesting and important factor. As figure 4.3 shows, the relationships are rather complex, with
several sub-themes being related both to the main theme and the sub-theme ’Tutor’.

Sub-theme: Tutor
The chatbot is used as a Tutor. Defining what a tutor is turned out to be a rather difficult
task, as there are many kinds of tutors and the definitions and tasks of a tutor may wary both
depending on the level of education and the geographical location. The word tutor can be used
both as a noun and a verb, one describing the person and the other the act. A tutor is typically a
paid teacher or instructor who works privately with one student or a small group. This might
be to provide extra help outside school hours with a subject the learner finds difficult or to
instruct and guide someone in some branch of learning, in summary, to give additional, special, or
remedial instruction to someone. The tutor can be a teacher without institutional connection

26
who assists students when preparing for examinations, or a teacher in some university who has a
lower rank than that of an instructor. With this definition it’s clear that a tutor can be many
things. (Cambridge dictionary, 2018; Oxford dictionaries, 2018; Collins English Dictionary, 2018;
Merriam-Webster, 2018; The free dictionary, 2018)

In 9 of the 14 paper the chatbots desired role was determined to be that of a tutor (papers 1,
2, 3, 5, 7, 9, 10, 11, 13) however, it was never the sole role. In the papers where a ’Tutor’ role
could be determined, it was usually in relation to other roles, e.g. in paper 3 the researchers
state that their chatbot “. . . gives students a virtual tutor capable of solving their doubts
as real ones”, as well as that it “. . . gives students a virtual evaluator capable of evaluating
and monitoring their progress. . . ”. The first quote explicitly stating the role of ’Tutor’ while
the second quote describes a role of ’Student Evaluator’. On one occasion however this was
not true, in paper 5 the researchers describe their chatbot as “. . . able to maintain a general
conversation with students, showing them contents of courses, and asking them questions about
the learning material” which is targeting the very core of what a tutor is but cannot comfortably
include any other sub-theme. That means a chatbot in the role of a tutor can be only that,
but most often it includes other roles as well, widening the functions of the chatbot in the process.

The relationships discovered in the data showed that 5 of the sub-themes could themselves be
sub-themes of ’Tutor’, since the definition of a tutor is very broad. These 3 sub-themes were
’Teacher communication’, ’Student Evaluation’, and ’Questions & Answers’. Of course a tutor
might also be able to hold a natural conversation but in the data it was not found that this was
ever the role of a chatbot, thus a distinction was made regarding the context of the themes.
These relationships will become more clear in the section where each paper’s chatbot and its
themes are presented, but a brief example will be given here as proof of the claim.

Furthermore, in paper 10 the researchers describe their chatbot as follows:

“. . . the avatar learner can ask the pedagogical agent questions on the scope
of an experiment, gets information about the experiment, listens to and sees
a demonstration to running an experiment visually. The IPA has the ability to
monitor learner actions versus expected ones to run the experiment and upon
errors, the IPA provides feedback, corrects errors, and updates learner assessment
data.”
This quote show that the chatbot has the roles of ’Questions & Answers’ and ’Student Evaluation’.
While it doesn’t explicitly use the term tutor the definition of what a tutor is encompass the same
values they describe here, thus it can be inferred to be a role as well.

Sub-Theme: Student evaluation

The role of the bot is to evaluate the students progress or performance, in order to help the
student improve or for the teacher to keep an eye on the development in the group. Two papers
present bots developed to perform student evaluation, paper 3, from which relevant quotes have
already been presented, and paper 10. Both chatbots have been given other roles as well, 3 is,
as mentioned given the role Tutor, whereas 10 has been given both the role of ’Tutor’ and of
’Questions & Answers’, made clear with the quote below.
“With this implementation, the avatar learner can ask the pedagogical agent
questions on the scope of an experiment, gets information about the experiment,
listens to and sees a demonstration to running an experiment visually. The IPA has
the ability to monitor learner actions versus expected ones to run the experiment
and upon errors, the IPA provides feedback, corrects errors, and updates learner
assessment data.”

27
Sub-theme: Questions & Answers
The theme included chatbots whose purpose was to answer student questions, often topic
specific questions. 9 papers were determined to have this role, paper 1, 2, 4, 7, 8, 10, 11,
12, and 14. In paper 8 the authors write that their chatbot is “giving immediate answers to
questions about library services and resources. [. . . ] It is also designed to guide researchers to ap-
propriate resources and help them solve problems in a similar way as the more conventional FAQ.”.

Sub-theme: Teacher communication

A chatbot can be used as a medium to communicate with a teacher. In the dataset one paper
(paper 1) used their chatbot as a communicator between student and teacher, giving the chatbot
a role of ’Teacher communication’. They state that “. . . in case of necessity it [the chatbot]
activates the PABX to send a direct short message (SMS) to the teaching team, containing the
problem’s object asked from the user to the agent.”. The chatbot first try to answer the students
question on its own, searching its knowledge base. If it does not find a suitable answer a teacher
is contacted either through email or phone (using text-to-speech technology) in order to give the
student an answer efficiently.

Sub-theme: Natural conversation

A chatbot used simply for a conversation, utilizing its ability to converse in natural language.
Paper 6 presented chatbots for language learning which students may interact with as an NPC in
a virtual world. The chatbots “. . . perform interactive dialogues with students. These behaviors
contribute to the acquisition of listening and writing skills. . . ”, the sole role is thus to keep a
’Natural Conversation’. Other chatbots has the ability to have natural conversations as well, but
it is not their role, rather just a feature.

Bot Features
A chatbots features is all the things it can do. For the most part it is connected to the role
of the chatbot, but a chatbot might have features outside the main role. For example, almost
all chatbots can keep a natural conversation, but not all chatbot has this as a role. However,
accepting input or giving output in natural language is not regarded a feature in this list, as it is
something every chatbot in the set is able to do. The 17 features found in the dataset are:

Demonstrate learning task Support feedback to students

Animated gestures supporting knowledge base Give recommendations about learning material
Allow input as speech Hold topic specific conversation
Control e-learning platform’s activity Hold general conversation
Provide means of contact with teacher Teacher can edit bot content
Point of interaction for learner Ask the student questions
Provide output as speech Conversational strategies
Help students 24/7 Answer student questions
Provide content from other knowledge sources

Sub-Theme: Demonstrate learning task

The chatbot can demonstrate a task for the learner, for the learner to better grasp the contents
of the task. In paper 10, the chatbot has been used as an NPC avatar in a virtual world and
can demonstrate the experiments within if the learner asks for it: “the avatar learner can ask

28
the pedagogical agent questions on the scope of an experiment, [. . . ] listens to and sees a
demonstration to running an experiment visually.”

Sub-Theme: Animated gestures supporting knowledge base

The chatbot uses animated gestures to enhance their message. This feature is only possible
with animated embodied agents, like that in paper 10. “Attention and gesture supporting
knowledge base to emotional state of the learner. For example, ’Can I help you? It is not
a difficult experiment (linked to gesture)’.”. The purpose of animated gestures seems to be
to provide emotional support as well as didactic and to enforce the chatbots message even more. ”

Sub-Theme: Allow input as speech

The chatbot can accept input in speech format. In order to allow blind learners the opportunity
to learn easily on their own, the researchers of paper 12 needed their chatbot to be able to accept
voice input. “It has the capability to get the input as speech format from the user. So [the] user
[has] no need to type the keyboard [to] provide the input.”

Sub-Theme: Control e-learning platform’s activity

The chatbot can control and monitor the activity on the platform and in need alert the responsible
teacher. In paper 1 this is done through the integration of a control deamon and a system to
contact a teacher. “control bases on the interaction of TutorBot which picks up eventual signals
from the learners, TutorBot when receives the signals sparks off [. . . ] alarm to a competent staff
for platform contents.”. The motivation of this feature in the paper is that “it’s necessary [to
have] a continuous monitoring activity of e-learning platforms, to spare the disadvantages that
could jeopardized the right course of on-line didactical activities”.

Sub-Theme: Provide means of contact with teacher

The chatbot can provide the learner with a means to contact the teacher, effectively providing
“prescreening” of the learners questions. This feature was found in the chatbot in paper 1: “in
case of necessity it [the chatbot] activates the PABX to send a direct short message (SMS) to
the teaching team, containing the problem’s object asked from the user to the agent.”

Sub-Theme: Point of interaction for learner

The chatbot can act as the point of interaction between the learner and the learning environment.
In this dataset this is explicitly stated in paper 6 and 10, where in both cases the chatbot acts as
a NPC which the learner interacts with. In paper 10 the chatbot is made to “Act as a central
point of interaction between the learner and the environment providing verbal and non-verbal
support to the learner.” and in paper 6 “chatbots perform interactive dialogues with students.”.

Sub-Theme: Provide output as speech

The chatbot can provide its output in a spoken format. In this dataset the output is always
given as text, but three papers (6, 10 and 12) also provided their chatbots with a voice output.
The three papers descibe it differently, but aims at the same thing, improving the learning
experience. “chatbots perform interactive dialogues with students. These behaviors contribute to
the acquisition of listening and writing skills” (Paper 6). “When the IPA [chatbot] receives the
question, ’is it an easy experiment’, the answer with ’yes’ in either voice or text ” (Paper 10).
“Output of this application will be given as text and speech format.” (Paper 12). Paper 12 also
has the additional goal of benefiting blind learners, making speech output a necessity.

Sub-Theme: Help students 24/7

A chatbot can provide its services anytime of the day and year, and will never have on off day.

29
4 papers specifically state this fact (papers 3, 8, 10, and 11), but unless the platform the bot
is set on has a time limited access, the same should be true for all chatbots in the dataset.
Paper 8 describe this feature in their chatbot Pixel: “Pixel will work 24 / 7 with less than 1
percent downtime, and will provide consistent answers. It searches across varied sources and
brings together information in a similar way a librarian would work with a user.”.

Sub-Theme: Provide content from other knowledge sources

The chatbots own knowledge base might not always be enough to provide adequate service to the
users, in that case it can use other knowledge sources to gather data in order to answer a query.
Such a source might be an online encyclopedia (e.g. wikipedia), a search engine (e.g. Google), or
some licensed database (e.g. a library database). This feature is present in 4 papers’ chatbots, 1,
2, 8, and 12. Paper 8 use both licensed databases and Google in their chatbot implementation:
“The libraries provides access to licensed databases [. . . ]. The chatbot Pixel incorporates all of
these resources into the context sensitive responses “she” generates”, and “It is [. . . ] a tool for
undergraduates and researchers looking for quick information and tips on searching databases
and conducting research. As such, Pixel sends factual questions (what is the population of the
US) to Google.”. Paper 12 on the other hand use Wikipedia: “The proposed android Chatbot
application could provide any kind of information which included in Wikipedia and it can also
provide user defined information”.

Sub-Theme: Support feedback to students

The chatbot supports providing feedback to students or teachers. This feature is present in
paper 3, 9, 10. The feedback might be related to the student progress, giving the teacher an
easy way to make sure no one falls behind in a large group and to allow the teacher utilize
his/hers resources accordingly. Paper 3 claims this: ”obtain direct feedback about student
difficulties [. . . ], teachers can concentrate on those concepts that require more expertise and
real human interaction.”. The feedback might also be given directly to the learner, as is the
case in paper 9 and 10: “[The chatbot] can interact with children [. . . ] by giving immediate
feedback” (9), “The IPA has the ability to monitor learner actions versus expected ones [. . . ] and
upon errors, the IPA provides feedback, corrects errors, and updates learner assessment data.”(10).

Sub-Theme: Give recommendations to students about learning material

The chatbot can give a student recommendations about learning content on for example the
e-learning platform the bot implemented in. It can also be used to simply present the student
with content of a specific course. The feature is present in 3 papers: 1, 5, and 13. Paper 5 says
that the chatbot is “showing them [the students] contents of courses” and paper 13 that “When
the student is viewing an exam question, he/she can asks the chatterbot for a recommendation
about educational contents related to the concepts of the question.”.

Sub-Theme: Hold topic specific conversation

The chatbot is capable of conducting a topic specific conversation. The chatbot can hold a
conversation on a specific subject contained in its knowledge base, letting the learner ask questions
or in some cases asking the learner questions. This feature was found in 7 papers, which means
that it is very well grounded in the dataset. The papers are 2, 3, 4, 5, 10, 13, and 14. Paper 4
for example describe their chatbot as: “programmed to chat in the first person about Freudian
theory, concepts and biographical events”, whereas paper 14 built a “chatbot for [the] educational
sector, where user [. . . ] can ask query regarding college admission, about college information and
other things related to academics”.

Sub-Theme: Hold general conversation

30
The chatbot is capable of conducting a general conversation. This could mean that the chatbot
can talk about general topics such as the weather or their preferences. This feature may give the
learner more motivation to interact with the chatbot and make it feel more human. 5 papers
explicitly stated this feature in their chatbot, papers 3, 5, 6, 9, and 10. Paper 3 explains that the
student can hold a conversation with the chatbot and that “This conversation can be a general
conversation (thanks to a set of predefined AIML files that contain general information)”. Paper
9 also provides the motivation of this feature: “[The chatbot] was designed to raise a child’s
interest since it could engage with them in a continuous conversation. The presences of PA
could motivate children to stay longer [. . . ] since [. . . the chatbot] maintained children’s level of
confidence and encouraged their learning”.

Sub-Theme: Teacher can edit bot content

The chatbots implementation allows the teacher or administrator to edit the content of the bot.
This might be the general information, course specific content, or the learning resources the chatbot
can access. Paper 3 and 5 implement this feature, note that these papers are written by almost the
same authors. Paper 3 claims that “it is possible to customize the bot giving it a name, hobbies,
home town, birthday, etc. and a background image for the conversation window” and that ”[The
chatbot] has an interface where teachers can select the resources from a structure of modules . . . ”.

Sub-Theme: Ask the student questions

The chatbot has the capability to ask the student questions on the learning content. 3 papers in
the dataset show this feature, paper 3, 5, and 13, again note that these papers are written by
almost the same authors. In all papers it is possible for the learner to ask the chatbot for questions,
these might be exam questions, questions associated with a predefined test or personalized, or,
what they call ’free questions’. Paper 13 says that “If the student asks for exam questions related
to specific concepts [. . . ], the chatterbot will [. . . ] search into a repository of exam questions
[. . . ] chose ones related to the concepts specified by the student.”. Furthermore, in paper 5 the
researchers write:
“while a student is talking to the bot, he/she can request several activities related
to tests: [1] Ask for a test: The bot chooses one over all the tests the administrator
had created. [2] Ask for a personalized test: The student must choose the number
of questions the test will include and the bot will create the test that fits this
requisite. [3] Ask for questions that do not belong to a test (free questions): The
bot starts to ask questions and keeps on it until the student wants to stop.”

Sub-Theme: Conversational strategies

A chatbot can exhibit conversational strategies, such as the ability to ’chain’ a conversation or
to return to a topic after a branching. The purpose of a conversational strategy is to keep the
learner focused on the important material and to provide believability to the chatbot. Strategies
such as these were found in 6 papers, papers 4, 5, 6, 7, 8, and 10. In paper 4 the researchers
write that the chatbot “has the capacity to return to a story after branching to a new location
and also retains the parts of a story that have been told to prevent repetition.” and paper 8
claims that “The metadata supports the ability to “chain” a conversation. This is accomplished
when Pixel asks a follow-up question in response to the initial input. Based on the user’s answer,
additional information is provided. Using this Q&A approach surprisingly complex conversations
can be held between the bot and the chatter.”

Sub-Theme: Answer student questions

The chatbot has the capability of answering the students questions. This may be the most basic of

31
an AIML-based chatbots features, and is one it is very well suited for due to the implementation.
Thus almost all papers’ chatbots exhibit this skill in one way or another. The remaining 3 papers’
chatbots (2, 5, 7) might contain the feature as well, but it cannot be inferred from the paper
without making several assumptions. Paper 3 for example state that their chatbot “is able to
answer students’ questions about different subjects using natural language.” and paper 14 describe
their chatbot as an “interactive chatbot for University related Frequently Asked Questions (FAQs)”

Encompassing system
The theme compass the system (if any) the chatbot has been implemented into. This system might
be used to limit the users of the chatbot to a specific course, to provide additional technology
to enhance the overall experience or it might be the opposite, that the chatbot is added to the
system to enhance its qualities. The 4 sub-themes of the theme are: ’Library system’, ’Virtual
environment’, ’E-learning system’, and ’Stand-alone’. The last is, of course, not technically a
system but rather the lack of one, but it is as important to the theme as the others.

Sub-Theme: Library system

The chatbot is incorporated in a library system, in this dataset it only applies to the chatbot
in paper 8, Pixel. Due to this integration the chatbot is easy to reach for the intended users
(students and researchers) and the chatbot has access to the library’s databases. The write that
the chatbot works by “giving immediate answers to questions about library services and resources”.
They also claim that “Pixel flattens the website so users do not need to interpret the website and
navigate through pages to find the specific information they are seeking. It is also designed to
guide researchers to appropriate resources and help them solve problems in a similar way as the
more conventional FAQ”.

Sub-Theme: Virtual environment

The chatbot is implemented in a virtual environment. In the dataset the chatbots are used as
virtual characters, or NPC’s, the learner can interact with. Three papers have used their chatbot
in this manner, paper 6, 9, and 10. In paper 6 this virtual world is a “3D multi-user virtual world
that imitates cultural sights of Madrid in which a community of learners experience auditory and
visual immersion.”. In paper 9 the virtual environment is: “Hikayat Land[: . . . ] a virtual folktales
land containing a series of Malaysian children’s literature, the folktales of Hikayat Sang Kancil.”.
Lastly in paper 10 the researchers virtual world is “implementing learning scenarios with intelligent
pedagogical agent in Open Wonderland virtual world.”

Sub-Theme: E-learning system

The chatbot works as a part of an e-learning system, it may be as the main interaction point or
as just one tool in the system. 5 papers fall into this sub-theme, papers 1, 3, 5, 7, and 13. In
paper 1, the chatbot “takes directly the role of front end on the elearning platform’. In paper
5 the chatbot is the intelligent part of a bigger system “INES is a functional prototype of an
online learning platform, which combines three essential capabilities related to e-learning activities.
These capabilities are those concerning to a LMS (Learning Management System), a LCMS
(Learning Content Management System), and an ITS (Intelligent Tutoring System).”.

Sub-Theme: Stand-alone
A stand-alone chatbot is not incorporated in any larger system, other than the platform on which
it is installed. 5 papers (2, 4, 11, 12, 14) were found to belong in this sub-theme. Paper 12 states
this simply with the sentence “This application can run in every android devices with Internet

32
connection”. Paper 11 explains that they “built the function of WEB and mobile usage”. With
paper 2, 4, and 14 however, no encompassed system or lack thereof was explicitly stated, they
were therefore assumed to be stand-alone components, something that is of course important to
note for the reader.

Bot Augmenting Tech

This theme encompass the technology that has been added to the bot to enhance it or to com-
pensate for some weakness it has. There were three sub-themes found in this theme: ’Linguistic
tools’, ’Animation’, and ’Text-to-speech’.

Sub-Theme: Linguistic tools

The language capabilities of an AIML-based chatbot are limited to what has been programmed
into the knowledge base, thus a linguistic tool might be used to compensate this limitation. In
the dataset 4 papers have used some linguistic tool in their chatbot, papers 3, 5, 11, and 14.
Papers 3 and 5 both implemented a linguistic analyzer in order to “obtain a recommendation of
key words/concepts related to the [newly added] course content.” (3). Paper 11 used a word
segmentation tool to aid their chinese reading chatbots understanding of the complex language, it
“is based on the research of a large semantic dictionary by Peking University, with the functions of
word segmentation and part-of-speech tagging.”. Paper 14 states that they intend to implement
latent semantic analysis (LSA) as an upgrade for their chatbot on order to “discover likenesses
between words as vector representation. So that the unanswered queries by AIML will be viewed
as a reply by LSA.”

Sub-Theme: Animation
The chatbot has been embodied and animated, the goal being to create a more engaging experi-
ence for the learner. Two papers explicitly state animating their chatbot, that is paper 4 and
10. Paper 4 created a Haptek animated image of Sigmund Freud for their Freudbot and then
“When the Haptek Player software is installed on the user’s computer, output from the AIML
server is fed through the Haptek client’s text-to-speech engine to produce speech output with
synchronized lip movement.”. Paper 10 used the built in functions in the Open Wonderland: “The
multi-modal communication module integrates gesture generation, text chat (through Natural
Language question making), and Text-To-Speech generation.”.

Sub-Theme: Text-to-speech
The chatput can give output/input as speech instead of text. This augmentation is relevant for the
animated chatbots, to actually achieve the full experience. It may also be relevant to reinforce the
text message or to provide a service to users who are blind or have reduced eyesight. 5 papers have
used this augmentation in their chatbot, papers 1, 4, 6, 10, and 12, i.e. both the animated chatbots
have it (see sub-theme: animation). Paper 1 use it in order to ’translate’ the students’ question for
their phone transmission to the teacher: “The phone call’s content, as we said, is a vocal synthesis
(text to speech) of conversation between the ChatBot and the learner.”. Paper 12, who specifi-
cally targets blind learners as their user group, write: “output will be given as speech by the TTS”.

Platform
Every chatbot has been implemented on some platform, the choice of platform is closely connected
to what encompassing system was used, for example most e-learning systems are located on the

33
’Web platform’. The sub-themes are thus, ’Web’, ’Mobile OS’, and ’Not-specified’.

Sub-Theme: Web
A chatbot implemented on a web platform has the advantage of platform independence in a
sense, since basically all devices today can reach the web. The papers who took advantage of this
fact are papers 1, 3, 5, 6, 7, 8, 10 and 11. Out of these, 6 papers had been encompassed in an
e-learning system, paper 3 for example simply write that “The students are able to connect to the
e-learning platform through the Internet”. The chatbot in paper 8 was published in a web-based
library system, “web pages on the libraries’ website’. Papers 6 and 10 was published in a virtual
world which provide the web-access: “Data common to all clients are stored in a WebDav-based
content repository hosted by the Open Wonderland Server. With this content repository, the
client can access these data via the HTTP protocol.”

Sub-Theme: Mobile OS
The chatbot was build for a mobile platform, this could Android, iPhone or Windows phone. 3
papers built their chatbot for a mobile platform, 12, 13, and 14, coincidentally these three papers
were published most recently. Paper 12 and 13 both implemented their chatbot on an Andoroid
system, “This application can run in every android devices with Internet connection” (12) and
“The Android application is basically a chatterbot, which provides a clean interface using text or
voice/audio to communicate with the user in natural language” (13). Paper 14 does not mention
a specific system, but show some application snapshots.

Sub-Theme: Not-specified
Some researchers did not specify the platform on which they implemented their chatbot. This
was paper 2, 4, and 9.

4.4.2 The papers and their chatbots

In this section the chatbots in the papers will be described in regards to which themes and
sub-themes of the ICT theme group they adhere to. Each section will start with the summary of
the paper from table 4.1.

Paper 1: Orlando and Giovanni (2008)

Summary
The paper describes an integrated system with the purpose of monitoring and supporting an
e-learning systems activities and discovering the students issues. The system alerts the teacher
with a phone call if a learner poses a question that cannot be answered, the question has then been
vocalized using text-to-speech technology. An AIML-based chatbot works as the front-end of the
application and is the instance the learner interacts with. For the present subthemes, see table 4.2.

Paper 2: Doering et al. (2008)

Summary
The study investigated the effect of conversational agents on communication and interaction
when used to assist participants in developing an online portfolio. Data gathered during four
weeks revealed that participants talked to the agents on a range of subjects, not only related to
the domain of e-portfolio. The participants didn’t find the agents very helpful in completing class

34
 Theme Sub-Theme found in chatbot
Tutor
Role of bot Questions & Answers
Teacher Communication
Control e-learning platform’s activity
Provide means of contact with teacher
Bot Features
Provide content from other knowledge sources
Answer student questions
Bot Augmenting Tech Text-To-Speech
Encompassing System E-learning system
Platform Web

Table 4.2: Paper 1: Chatbot sub-themes

activities, but used them as social companions. The researchers conclude that further research
is needed on three points: 1) Learner-developed conversational agents, 2) Improve the agents
“intelligence” regarding the domain knowledge, and 3) Agents that better meet users’ humanistic
and utilitarian expectations. For the present subthemes, see table 4.3.

Theme Sub-Theme found in chatbot

Tutor
Role of bot
Questions & Answers
Provide content from other knowledge sources
Bot Features Hold topic specific conversation
Answer student questions
Bot Augmenting Tech None
Encompassing System Stand-Alone
Platform Not-Specified

Table 4.3: Paper 2: Chatbot sub-themes

Paper 3: Mikic et al. (2008)

Summary
The authors have developed two AIML-based bots with the purpose of helping students during
the learning process and supporting teaching activities on an e-learning platform like Claroline
or Moodle. One bot has the role of a tutor (T-bot) and communicates with the learner in
natural language to give them adequate and domain specific answers and guiding students to
the right course material. The other bot had the role of an evaluator (Q-bot) and can track and
supervise the student through personalized questionnaires. Both bots have been developed as
easily integrated modules for Claroline or Moodle. For the present subthemes, see table 4.4.

Paper 4: Heller and Procter (2009)

Summary
The authors perform an experiment to explore how an historical figure application of a pedagogical
agent affected student engagement and motivation. 88 participants where randomly assigned to
one of three groups; a no image agent, a static image agent and an animated image agent. The

35
 Theme Sub-Theme found in chatbot
Tutor
Role of bot Student Evaluation
Questions & Answers
Help students 24/7
Support feedback to students
Hold topic specific conversation
Bot Features Hold general conversation
Teacher edit bot content
Ask student questions
Answer student questions
Bot Augmenting Tech Linguistic tool
Encompassing System E-learning system
Platform Web

Table 4.4: Paper 3: Chatbot sub-themes

results contradicted the expectations and previous research on animated pedagogical agents in
that the no image agent received higher ratings for half the measures. For the present subthemes,
see table 4.5.

Theme Sub-Theme found in chatbot

Role of bot Questions & Answers
Hold topic specific conversation
Bot Features Conversational strategies
Answer student questions
Animation
Bot Augmenting Tech
Text-To-Speech
Encompassing System Stand-alone
Platform Not-specified

Table 4.5: Paper 4: Chatbot sub-themes

Paper 5: Mikic et al. (2009)

Summary
INES (INtelligent Educational System) is an e-learning platform which comprises a set of different
technologies. One of these technologies is a chatbot which communicates with the students in
natural language and act as the “brain” of the system. The paper focuses on the description of
this chatbot called CHARLIE (CHAtteR Learning Interface Entity). CHARLIE can communicate
with students in natural language and answer general or domain specific questions. The student
can also request questionnaires or free questions from the bot to test their knowledge. For the
present subthemes, see table 4.6.

Paper 6: Ibáñez et al. (2011)

Summary
The paper describes a 3D multi-user virtual world for language learning. It was developed to test

36
 Theme Sub-Theme found in chatbot
Tutor
Role of bot
Questions & Answers
Give recommendations about learning material
Hold topic specific conversation
Hold general conversation
Bot Features
Teacher can edit bot content
Ask student questions
Conversational strategies
Bot Augmenting Tech Linguistic tool
Encompassing System E-learning system
Platform Web

Table 4.6: Paper 5: Chatbot sub-themes

if it it would create an engaging learning experience and thus foster communication skills with
minimal teacher involvement. The open-source development toolkit Open Wonderland was used
as a base and then extended to provide natural language conversation with Non-Player Character
chatbots, textual tagging of virtual objects, automatic reading of texts in learning sequences and
the arrangement of learning activities meant to promote collaboration. The authors preliminary
evaluation of the experience shows great promise. For the present subthemes, see table 4.7.

Theme Sub-Theme found in chatbot

Role of bot Natural conversation
Point of interaction from learner
Bot Features Hold general conversation
Conversational strategies
Bot Augmenting Tech Text-To-Speech
Encompassing System Virtual environment
Platform Web

Table 4.7: Paper 6: Chatbot sub-themes

Paper 7: Rossi and Carletti (2011)

Summary
The researchers describe an open source Learning management system containing a chatbot. The
system is aimed at supporting online teachers work by implementing a knowledge base specific
to pedagogical aspects and not a specific subject. The chatbot developed was based on the
ALICE chatbot and should retrieve information already coded in the courses or originated in
student activity logs and then answer the students most common questions in order to relieve
teachers from the task giving them time to answer more difficult questions better. For the present
subthemes, see table 4.8.

Paper 8: Allison (2012)

Summary
The paper describes a chatbot pilot “Pixel” developed to answer questions about the library and

37
 Theme Sub-Theme found in chatbot
Tutor
Role of bot
Questions & Answers
Conversational strategies
Bot Features
Answer student questions
Bot Augmenting Tech None
Encompassing System E-learning system
Platform Web

Table 4.8: Paper 7: Chatbot sub-themes

its resources at the University of Nebraska-Lincoln. The chatbot could answer questions from a
variety of users from around the world and, surprisingly for the author, attracted a lot of social
chatter. The majority of questions where of a factual nature which the Pixel could handle and
the database was proved easily revised as library resources or personnel changed. The chatbot
provided a consistent service 24/7 and also a playful interface which engaged users, relieving them
from complicated navigation systems and endless scrolling through search results. At the same
time it could refer the more complicated questions to the librarians. For the present subthemes,
see table 4.9.

Theme Sub-Theme found in chatbot

Role of bot Questions & Answers
Help students 24/7
Provide content from other knowledge sources
Bot Features
Conversational strategies
Answer student questions
Bot Augmenting Tech Text-To-Speech
Encompassing System Library system
Platform Web

Table 4.9: Paper 8: Chatbot sub-themes

Paper 9: Masmuzidin and Wan (2012)

Summary
Pedagogical agents have been praised as a tool to enhance learning and teaching, only the authors
identified a limited application with children as participants. The research then investigate if
a pedagogical agent can enhance children’s learning as well. They developed an agent set in
a virtual learning world called Hikayat Land, with the purpose of teaching children Malaysian
folktales and moral values. The agent has the role of a traditional storyteller and the children can
interact with him using natural language. For the present subthemes, see table 4.10.

Paper 10: Soliman and Guetl (2013)

Summary
Virtual worlds are providing an interesting medium for engineering education since it can provide vi-
sual collaboration and authentic and active learning experiences. However, they lack the important
educational support provided by a teacher. The authors solution is to include pedagogical agents

38
 Theme Sub-Theme found in chatbot
Tutor
Role of bot Questions & Answers
Teacher Communication
Support feedback to students
Bot Features Hold general conversations
Answer student questions
Bot Augmenting Tech None
Encompassing System Virtual environment
Platform Not-specified

Table 4.10: Paper 9: Chatbot sub-themes

in the virtual worlds and thus adding the lacking learning support. The paper presents a prototype
implementation of an IPA which can interact with a learner in natural language and support their
understanding of the virtual experiments. Both the IPA and the learning scenarios are imple-
mented in the open source platform Open Wonderland. For the present subthemes, see table 4.11.

Theme Sub-Theme found in chatbot

Tutor
Role of bot Student evaluation
Questions & Answers
Demonstrate learning task
Gestures supporting knowledge base
Point of interaction for learner
Provide output as speech
Bot Features Help students 24/7
Hold topic specific conversation
Support feedback to students
Conversational strategies
Answer student questions
Animation
Bot Augmenting Tech
Text-To-Speech
Encompassing System Virtual environment
Platform Web

Table 4.11: Paper 10: Chatbot sub-themes

Paper 11: Gang et al. (2014)

Summary
The paper shows different aspects of an implemented Chinese Intelligent Chatbot, like the in-
ternal structure, the foundation, the final effect and most importantly the problems met during
construction and how they were solved. A Chinese chatbot encounters unique problems due to
the lack of a perfect corpus and the Chinese word segmentation system. The paper presents
a way to handle these problems through the inclusion of a word segmentation system in the
bot as well as a means to handle synonymous sentences. For the present subthemes, see table 4.12.

39
 Theme Sub-Theme found in chatbot
Role of bot Questions & Answers
Help students 24/7
Bot Features
Answer student questions
Bot Augmenting Tech Linguistic tool
Encompassing System Stand-alone
Platform Web

Table 4.12: Paper 11: Chatbot sub-themes

Paper 12: Kumar et al. (2016)

Summary
The purpose of the paper is to present an Android application containing an educational Chatbot
designed for visually impaired people. The application can be launched with Google Voice Search
and is used by asking questions in spoken natural language, the application will then convert it to
text and run the query against the AIML database or, if no pre-saved answer was found, against
the Wikipedia API. The answer is then returned to the application as both text and voice. For
the present subthemes, see table 4.13.

Theme Sub-Theme found in chatbot

Role of bot Questions & Answers
Allow input as speech
Provide output as speech
Bot Features
Provide content from other knowledge sources
Answer student questions
Bot Augmenting Tech Text-To-Speech
Encompassing System Stand-alone
Platform Mobile

Table 4.13: Paper 12: Chatbot sub-themes

Paper 13: Fonte et al. (2016)

Summary
The researchers present a system consisting of two parts, an android application and a server
platform. The android application implements a chatbot which interacts with both the student
and the server. The objective for the system was to enable the student to carry out several actions
related to their studies like: consult exams questions, receive recommendations about learning
materials, ask questions about a course, and check their assessed exams. The purpose of the
chatbot is thus to provide a friendly interface which can be interacted with using natural language.
For the present subthemes, see table 4.14.

Paper 14: Ranoliya et al. (2017)

Summary
The paper presents a chatbot design implemented to be able to answer FAQs regarding university
related questions in an efficient and accurate way. The implemented chatbot is based on AIML
and the researchers have the goal to further improve the capabilities by eventually also using

40
 Theme Sub-Theme found in chatbot
Role of bot Tutor
Provide content from other knowledge sources
Give recommendations about learning material
Bot Features Hold topic specific conversation
Ask student questions
Answer student questions
Bot Augmenting Tech None
Encompassing System E-learning system
Platform Mobile

Table 4.14: Paper 13: Chatbot sub-themes

Latent Semantic Analysis. The bot works well to provide 24/7 service to students or others
interested in the university. For the present subthemes, see table 4.15.

Theme Sub-Theme found in chatbot

Role of bot Questions & Answers
Hold topic specific conversation
Bot Features
Answer student questions
Bot Augmenting Tech Linguistic tool
Encompassing System Stand-alone
Platform Mobile

Table 4.15: Paper 14: Chatbot sub-themes

41
Chapter 5

Analysis

The chapter contains the analysis of the results acquired in the literature review. The structure of
the chapter will follow that of the results presentation, in order to provide a clear and concise
report. The chapter starts with the analysis of the results from the publication theme group,
move on to the analysis of the educational context theme group, and finish with the analysis of
the themes and sub-themes in the ICT implementation details theme group.

5.1 Publication details

The finding in the publication theme group is analyzed. The most interesting findings in this group
include the diversity of the authors, institutions, publication dates, keywords, and publication
venues used within the dataset. This diversity imply that the subjects is of interest for a large
group of researchers, within diverse fields of studies.

5.1.1 Author(s)
11 of the 14 papers had different authors, something that indicates a dispersed interest for the
subject, rather than a limited circle of researchers. A benefit of this might be that the research
in the area is broader and more diverse, although a disadvantage might be that no depth of
knowledge in specific factors are procured. 3 papers had authors common authors, and in these
papers a similarity and progression of ideas was clearly visible, i.e. here we can see the advantage
of the knowledge deepening.

5.1.2 Institution
This themes’ results really tell the same tale as that of the authors, no single institution stands
out as having a greater representation. Naturally the same three papers which shared authors
also share an institution and the analysis remained the same, a similarity and progression of ideas
was clearly visible.

5.1.3 Time published

The publications spanned 9 years, from 2008 until 2017. The surprising fact here was that no
found papers had been published between 1995 and 2007. 1995 is the time the ALICE-bot was first

42
published, so that there was be a slight time gap before others caught on to the technology might
be expected but 12 years seem excessive. This might be due to some error in the search strategy
or the fact that educational uses might take longer to be developed within a new technology.

5.1.4 Keywords in paper

Within this theme it is interesting to note that the amount of unique keywords in the dataset,
50 unique keywords in only 14 papers. This implied a breadth of the field and use of chatbots
even in this very limited scope of context (education and AIML-based bots in 4 databases). This
analysis is supported by the earlier analysis that a diverse set of authors produce a broader set of
research.

5.1.5 Journal/conference
The finding that the papers were published all in different journals/conference proceedings might
imply a lack of a clear venue for these kinds of cross-subject projects, the mix of education and
IT/HCI/Informatics, in the academic world. Or it might just imply that the same cross-subject
platform contains to many suitable venues, leading the work to be spread out over several different
conferences and journals. 8 out of 14 papers were published for conferences, this fact might
indicate that the research was young when published and then abandoned for more fruitful
endeavors. It might also mean that when the paper was published in a journal, other keywords
were used and thus the paper was not collected in the search.

5.2 Educational context

The findings of the educational context theme group are presented in this section. On the whole,
the four themes in this group yielded a lot fewer results than was expected. Few researchers
had gotten to the stage of testing their chatbot and thus less information on the setting of the
chatbot was reviled. This is the most likely reason to the unexpected lack of results. However,
some were found and these will be presented below.

5.2.1 Level of education

There might be a very simple reason for this clear pattern of target level, namely that since the
technology is still young, and it demands expertise in both the desired corpus and the implementa-
tion details, researchers might find it both safest and easiest to start development “close to home”
by trying it on their own students initially. Education on all levels is a critical endeavor, not easily
changed or experimented with. When testing new software on non-critical courses or aspects of
courses this risk is lessened. In the same way, the burden of finding corpus expertise is lessened
by implementing the chatbot into a context the researchers themselves are already familiar with.

5.2.2 Student demographics

The majority of the targeted group were adults, and where information was given (paper 2 and 4)
another majority were women. How this might have affected the development or evaluation of
the chatbot was impossible to tell since the researchers themselves did not reflect more closely

43
on it. None of these findings seemed to have particular relevance to the development, testing or
evaluation of the chatbots.

5.2.3 Subject & Courses

There was 6 findings in this theme, i.e. less than half of the researchers had specified which
subject or course their chatbot was developed for. This might mean that the goal of the chatbot
was not to serve a single subject or course, instead developing a tool that might be easily adapted.
It might also mean that the research and development is very new to the researchers, i.e they
have not been applied in any specific course contexts get. The last implication is supported by
the conclusion from the Journal/Conference theme; that the research is young.

5.2.4 Type of Education

This theme did not yield as many results as expected and because of this it will be impossible
to draw any relevant conclusions from it. The expected result was that a lot of the chatbots
would be implemented in online courses with many students, as this is the area where a chatbot
might shine the most. The two papers which mentioned the type of education supported this
expectation, by both claiming to be used in courses with full or part time distance online.

5.3 ICT implementation details

In this section the results from the two subsections in the ICT implementation details section
(4.4), 4.4.1 and 4.4.2, are looked at collectively and referred to across the sub-themes. The most
important findings in this group was located within the themes of ’Role of chatbot’ and ’Bot
features’, since the sub-themes of these themes explain what a chatbot can be used for and what
a chatbot can do. Secondly the sub-themes of ’Encompassing system’ and ’Bot augmenting tech’
are important to see if the chatbot technology can stand on its own of if it requires additional
technology to be of value.

5.3.1 Role of Chatbot

The theme was divided into 5 sub-themes to encompass the different roles that were found. The
found roles were ’Student evaluation’, ’Questions & Answers’, ’Teacher communication’, ’Natural
conversation’ and ’Tutor’. The roles encompass different aspects of a teachers responsibilities and
thus show that the researchers have thought of the chatbot as an extension of the teacher role,
rather than something separate. The roles are generous in their definition, and might encompass
varied features and other attributes.

The most common role of the chatbots in the dataset were ’Tutor’, a likely reason for this might
be that a tutor is viewed as someone who helps a teacher. It may also be the fact that a tutor is
difficult to define and the views on what a tutor-role include are varied. However none of the 9
chatbots found to be tutors were given only this role, the tutor role came in relation to other
roles leading to the figure presented in the results chapter (see fig. 4.3). This implies that a tutor
might include these other roles or tasks, but, e.g. a ’Student Evaluation’ chatbot does not have
to be a tutor, it can simply give students feedback.

44
5.3.2 Bot Features
There were 17 features found in the 14 chatbots, most appearing more than once. The fact
that the dataset showed such a breadth in the chatbots capabilities is not surprising, taking the
conclusion from the publication details theme group into consideration, especially the wide range
of keywords used. The fact that so many features were found point to a conclusion that chatbots
are extremely flexible and extensible, lending themselves able to perform many different tasks.
The four most used features were ’Answer student questions’, ’Conversational strategies’, ’Hold
topic specific conversation’ and ’Hold general conversation’. The of the most used features are
related to holding a conversation, something chatbots are documented to be good at. That
these four are in the top is in line with the findings in the ’Role of bot’ theme, where ’Tutor’
and ’Questions & Answers’ were the most popular, both roles which often include questions or
conversations with students.

5.3.3 Encompassing System

Most chatbots had been implemented in some encompassing system, the three present in the
data being ’Library system’, ’Virtual environment’, and ’E-learning system’, but 5 chatbots were
’Stand-alone’ applications. This shows that the chatbot can be implemented into other systems,
as well as work on its own. The most used encompassing system was the E-learning system, with
5 chatbots as well. This may be because it is easy to keep all tools in one place, as well as giving
the chatbot easy access to the learning material and the students easy access to the chatbot. An
interesting finding in this group were the chatbots used as NPC in virtual worlds, the chatbot
working as the brain of the NPC and thus being able to communicate with the users avatar using
natural language. The chatbot used in the context of a library system was one of a kind, but
when looking at the general functionality of the system, it is simply a website containing access
to several databases, of which the chatbot was made a part. Thus the generalization could be
made that a chatbot might be implemented into a database heavy system and work well as its
front-end interaction point.

5.3.4 Bot Augmenting Tech

Three sub-themes were found in the scope of this theme, the researchers had improved the
chatbots’ capabilities by adding ’Linguistic tool’, ’Animation’, and ’Text-to-speech’. Not all
chatbots had used augmenting technology, which implied that the chatbot may be equally valuable
with or without it, the depending factor seems to be the general aim and goal of the chatbot
implementation. For example, the chatbot aimed at helping blind students would not be valuable
without the text-to-speech function.

5.3.5 Platform
The platform of the chatbot seems to have little relevance, other than it being connected to the
Internet in some way. The platforms used were either the ’Web’ or some ’Mobile OS’, or it was
not specified in the papers. The conclusion from this theme is thus that Internet connection is a
crucial function for any chatbot, since otherwise it will not be easy to reach for the students who
need to use it.

45
Chapter 6

Discussion

The results were interesting and despite the small dataset they showed a lot of diversity. The
publication details themes showed the most diversity, implying that the subject of chatbots in
education is of interest to a wide array of authors, institutions, and journals/conferences. It also
showed a broad interest and usability in the application of the chatbot technology, with the high
amount of unique keywords discovered in the papers. This finding was also backed by the high
amount of chatbot features discovered, really showing the diversity of the tool. The fact that the
findings here were so diverse also shows a lack of focused expertise from any single researcher,
institution or publication, no one seems has chosen the application of the chatbot technology in
education as their main field of interest. This of course excludes the three papers actually written
by almost the same authors, where then the opposite could be said, they chose to have this focus
and a clear development of knowledge and ideas can be seen through the three papers.

The educational context themes yielded less coherent findings, with fewer results than the other
theme-groups. The most important finding in this group was that most researchers had targeted
students or adults rather than children or teens, the reasons for this may be many, for example
students are a simple target since most researchers have access to a lot of them already. An-
other reason might be that despite the chatbots natural language proficiency, it is quite bad at
interpreting incorrect spelling or grammar, something that is most likely more common among
children, making them a more difficult target group during this time of implementation. Another
interesting point of finding in this theme group was that only a few of the chatbots in the papers
had been tested in actual education as of the publication of the paper, this might mean that the
papers located in the search only depicted the development, and that other keywords or search
criteria are necessary to find the testing of the chatbots. It might also mean that the researchers
never got that far, but instead abandoned the research for other ventures.

The most important findings were located in the third theme group, ICT implementation details,
the theme group created to encompass all the details of the chatbot itself. Within this group
different possible roles of the chatbots were discovered and the capabilities of the chatbots in the
form of their features. Supporting factors of the chatbot were found as well, in encompassing
systems, chatbot augmenting technology and platforms. The findings imply that an AIML-based
chatbot may be used for many different purposes and aims, include many different features, work
well on it own or together with additional technology and be implemented on various platforms.

The roles a chatbot could take that were found in the data included: ’Tutor’, ’Student evaluator’,
’Questions & Answers’, ’Teacher Communication’, and ’Natural conversation’. The most common
of these were Tutor, closely followed by Questions & Answers. Both these roles are big in their
definitions, and may thus encompass a lot of different things, something that might be seen in the

46
large number of features these bots include. That tutor is a common role may be quite natural
since a tutor is practically an extension of the teacher, ultimately performing some of the same
tasks as a teacher might perform. The tutor role was always in combination with other roles,
indicating that the role involves multiple responsibilities, again, just as a teacher. The other roles
may be less frequently represented, but they were represented separate from the role of ’Tutor’
too, implying that for example the ’Student evaluation’ is a valuable role for a chatbot even
without the added responsibilities of a tutor. The chatbots role can thus be adapted to suit a
range of needs, something that is also clearly shown in the large range of features present in the
results.

The results reveled 17 features a chatbot might have, the most common being ’Answer stu-
dent questions’, ’Conversational strategies’, ’Hold topic specific conversation’ and ’Hold general
conversation’. This indicates that a chatbots’ main proficiency is still holding natural language
conversations or answering questions in natural language, the core skills of a chatbot and how it
all started with ELIZA and ALICE. It can be concluded that it is important to remember this
main skill when implementing a chatbot, as this is where both its greatest strength, simple natural
conversation, and weakness, understanding incorrect grammar or unusual patterns, seems to lie.
By keeping the focus on this main skill the chatbot can show its greatest applications, and the
weakness must be remembered in order to mitigate it as much as possible. If natural language
communication is not the key feature or requirement, another type of technology might be better
suited for that particular development.

Some of the reported roles or features would not have been possible without the use of the
augmenting technology like linguistic tools, text-to-speech or animation, or without implementing
the chatbot into a larger system, like an e-learning platform, a virtual environment or a database
collecting system such as that of a library. The results and analysis show that the AIML-based
chatbot is valuable without these additions, but also that the chatbot may perform even better
when they are added. When making these decisions it will be important to look at the role
the chatbot should take, what features it needs to include and then start thinking about if the
AIML chatbot can manage this on its own or if it requires additional technology. For example
if a requirement is for the chatbot to output speech obviously text-to-speech would have to
be included. Another example might be if the chatbots’ purpose is language learning, some
semantic tool to help the chatbot better handle spelling and grammar mistakes would most likely
be necessary in order to create the best pedagogical value.

A quite surprising but very interesting find was the fact that the chatbot could be used as an NPC
(non-player character) in a virtual world. The chatbot was used as the decision making entity or
brain of the NPC’s that the users avatars could interact with, creating an interactive learning
experience for the user. This kind of use of the chatbot may lead to even more applications of
it for learning, for example it could be used as a guide in an historical virtual world to who the
listeners can ask questions, or as a companion that gives clues in a puzzle game that promotes
logical thinking.

47
Chapter 7

Conclusion

The thesis ends with the conclusion which looks at the research questions and the aim, and
summarizes the found answers and the key points of the discussion. Future research is also
presented and discussed.

The aim of the thesis was to further the development of educational chatbots by reviewing
what had been done and summarize this knowledge. This would be done by finding out what
capabilities the chatbot might have in an educational context and if it could stand on its own or if
it required additional technology to add pedagogical value in education. This aim was boiled down
to the open-ended questions: What roles can AIML-based chatbots take in an educational
context? and What features can AIML-based chatbots have in an educational context?.
These question has been answered thoroughly in this thesis but the answer might be summarized
as: The features and uses of AIML-based chatbots are many, and they seem only to be limited by
what we humans might imagine it doing. It is a flexible and diverse tool, simultaneously simple
and complex, limited only by the inventiveness of its creator. It can work well on its own and
provide educational value, but may reach even better results when combined with other technology.

A more specific answer might be that a chatbot can be used as a tutor, a student evaluator, for
questions and answers, to communicate with a teacher or simply for natural conversation. The
applications of these roles seem endless, and the 17 features (see table 7.1) found in this work
is most likely only a subset of the things chatbots can really do. The chatbots capabilities can
be expanded by including it in other systems such as e-learning systems, virtual environment
or library system (or other database heavy systems) or by adding augmenting technology like
text-to-speech technology, linguistic tools or animation.

Demonstrate learning task Support feedback to students

Animated gestures supporting knowledge base Give recommendations about learning material
Allow input as speech Hold topic specific conversation
Control e-learning platform’s activity Hold general conversation
Provide means of contact with teacher Teacher can edit bot content
Point of interaction for learner Ask the student questions
Provide output as speech Conversational strategies
Help students 24/7 Answer student questions
Provide content from other knowledge sources

Table 7.1: Chatbot features

48
7.1 Future research
As technology in education is ever changing, the chatbot may well have a chance to make a
real impact in the field. This work might be a stepping stone for the researchers looking to
make that possibility into reality. From this point there is a lot of possible future research to
suggest. For example, to look more into the actual pedagogical benefits of the different chatbot
implementations, this would be both important and interesting, as only a few of the works reviewed
in this work actually go on to perform that step. Another possible research idea might be to
implement a chatbot system that is easy to adapt to the current subject or course, so that the
teacher managing the chatbot could do so without the requirement of development skills. It
would also be interesting to investigate if a chatbot might bring something completely new to the
table of pedagogics, rather than just being a substitute for a teacher or an interactive FAQ, one
examples of this might be a study buddy, like an assistant keeping track of tasks, submissions and
results, in a more personal way than a common calendar.

A separate line of future research might be to look more at what the chatbot could bring to the
virtual world and if this could become a valuable pedagogical tool. The chatbot as an NPC is an
interesting concept that could well be used not only in the field of education, but maybe even
more in the field of entertainment. This idea again shows how diverse the uses of the chatbot
technology can be.

49
Bibliography

Allison, DeeAnn (2012). “Chatbots in the library: is it time?” In: Library Hi Tech 30.1, pp. 95–107.
Braun, Virginia and Victoria Clarke (2006). “Using thematic analysis in psychology”. In: Qualitative
research in psychology 3.2, pp. 77–101.
Cambridge dictionary (2018). Tutor. https://dictionary.cambridge.org/dictionary/english/tutor.
[Online; accessed 2018-05-12].
Collins English Dictionary (2018). Tutor. http://www.dictionary.com/browse/tutor. [Online;
accessed 2018-05-12].
Copeland, B.J (2018). Artificial Intelligence. https://www.britannica.com/technology/artificial-
intelligence. [Online; accessed 2018-05-02].
Deryugina, OV (2010). “Chatterbots”. In: Scientific and Technical Information Processing 37.2,
pp. 143–147.
Doering, Aaron, George Veletsianos, and Theano Yerasimou (2008). “Conversational agents and
their longitudinal affordances on communication and interaction”. In: Journal of Interactive
Learning Research 19.2, p. 251.
Floridi, Luciano, Mariarosaria Taddeo, and Matteo Turilli (2009). “Turing’s imitation game: still
an impossible challenge for all machines and some judges—-an evaluation of the 2008 Loebner
contest”. In: Minds and Machines 19.1, pp. 145–150.
Fonte, Fernando A Mikic, Martı́n Llamas Nistal, Juan C Burguillo Rial, and Manuel Caeiro
Rodrı́guez (2016). “NLAST: A natural language assistant for students”. In: Global Engineering
Education Conference (EDUCON), 2016 IEEE. IEEE, pp. 709–713.
Gang, Wei Yun, Sun Bo, Sun Ming Chen, Zhao Cui Yi, and Ma Pei Zi (2014). “Chinese intelligent
chat robot based on the AIML language”. In: Sixth International Conference on Intelligent
Human-Machine Systems and Cybernetics (IHMSC), 2014. Vol. 1. IEEE, pp. 367–370.
Heller, Robert and Mike Procter (2009). “Animated pedagogical agents: The effect of visual
information on a historical figure application”. In: International Journal of Web-Based Learning
and Teaching Technologies 4.1, p. 54.
Ibáñez, Marı́a Blanca, José Jesús Garcı́a Rueda, Sergio Galán, David Maroto, Diego Morillo, and
Carlos Delgado Kloos (2011). “Design and implementation of a 3D multi-user virtual world
for language learning.” In: Educational Technology & Society 14.4, pp. 2–10.
Jennifer Lachs (2017). Will chatbots revolutionise education?
https://www.opencolleges.edu.au/informed/features/will-chatbots-revolutionise-education/. [On-
line; accessed 2018-05-28].
Kerly, Alice, Phil Hall, and Susan Bull (2007). “Bringing chatbots into education: Towards
natural language negotiation of open learner models”. In: Knowledge-Based Systems 20.2,
pp. 177–185.
Kumar, M Naveen, PC Linga Chandar, A Venkatesh Prasad, and K Sumangali (2016). “Android
based educational Chatbot for visually impaired people”. In: International Conference on
Computational Intelligence and Computing Research (ICCIC), 2016 IEEE. IEEE, pp. 1–4.
Levy, Yair and Timothy J Ellis (2006). “A systems approach to conduct an effective literature
review in support of information systems research”. In: Informing Science 9.

50
Loebner, H. (1994). “In Response”. In: Communications of the ACM 37.6, pp. 79–82.
Loebner.net (2017). What is the Loebner Prize? https://bit.ly/2G3EgfQ. [Online; accessed
2018-05-02].
Masmuzidin, Masyarah Zulhaida and Taoran Wan (2012). “V-Penglipur Lara: The development
of a pedagogical agent in Malaysian folktales land”. In: Proceedings of the 11th International
Conference on Interaction Design and Children. ACM, pp. 280–283.
Merriam-Webster (2018). Tutor. https://www.merriam-webster.com/dictionary/tutor. [Online;
accessed 2018-05-12].
Mikic, Fernando A, Juan C Burguillo, Daniel A Rodrı́guez, Eduardo Rodrı́guez, and Martı́n Llamas
(2008). “T-Bot and Q-Bot: A couple of AIML-based bots for tutoring courses and evaluating
students”. In: 38th ASEE/IEEE Frontiers in Education Conference, 2008. IEEE, S3A–7.
Mikic, Fernando A, Juan C Burguillo, Martı́n Llamas, Daniel A Rodrı́guez, and Eduardo Rodrı́guez
(2009). “Charlie: An aiml-based chatterbot which works as an interface among ines and
humans”. In: EAEEIE Annual Conference, 2009. IEEE, pp. 1–6.
Orlando, De Pietro and Frontera Giovanni (2008). “An integrated system, with natural language
management, for the monitoring activities in e-learning environments”. In: International
Conference on Complex, Intelligent and Software Intensive Systems (CISIS) 2008. IEEE,
pp. 965–970.
Oxford dictionaries (2018). Tutor. https://en.oxforddictionaries.com/definition/tutor. [Online;
accessed 2018-05-12].
Pandorabots (2018a). AIML fundamentals. https://pandorabots.com/docs/aiml/aiml-fundamentals.html.
[Online; accessed 2018-05-15].
— (2018b). Artificial Intelligence Markup Language (AIML). https://pandorabots.com/docs/aiml/aiml-
basics.html. [Online; accessed 2018-05-15].
Pappas, Christopher (2018). What Is A Learning Management System? LMS Basic Functions And
Features You Must Know. https://elearningindustry.com/what-is-an-lms-learning-management-
system-basic-functions-features. [Online; accessed 2018-05-08].
Ranoliya, Bhavika R, Nidhi Raghuwanshi, and Sanjay Singh (2017). “Chatbot for University Related
FAQs”. In: 2017 International Conference on Advances in Computing, Communications and
Informatics (ICACCI). Udupi, pp. 1525–1530.
Rossi, Pier Giuseppe and Simone Carletti (2011). “MAPIT: a pedagogical-relational ITS”. In:
Procedia Computer Science 3, pp. 820–826.
Searle, John R (1980). “Minds, brains, and programs”. In: Behavioral and brain sciences 3.3,
pp. 417–424.
Shawar, Bayan Abu and Eric Atwell (2007). “Chatbots: are they really useful?” In: Ldv forum.
Vol. 22. 1, pp. 29–49.
Soliman, Mohamed and Christian Guetl (2013). “Implementing Intelligent Pedagogical Agents
in virtual worlds: Tutoring natural science experiments in OpenWonderland”. In: Global
Engineering Education Conference (EDUCON), 2013 IEEE. IEEE, pp. 782–789.
Strauss, Anselm and Juliet Corbin (1998). Basics of qualitative research: Techniques and procedures
for developing grounded theory. Thousand Oaks, CA, US: Sage Publications, Inc.
The free dictionary (2018). Tutor. https://www.thefreedictionary.com/tutor. [Online; accessed
2018-05-12].
The Oxford Dictionary (2018). chatbot. https://en.oxforddictionaries.com/definition/chatbot.
[Online; accessed 2018-04-24].
Travis, Tiffini (2008). “What Is E-Learning?” In: E-learning and business plans: National and
international case studies, pp. 19–36.
Turing, AM (1950). “Mind”. In: Mind 59.236, pp. 433–460.

51
University of Southern California (2018). Organizing Your Social Sciences Research Paper: 5. The
Literature Review. http://libguides.usc.edu/writingguide/literaturereview. [Online; accessed
2018-05-15].
Weizenbaum, Joseph (1966). “ELIZA—a computer program for the study of natural language
communication between man and machine”. In: Communications of the ACM 9.1, pp. 36–45.

52
 Heliyon 9 (2023) e19517

Contents lists available at ScienceDirect

Heliyon
journal homepage: www.cell.com/heliyon

Research article

Chatbot for communicating with university students in emergency

situation
Antonio Balderas a,∗ , Roberto Fermín García-Mena a , Milagros Huerta b ,
Nestor Mora c , Juan Manuel Dodero a
a
Departamento de Ingeniería Informática, Escuela Superior de Ingeniería, Puerto Real, 11519, Cádiz, Spain
b
Departamento Ingeniería Mecánica y Diseño Industrial, Escuela Superior de Ingeniería, Puerto Real, 11519, Cádiz, Spain
c
Departamento Ingeniería en Automática, Electrónica, Arquitectura y Redes de Computadores, Escuela Superior de Ingeniería, Puerto Real, 11519,
Cádiz, Spain

A R T I C L E I N F O A B S T R A C T

MSC: Chatbots have arrived in higher education, and professors are trying to make the most of them.
0000 Typically, chatbots are used to help students learn academic subjects. In times of crisis, such
1111 as the COVID-19 pandemic, students who were not living with their families during the course,
especially international students, were isolated and in critical situations. The student services
Keywords:
offices were in constant contact with these students to solve problems, advise them and support
Chatbot
them during their stay, within the constraints of confinement and the guidelines dictated by
Students
Emergency situation the country at the time. The student services offices were overwhelmed trying to help these
COVID-19 students because, although the students’ problems were very recurrent, the government guidelines
changed from one day to the next. This article proposes the use of a chatbot to provide initial
support to students during crisis situations, and facilitate communication between them and the
university. The chatbot was tested by more than 160 students and student services staff. The
findings support the use of chatbots as a potential tool to facilitate communication with students
in emerging emergency situations, and encourage universities to adopt these types of smart tools
to be prepared to respond quickly and efficiently to students in times of crisis.

1. Introduction

The COVID-19 crisis demonstrated that higher education institutions were not digitally prepared to respond to the needs of their
students [1]. Therefore, the pandemic and confinement has been taken advantage of by universities to carry out a necessary digital
transformation, expanding the catalogue of tools and methodologies to support blended learning [2].
However, the problem of maintaining teaching in higher education was not the only one that needed to be addressed for students.
The COVID-19 crisis and related measures resulted in negative effects on students’ social networks and mental health [3]. Students,
as well as general citizens, were faced with an unfamiliar situation in which there were specific prohibitions to comply with due to
the extraordinary measures. A particularly problematic case was that of students who were not living at their family house during
the course, especially international students, as they were not close to their family and friends, and could be much more affected

* Corresponding author.
E-mail address: [email protected] (A. Balderas).

https://doi.org/10.1016/j.heliyon.2023.e19517

Available online 30 August 2023

2405-8440/© 2023 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license
(http://creativecommons.org/licenses/by/4.0/).
A. Balderas, R.F. García-Mena, M. Huerta et al. Heliyon 9 (2023) e19517

by not having the necessary social support to cope with the stressful situation [4]. Universities made efforts to communicate with
and help these students through message groups, telephone support, etc. The student services offices were very busy as the borders
were closed and these students could not even consider returning home and the universities had to support them. These staff would
have needed some kind of tool to deal with these students, because although many of the students’ issues were similar, government
guidelines on what could and could not be done changed from one day to the next, and students needed to be kept up to date.
This research proposes the use of a chatbot to facilitate communication with student services offices and support students in
emergency situations in the future. Chatbots are technological artefacts that interact with users through natural language, and are
generally designed to provide information to users in a specific area. Chatbots were adopted after the COVID-19 pandemic to support
students’ autonomous learning [5,6]. Our chatbot has been implemented using Google’s DialogFlow technology and has been tested
by both the staff of the student services offices from six European universities and students from a university in southern Spain. In
order to be prepared for a new emergency situation, the aim of this work is to assess students’ perceptions of the use of chatbots as
a means of communicating with student services in such a situation.
This research is conducted within the Research for Innovative Practices in Emergency Management of Erasmus (RIPEC) project.
The project aims to compile the experiences of university staff and students in order to identify the main issues and identify innovative
solutions for Erasmus mobility in case of a new crisis.
The rest of the article is organised as follows. Next, the background is presented. Then, the technical details of the chatbot are
presented. Fourth, the results of the evaluation are presented. Finally, it concludes with a discussion and the conclusions of the
research.

2. Background

Virtual applications were essential during the pandemic, allowing society to continue its activities in many sectors despite the
lockdown and subsequent distancing measures. In this sense, the higher education sector was able to continue its activities, rapidly
adapting its infrastructures in order to continue teaching in the best possible conditions. However, aspects such as support to stu-
dents who live away from home in emergency situations may have been neglected, with student services offices having to assist these
students by providing information and support in their particular circumstances. Students had to cope with numerous uncertainties,
including whether they would be able to return home given the restrictions on travelling, whether they could leave the accommo-
dation to shop or eat, or even whether they could go to the doctor. In addition, they suffered the psychological effects of being
isolated from other people, compounded by the fact that they had no family nearby. Student services offices were overwhelmed by
the number and variety of enquiries they received from students during the emergency. This background is divided into two sections;
firstly, we present research that uses technology to support students in emergency situations. Secondly, we present research that uses
chatbots to support students in higher education.

2.1. Student support software for emergencies

Higher education institutions have been able to continue operating because they mostly followed a blended learning approach,
with a virtual learning environment (VLE) for their courses [7]. VLE provide teaching staff with numerous tools to communicate
with students, such as forums, portfolios, chats or the messaging system of the virtual campus itself. These tools supported the
continuation of higher education, although they were mainly used for academic subjects and students were negative about their lack
of interactivity [8].
One of the tools that provide more interactive communication, and whose use has grown significantly since lockdown, is video
conferencing [9]. These integrate seamlessly with virtual courses and allow teachers to continue with their classes [10]. In this
study, psychology students used videoconferencing to conduct psychotherapy during the lockdown, demonstrating that this tool can
be used to communicate and treat their patients [11]. However, the literature also suggests that videoconferencing can cause anxiety
in students. [12], and in general students are reluctant to connect their camera when they are off campus. [13]. Another emergency
scenario in which video conferencing has been used to help people can be found in the context of the invasion of Ukraine. In this
work, video conferencing has been used to assist in surgical operations [14].
The problem faced by the student services offices was the difficulty in handling communications, finding an adequate and timely
response to the students who needed it, some of them with specific problems, others common to many. One of the solutions was to
use broadcasting groups in applications such as Whatsapp or Telegram [15]. This study defined profiles in the use of social networks
by international students during the period of confinement and claimed that students facing psychological challenges communicate
more online than others [16].
A tool that allows communication with a large number of students in a scalable and usable way through a communication channel
such as Telegram is a chatbot. In the next sub-section we will explore approaches for using chatbots to help students.

2.2. Chatbots to assist students

There is already research in educational technology that has used chatbots in higher education subjects, with satisfactory results
in helping students improve their academic performance [17,18,5]. Students who have been able to use a chatbot in some of their
subjects are discovering the benefits of chatbots and are requesting that they be made available in other aspects of their academic
life [19].

2
A. Balderas, R.F. García-Mena, M. Huerta et al. Heliyon 9 (2023) e19517

Fig. 1. Architecture of the components integrated in the design of the chatbot.

Along the same lines, other work has demonstrated the effectiveness of chatbots in helping people communicate in difficult
situations. CapacitaBOT is an educational chatbot that aims to help users with intellectual disabilities to understand, control and
improve their social skills [20]. SPeCECA is a first aid chatbot designed to help people respond well to health emergencies [21]. This
research shows how the processes of empathy and resilience through the interaction between the student and the chatbot could help
to cope with the COVID-19 disorder [22].
COVID-19 has also led to a growth in the use of chatbots to support students. The students in Kohnke’s research indicated that
being able to interact with a chatbot reduced their sense of isolation and had a positive impact on their learning [23]. The work
presented by Tiwari et al. used a chatbot to answer questions about COVID-19 [24], while the chatbot introduced by Kim and Riio
was used to promote social distancing [25].
Unfortunately, the implementation details of chatbots are not published and most are based on a series of questions and answers,
similar to a list of frequently asked questions (FAQ), and can hardly provide the context of a conversation similar to the one we
would have with a human.

3. Chatbot

The chatbot to assist students in emergency situations is developed within the framework of the RIPEC project. This section first
describes the architecture of the system that comprises the chatbot. Secondly, it presents the implementation of the assistant to
converse and provide the appropriate responses. Thirdly, the configuration of the chatbot memory system is shown.

3.1. System architecture

The chatbot was designed and built using Dialogflow [26]. Dialogflow is a platform developed by Google that makes it easy to
build conversational bots. It provides a pre-configured platform and interface, as well as easy integration with other services. The
chatbot is available in a GitHub repository.1
Two interfaces were implemented for the chatbot: a chat integrated into the RIPEC website and another one using Telegram.
Dialogflow Messenger was utilized to incorporate the chatbot into the website, which offers an HTML code that needs to be inserted
into the website’s code. This code generates a floating button that, when clicked, opens a chat window. As for the Telegram bot, it
was created using a separate bot called BotFather. This bot generates a unique token that identifies the new bot. The token should be
entered in a text box within the Dialogflow integrations screen.
The architecture of the system that builds the chatbot is shown in Fig. 1. This figure shows how the student will be able to talk
to the chatbot via the chat enabled on the project website (1) or via the instant messaging application Telegram (2). The student’s
message is then sent to be processed by the DialogFlow assistant, regardless of whether the message comes from the website chat
(3) or the Telegram chat (4). The assistant regularly updates its knowledge base entries via the FAQ entries on the project website

1
https://github.com/RIPEC/Dialogflow/wiki.

3
A. Balderas, R.F. García-Mena, M. Huerta et al. Heliyon 9 (2023) e19517

Fig. 2. Components that connect to DialogFlow and are involved in the elaboration of the response.

(5). The FAQ contains general information that would be of interest to any student, regardless of their university. If the student’s
question can be answered by its knowledge base, it will send the answer via its source channel (7) or (8). On the other hand, if it
does not find an answer in its knowledge base, it will try to manage the answers through the information contained in the database
(6). The information in this database is specific to each university, so the chatbot will have asked the student’s university beforehand
to provide the appropriate answer. Similarly, it will return the appropriate answer to the web chat (7) or the telegram (8), as the
case may be. Finally, the answer is presented to the user via web chat (9) or Telegram (10).

3.2. Language recognition and automatic response

When the assistant developed in DialogFlow receives a message from the student, it must provide a response to the student. The
DialogFlow components used for language recognition and automatic response are Knowledge Connectors and Intents (Fig. 2).

3.2.1. Knowledge connectors

A knowledge connector is a component within Dialogflow that enables the system to train itself by analysing different documents
written in natural language. This method is particularly useful for incorporating multiple question-answer pairs, making it ideal
for creating interactive FAQ websites. In this project, the Knowledge Connector used is based on a FAQ section from the official
RIPEC website. By parsing the web page, Dialogflow extracts the question/answer pairs. A major advantage of this approach is the
automated updating process. The system periodically scans the website for changes, ensuring that the knowledge base is kept up to
date without any additional manual effort.
Knowledge connectors are useful when information can be structured in question-answer form, but they have limitations when it
comes to providing tailored responses for each situation. To provide personalised answers and build more complex conversations, it
is necessary to use another tool called Intent.

3.2.2. Intents
Intents are used to categorise user interactions so that responses can be tailored to each conversation. A more complex chatbot
will have more intents. A basic Intent contains the following information:

• Training Phrases: A set of phrases that the user can use. It is unnecessary to include variants of the same phrase. However, if
relevant parameters need to be extracted from interactions, it is useful to include examples of sentences in which they are used.
In the Fig. 3, you can see the training phrases for the intent Medical Care.
• Parameters: the system can search for relevant data in interactions. Parameters are classified according to the type of entity they
represent. DialogFlow provides many default entity types (e.g. cities, dates or numbers), but it is possible to create custom ones.
For this chatbot, entities were created to recognise names of universities or specific data of the different services (e.g. phone
numbers, emails or opening hours). In Fig. 3, you can also see how the parameters are highlighted in the training sentences.
• Responses: information sent back to the user. The extracted parameter values can be used within the responses to reassure the
user that they have been captured correctly.

In addition, the Intent includes some additional features to add complexity to conversations and responses. The following were
used in this project

• Context: a structure that contains information relevant to contextualise the conversation. This chatbot collects in context the
university where the student is studying or the student’s country of origin.
• Fulfillment: functionality in Dialogflow that enables the integration of webhook services. When a message matches an Intent
with Fulfillment activated, a request containing all the retrieved data is sent to the associated webhook service. A webhook is
an HTTP callback that allows the NLP service to use a specific function defined in the webhook to make data processing more
flexible [27]. The webhook service is responsible for processing the response based on the information available.

4
A. Balderas, R.F. García-Mena, M. Huerta et al. Heliyon 9 (2023) e19517

Fig. 3. Training phrases for the Medical care intent.

Twenty two intents have been implemented for this chatbot with the following names: Academic, Ask country, Ask service,
Ask university, Default Fallback Intent, Add follow-up intent, Default Welcome Intent, Digital platform, Embassy information, Eras-
mus buddy program, Erasmus office, Goodbye, Housing, International airport, Library information, Local transport, Medical Care,
Municipality, Psychological attention, Security, Service map, Student attention and, Visa service.

3.3. Webhooks and information management

Two Google services were used to create the webhooks for this chatbot: Google Apps Script and Google Drive.

3.3.1. Google Apps Script

Apps Script is a platform for developing applications that can be easily integrated with the company’s most popular services (such
as Google Mail, Drive or Calendar). One of the most common uses of this service is to create macros to automate processes, but
the same system also allows the creation of web applications. To handle the functionality of the chatbot, a service was necessary
that could handle POST requests and respond with a JSON payload containing structured information. This information is critical for
Dialogflow to process and generate appropriate responses for the bot user. The chatbot’s programming language is primarily based on
JavaScript, complemented by specific libraries tailored to interact with the various services involved in its operation. To implement
the required behaviour, a function called doPost (a reserved name for this particular purpose) must be created, taking an input
parameter to store the information from the POST request. The function should return a special type object called TextOutput, which
contains the reply information for the request. Finally, the application needs to be deployed to make it operational and accessible.

3.3.2. Google Drive

The decision to use Google Sheets as the storage platform for the bot’s information was driven by several factors:

• Ease of use: Google Sheets provides a user-friendly interface that is familiar to most people. This was an advantage when staff
from different universities involved in the project had to enter some of the information.
• Manageable amount of data: As the intention was not to handle a large amount of information, a complex storage system was
unnecessary. Google Sheets provided a suitable solution for the project’s needs.
• Seamless integration with Google Apps Script: Google Apps Script, which uses libraries that enable easy communication with
Google Sheets, showed good compatibility with the spreadsheet service.

To ensure organised data management, the document consists of several sheets dedicated to storing information about countries,
embassies, universities and the services they provide. In addition, to prevent errors, certain cells containing critical information
essential to the operation of the system were locked. This ensures that these cells only accept a pre-defined set of values, guaranteeing
consistent naming conventions for services and universities throughout the rows in which they are referenced.

5
A. Balderas, R.F. García-Mena, M. Huerta et al. Heliyon 9 (2023) e19517

Fig. 4. Illustrative Example 1 – conversation between the chatbot and a student using the chat within the website.

Fig. 5. FAQ entry matching the student’s question from use case 1.

It is the responsibility of each university’s student services staff to complete their university’s service information in this database
so that the chatbot can respond to students with information specific to their university when they request it.

4. Evaluation

This evaluation will assess the chatbot to facilitate communication between student services offices and students in emergency
situations in the future. For this purpose, firstly, some illustrative examples are introduces to show how the interactions between
students and the chatbot look like. Secondly, the results of the evaluation carried out with users are shown.

4.1. Illustrative examples

In Illustrative Example 1 (Fig. 4), we show a use of the chatbot from the chat integrated into the RIPEC project website. We have a
student from the University of Tuscia (UNITUS) who is feeling anxious and is talking to the chatbot to see who he/she can contact.
If you look at the screenshot on the left, you can see that the chatbot starts with a greeting and the student indicates that he/she
is anxious. At this point, the DialogFlow assistant checks to see if there are any entries in its knowledge base that can be answered
based on the information provided. It finds a related entry that matches the entry “i feel anxious” in the FAQ (Fig. 5). Based on the
content of the FAQ, the chatbot recommends that the student contact the university’s psychological support service.
The student then asks for the contact details of the psychological attention office. The chatbot does not find any FAQ entry that
matches this question and tries to answer through the intents, matching the student’s entry with the intent Psychological attention.
The chatbot asks for the name of the university to establish this parameter, to set the context of the conversation and to direct the
student to the specific services offered by his or her institution. The student indicates that his/her university is UNITUS. The chatbot
then searches the database for information about UNITUS’ psychological service and provides this information to the student.
In Illustrative Example 2 (Fig. 6), we show the use of the chatbot through the Telegram messaging application. In this case, a
foreign student studying at the University of Cádiz (UCA) wants to know if he/she can travel to his/her home country, so he/she asks
the chatbot the question. The chatbot finds an entry in its knowledge base that matches the entry “Can I travel to my home country?”
(Fig. 7) and invites the student to consult a web address.

6
A. Balderas, R.F. García-Mena, M. Huerta et al. Heliyon 9 (2023) e19517

Fig. 6. Illustrative Example – conversation between the chatbot and a student using the Telegram application.

Fig. 7. FAQ entry matching the student’s question from use case 2.

After consulting the website, the student wants to know which airport he/she can fly from and asks the chatbot for this informa-
tion. The chatbot has no knowledge base entry on this topic, so the intent International airport matches the question and the chatbot
asks the student about his home university to provide the specific airport. The chatbot queries its database and returns to the student
the airports registered as being close to the UCA.

4.2. Evaluation with users

This study was approved by the Ethics Committee in Non-Medical Experimentation and Genetically Modified Organisms of the
University of Cádiz, approval number 005_2023. Additionally, informed consent was obtained from the participants in this research,
whose participation was completely voluntary. The anonymity of the participants was maintained at all times.

7
A. Balderas, R.F. García-Mena, M. Huerta et al. Heliyon 9 (2023) e19517

Table 1
Survey, summary of results for items: identifier, category, question and average rating for staff and stu-
dents.

Id Cat. Items Staff Students

Q01 PA The chatbot function was easily detectable 4.00 3.65

Q02 PA It was easy to find the chatbot 4.00 3.36
Q03 PQ Communicating with the chatbot was clear 4.07 3.77
Q04 PQ I was immediately made aware of what 3.85 3.55
information the chatbot can give me
Q05 PQ The interaction with the chatbot felt like an 3.92 3.65
ongoing conversation
Q06 PQ The chatbot was able to keep track of context 4.07 3.72
Q07 PQ The chatbot was able to make references to the 3.78 4.06
website or service when appropriate
Q08 PQ The chatbot could handle situations in which the 3.71 3.22
line of conversation was not clear
Q09 PQ The chatbot’s responses were easy to understand 4.50 3.99
Q10 PC I find that the chatbot understands what I want 3.92 3.67
and helps me achieve my goal
Q11 PC The chatbot gives me the appropriate amount of 3.85 3.76
information
Q12 PC The chatbot only gives me the information I need 3.92 3.56
Q13 PC I feel like the chatbot’s responses were accurate 3.92 3.70
Q14 PPS The interaction with the chatbot felt secure in 4.51 3.50
terms of privacy
Q15 TR My waiting time for a response from the chatbot 4.50 4.21
was short

This study adopts Oates et al.’s design and creation research strategy [28], in which we developed an IT artefact, specifically a
chatbot, to investigate the perceptions of affected users in an academic context regarding the use of a chatbot in emergency situations.
To evaluate the chatbot, a questionnaire was used as one of the accepted methods in this design and development strategy. For this
purpose, the evaluation was carried out using the Bot Usability Scale (BUS) designed for chatbot users by Borsci et al. [29]. This
questionnaire provides a list of attributes specifically developed to measure satisfaction with chatbots. The BUS questionnaire consists
of 15 items divided into the following categories:

• Perceived accessibility (PA): Q01-Q02

• Perceived quality (PQ): Q03-Q09
• Perceived quality of conversation and information received (PC): Q10-Q13
• Perceived privacy and security (PPS): Q14
• Response time (TR): Q15

The items in the questionnaire are rated on a 5-point Likert scale, with 1 being the lowest (strongly disagree) and 5 being the
highest (strongly agree).
The evaluation with users was carried out in two different scenarios.

4.2.1. Scenario I: staff of the student services offices

Firstly, the chatbot was made available to the staff in charge of managing Erasmus students in 6 European universities: Uni-
versity of Tuscia (Italy), Scuola Superiore Carolina Albasio (Italy), Universidad de Cádiz (Spain), Panteio Panepistmio Koinonikon
Kaipolitikon Epistimon (Greece), Wyzsza Szkola Kultury Spolecznej i Medialnej (Poland) and University Univerzita Jana Evangelisty
Purkyne V Usti Nad Labem (Czech Republic). The evaluation of the chatbot was carried out as follows:

• Step 1: staff from these universities attended a presentation of the chatbot, which took place at one of the RIPEC project meetings.
• Step 2: after the meeting, they had two weeks to enter information from their universities into the chatbot database. Among the
6 universities, they created a total of 105 registers with services and information about their universities.
• Step 3: staff were asked to interact with the chatbot by posing different scenarios to it, and then to answer the BUS questionnaire.
This evaluation was presented in an earlier paper [30] and is summarised in Table 1 for comparison with the students’ evaluation.

4.2.2. Scenario II: students

Secondly, the chatbot was tested by 165 university students. The chatbot was introduced in several sessions to groups of 15 to 30
undergraduate and graduate students. All sessions followed the same structure:

8
A. Balderas, R.F. García-Mena, M. Huerta et al. Heliyon 9 (2023) e19517

Table 2
Chatbot acceptance data.

Gender Age Chatbot Use Erasmus Family Home

Chatbot acceptance 0.475 0.165 0.793 0.529 0.403

Table 3
Data for questionnaire dimensions.

Gender Age Chatbot Use Erasmus Family Home

PA 0.045 0.975 0.518 0.104 0.985

PQ 0.161 0.723 0.670 0.947 0.186
PC 0.297 0.168 0.439 0.649 0.191
PPS 0.690 0.462 0.392 0.535 0.099
TR 0.735 0.518 0.246 0.315 0.623

• Step 1: the chatbot was introduced to the students and different use cases were shown, similar to those shown in Figs. 4 and 6.
• Step 2: students had time to interact with the chatbot, resulting in a total of 405 interactions.
• Step 3: they completed the BUS questionnaire to evaluate the usefulness and usability of the chatbot.

4.2.3. Results
The mean ratings of the staff and student evaluations of the BUS questionnaire are shown in the Table 1, columns four and five
respectively.
In terms of perceived accessibility (PA), students’ evaluation is above average, both in terms of the ease of recognising the
functionality of the chatbot (3.65 for Q01) and the clarity of communication with the chatbot (3.36 for Q02). However, it is worth
noting that the evaluation is slightly lower than that of the staff of the student services offices (4.00 for both Q01 and Q02).
In terms of perceived quality (PQ), students’ evaluations are also above average for all items (Q03-Q09). The items that students
rate most positively are the chatbot’s ability to reference websites or services when appropriate (4.06 for Q07) and the ease of
understanding the chatbot’s responses (3.99 for Q09). In the case of Q07, it is worth noting that the students’ evaluation was higher
than that of the staff of the student services offices (3.78 for Q07).
Regarding the perceived quality of the conversation and the information provided, the students’ evaluation is very balanced
across the 4 items (Q10-Q13), with ratings ranging from 3.56 to 3.76. There is a generally positive evaluation from the student body,
although it is also slightly lower than that of the staff of the student services offices.
In terms of privacy, the students’ evaluation is one point lower than that of the staff of the student services offices (3.50 vs. 4.51
for Q14), although it is still above the average, this is an aspect to consider when implementing the chatbot.
Finally, the response time rating is very good (TR), with a rating of 4.21 (Q15), the highest average of all the students’ items in
the questionnaire.

4.2.4. Validation
In order to check the validity of the survey, a Cronbach’s alpha test was carried out on all the items as a whole, with a high level
of confidence (0.94).
To test the significance of the data, we used the Chi-square test. As the data were collected using a 5-point Likert scale (BUS
questionnaire), we will use the Stanine scale to organise the data. We will use a global variable that is the sum of all the ratings of
the items of the BUS questionnaire and that reflects the acceptance of the chatbot by the students, i.e. chatbot acceptance.
Next, the validity of the results is analysed by looking at whether the chatbot acceptance is independent of gender, age, previous
use of chatbots, being an Erasmus student and living away from home.
As can be seen in the Table 2, all p-values are greater than the significance value 0.05, so the null hypotheses that there is no
relationship between student acceptance and respectively gender, age, having used chatbots previously, having been an Erasmus
student and living away from home during the course cannot be rejected.
Then, we assessed the dependence of these same variables with respect to each of the questionnaire dimensions. If we look at the
Table 3, we see that the p-value is always above the significance value 0.05, therefore, the null hypothesis cannot be accepted in any
case, except in the relationship between gender and perceived accessibility.
It is worth noting that students were able to give feedback on their impressions of the chatbot. This was collected via an open text
field at the end of the questionnaire. Only 14 opinions were collected, but they were mostly positive. For instance, they highlighted
its role as an intermediary that quickly and concisely directing you to any university website or phone number, and they highlighted
the speed of the chatbot. In the area of improvements, they indicated that it would be important to disseminate it so that students
could use it.
In this vein, we also analysed the students’ evaluations in order to implement a chatbot that facilitates the life of university
students beyond the management of emergency situations. This proposal received an evaluation of 4.17 out of 5 and, as can be seen
in the Table 4, the results were also independent of the variables examined, i.e. gender, age, having used chatbots previously, having
been an Erasmus student and living away from home during the course.

9
A. Balderas, R.F. García-Mena, M. Huerta et al. Heliyon 9 (2023) e19517

Table 4
Data chatbot implantation.

Gender Age Chatbot Use Erasmus Family Home

Chatbot implantation 0.799 0.700 0.986 0.215 0.656

5. Discussion

In this paper, we propose the use of a chatbot to facilitate communication between students and student services offices in
emergency situations, where protocols are evolving and users need to have up-to-date information. The tests conducted with the im-
plemented chatbot and the evaluation conducted with users show a positive perception of the users involved, students and university
staff, towards the usefulness of a chatbot for the intended purpose in the event of a new emergency situation.
The evaluation of the students has been positive in all dimensions, being always above average. This is the main objective, as they
were the most affected when they were isolated by the confinement and subsequent measures due to the COVID-19 pandemic [31].
However, in almost all items, these ratings were slightly below the rating of the university staff. This is because the staff were part
of the consortium in charge of developing the chatbot, so they are more knowledgeable about the problem.
On the other hand, the results have been validated regardless of whether students live away from home or not. Therefore, although
the problem originates in the attention to this type of students, the data support that other students also detect the benefits of the
chatbot and would consider using it if it were available.
The findings of this paper indicate that such chatbots can be used in future epidemics and emergency situations. It is the respon-
sibility of universities to get involved and maintain this type of project so that, when the time comes, the university will be prepared
to support its students [32,33].
In conclusion, we consider it essential and therefore recommend implementing this type of solution in universities to improve
communication and assistance to students, both in emergency situations and to obtain information on more common services of
the institution. On many occasions, students complain about the difficulty of finding information within university websites and use
Google instead of searching the university’s own web pages. With a chatbot on the university’s official website, students could be
assisted in navigating institutional sites.

5.1. Threats to validity

Despite the positive results of our evaluations, it is important to recognise that the validity of our findings may be affected by
certain factors.

• First, it should be noted that our study was conducted in a non-emergency setting. Therefore, it is necessary to test the chatbot
in a real emergency context in order to fully assess its effectiveness and applicability.
• The chatbot requires maintenance by the universities. This is not a problem because using the chatbot is as simple as maintaining
a FAQ and updating a spreadsheet-type database. However, university staff need to be convinced of its usefulness, and this can
sometimes be an obstacle [34,35].
• Although students do not provide information that identifies them, they do provide various pieces of information that could be
considered quasi-identifying and that, all together, could be used to identify the person [36]. Although the chatbot developed
does not collect logs, Google DialogFlow keeps a log for 400 days if you have logging enabled. And it is necessary to keep this
log enabled for the chatbot to learn. Therefore, it would be advisable to consider an implementation in an open source tool that
allows greater privacy and control by the university of this type of logs [37].

6. Conclusions

In universities, the use of chatbots has increased in recent years, as it allows teachers to facilitate students’ autonomous work
thanks to chatbot support. However, the use of these chatbots has been almost exclusively dedicated to teaching practice, and the
COVID-19 pandemic and the subsequent measures to restore normality have shown that the technology has not covered all the needs
of the student body.
This research proposes the use of a chatbot to provide information to students in emergency situations such as that experienced
during COVID-19. The chatbot has been implemented and tested by students and student services staff, and the evaluation conducted
shows positive evidence of the acceptance of the chatbot. The results show a positive perception of the effectiveness of chatbots in
improving communication between the university and students, and their ability to respond quickly and accurately to users’ questions
and needs. The results also suggest that student services offices can use chatbots for more effective and dynamic communication with
students, adaptable to changing regulations in emergency contexts. This type of tool would be particularly useful for students living
away from home during their course, especially those outside their home country and unable to travel due to current government
guidelines.
In future work, we suggest exploring the use of language models such as GPT to further improve the chatbot’s ability to understand
and respond to student questions more accurately and naturally, as well as implementing additional security measures to ensure user
privacy.

10
A. Balderas, R.F. García-Mena, M. Huerta et al. Heliyon 9 (2023) e19517

CRediT authorship contribution statement

Antonio Balderas: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data;
Contributed reagents, materials, analysis tools or data; Wrote the paper.
Roberto Fermín García-Mena: Performed the experiments; Contributed reagents, materials, analysis tools or data.
Milagros Huerta: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data.
Néstor Mora: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Wrote the
paper.
Juan Manuel Dodero: Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing
interests:
Juan Manuel Dodero reports financial support was provided by Spanish National Research Agency (AEI), Grant PID2020-
115844RB-I00. Nestor Mora reports financial support was provided by Erasmus+ Programme, Grant 2020-1-IT02-KA203-079711.

Data availability

Data will be made available on request.

Appendix A. Supplementary material

Supplementary material related to this article can be found online at https://doi.org/10.1016/j.heliyon.2023.e19517.

References

[1] J.M.F. Batanero, M.M. Rueda, J.F. Cerero, P. Tadeu, Online education in higher education: emerging solutions in crisis times, Heliyon (2022) e10139.
[2] F.J. García-Peñalvo, Avoiding the dark side of digital transformation in teaching. An institutional reference framework for elearning in higher education,
Sustainability 13 (4) (2021) 2023.
[3] M.J. Hossain, F. Ahmmed, S.A. Rahman, S. Sanam, T.B. Emran, S. Mitra, Impact of online education on fear of academic delay and psychological distress among
university students following one year of COVID-19 outbreak in Bangladesh, Heliyon 7 (6) (2021) e07388.
[4] T. Elmer, K. Mepham, C. Stadtfeld, Students under lockdown: comparisons of students’ social networks and mental health before and during the COVID-19 crisis
in Switzerland, PLoS ONE 15 (7) (2020) e0236337.
[5] A. Balderas, R. Baena-Pérez, T. Person, J.M. Mota, I. Ruiz-Rube, Chatbot-based learning platform for SQL training, Int. J. Interact. Multimed. Artif. Intell. (2022),
https://doi.org/10.9781/ijimai.2022.05.003.
[6] M. Co, T.H.J. Yuen, H.H. Cheung, Using clinical history taking chatbot mobile app for clinical bedside teachings–a prospective case control study, Heliyon 8 (6)
(2022) e09751.
[7] S. Dhawan, Online learning: a panacea in the time of COVID-19 crisis, J. Educ. Technol. Syst. 49 (1) (2020) 5–22.
[8] C.H. Karjo, W. Andreani, A. Herawati, Y. Ying, A.P. Yasyfin, K. Marie, A survey of teachers’ and students’ demands of ideal e-learning management system
features during the COVID-19 pandemic, in: 2021 International Conference on Advanced Computer Science and Information Systems (ICACSIS), IEEE, 2021,
pp. 1–5.
[9] S. Iglesias-Pradas, Á. Hernández-García, J. Chaparro-Peláez, J.L. Prieto, Emergency remote teaching and students’ academic performance in higher education
during the COVID-19 pandemic: a case study, Comput. Hum. Behav. 119 (2021) 106713.
[10] F.J. García-Peñalvo, A. Corell, V. Abella-García, M. Grande-de Prado, Recommendations for mandatory online assessment in higher education during the COVID-
19 pandemic, in: Radical Solutions for Education in a Crisis Context: COVID-19 as an Opportunity for Global Learning, Springer, 2020, pp. 85–98.
[11] J. Rowen, G. Giedgowd, A. Demos, Effectiveness of videoconferencing psychotherapy delivered by novice clinicians in a training clinic, Train. Educ. Prof.
Psychol. (2022).
[12] H. Mamtani, R. Karaliuniene, R. de Filippis, S. Nagendrappa, Impact of videoconferencing applications on mental health, BJPsych Int. 19 (1) (2022).
[13] J. Petchamé, I. Iriondo, G. Azanza, “Seeing and being seen” or just “seeing” in a smart classroom context when videoconferencing: a user experience-based
qualitative research on the use of cameras, Int. J. Environ. Res. Public Health 19 (15) (2022) 9615.
[14] J. Best, From Ukraine to remote robotics: how videoconferencing and next generation technology are transforming surgery, BMJ 377 (2022).
[15] K. Sarwari, A.F. Kakar, J. Golzar, M.A. Miri, Distance learning during COVID-19 in Afghanistan: challenges and opportunities, E-Learn. Digital Med. 19 (2)
(2022) 144–162.
[16] Y.A. Chen, T. Fan, C.L. Toma, S. Scherr, International students’ psychosocial well-being and social media use at the onset of the COVID-19 pandemic: a latent
profile analysis, Comput. Hum. Behav. 137 (2022) 107409.
[17] S.H.M. Daud, N.H.I. Teo, N.H.M. Zain, Ejava chatbot for learning programming language: a post-pandemic alternative virtual tutor, Int. J. Emerg. Trends Eng.
Res. 8 (7) (2020) 3290–3298, https://doi.org/10.30534/ijeter/2020/67872020.
[18] D. Lee, S. Yeo, Developing an AI-based chatbot for practicing responsive teaching in mathematics, Comput. Educ. 191 (2022) 104646.
[19] M. Ismail, A. Ade-Ibijola, Lecturer’s apprentice: a chatbot for assisting novice programmers, in: 2019 International Multidisciplinary Information Technology
and Engineering Conference (IMITEC), IEEE, 2019, pp. 1–8.
[20] M. Mateos-Sanchez, A.C. Melo, L.S. Blanco, A.M.F. García, Chatbot, as educational and inclusive tool for people with intellectual disabilities, Sustainability
14 (3) (2022) 1520.
[21] N. Ouerhani, A. Maalel, H. Ben Ghézela, Spececa: a smart pervasive chatbot for emergency case assistance based on cloud computing, Clust. Comput. 23 (4)
(2020) 2471–2482.
[22] Q. Jiang, Y. Zhang, W. Pian, Chatbot as an emergency exist: mediated empathy for resilience via human-AI interaction during the COVID-19 pandemic, Inf.
Process. Manag. 59 (6) (2022) 103074.
[23] L. Kohnke, A qualitative exploration of student perspectives of chatbot use during emergency remote teaching, Int. J. Mob. Learn. Organ. 16 (4) (2022) 475–488.

11
A. Balderas, R.F. García-Mena, M. Huerta et al. Heliyon 9 (2023) e19517

[24] V. Tiwari, L.K. Verma, P. Sharma, R. Jain, P. Nagrath, Neural network and NLP based chatbot for answering COVID-19 queries, Int. J. Intell. Eng. Inf. 9 (2)
(2021) 161–175.
[25] W. Kim, Y. Ryoo, Hypocrisy induction: using chatbots to promote COVID-19 social distancing, Cyberpsychol. Behav. Soc. Netw. 25 (1) (2022) 27–36.
[26] N. Sabharwal, A. Agrawal, Introduction to Google dialogflow, in: Cognitive Virtual Assistants Using Google Dialogflow, Springer, 2020, pp. 13–54.
[27] E. Handoyo, M. Arfan, Y.A.A. Soetrisno, M. Somantri, A. Sofwan, E.W. Sinuraya, Ticketing chatbot service using serverless NLP technology, in: 2018 5th
International Conference on Information Technology, Computer, and Electrical Engineering (ICITACEE), IEEE, 2018, pp. 325–330.
[28] B.J. Oates, M. Griffiths, R. McLean, Researching Information Systems and Computing, Sage, 2022.
[29] S. Borsci, A. Malizia, M. Schmettow, F. Van Der Velde, G. Tariverdiyeva, D. Balaji, A. Chamberlain, The chatbot usability scale: the design and pilot of a usability
scale for interaction with AI-based conversational agents, Pers. Ubiquitous Comput. 26 (1) (2022) 95–119.
[30] R. García-Mena, A. Balderas, M. Huerta, J. Dodero, N. Mora-Nuñez, Chatbot to provide assistance to erasmus students in emergency situations, in: F.J. García-
Peñalvo, A. García-Holgado (Eds.), Proceedings TEEM 2022: Tenth International Conference on Technological Ecosystems for Enhancing Multiculturality, TEEM
2022, in: Lecture Notes in Educational Technology, 2023.
[31] H. Blake, H. Knight, R. Jia, J. Corner, J.R. Morling, C. Denning, J.K. Ball, K. Bolton, G. Figueredo, D.E. Morris, et al., Students’ views towards Sars-Cov-2 mass
asymptomatic testing, social distancing and self-isolation in a university setting during the COVID-19 pandemic: a qualitative study, Int. J. Environ. Res. Public
Health 18 (8) (2021) 4182.
[32] D. Burns, N. Dagnall, M. Holt, Assessing the Impact of the COVID-19 Pandemic on Student Wellbeing at Universities in the United Kingdom: A Conceptual
Analysis, Frontiers in Education, vol. 5, Frontiers Media SA, 2020, 582882.
[33] W. Ali, Online and remote learning in higher education institutes: a necessity in light of COVID-19 pandemic, High. Educ. Stud. 10 (3) (2020) 16–25.
[34] C.I. Maican, A.-M. Cazan, R.C. Lixandroiu, L. Dovleac, A study on academic staff personality and technology acceptance: the case of communication and
collaboration applications, Comput. Educ. 128 (2019) 113–131.
[35] A. Shankar, A. Nigam, Explaining resistance intention towards mobile HRM application: the dark side of technology adoption, Int. J. Manpow. 43 (1) (2022)
206–225.
[36] M. García-Rodríguez, A. Balderas, J.M. Dodero, Data mashups privacy preservation for learning analytics, in: Proceedings of the Learning Analytics Summer
Institute Spain 2022 (LASI Spain 2022), CEUR Workshop Proceedings, 2022, pp. 39–45.
[37] N. Malamas, A. Symeonidis, Embedding Rasa in edge devices: capabilities and limitations, Proc. Comput. Sci. 192 (2021) 109–118.

12