Recent Deep Learning Based NLP Techniques for

Chatbot Development: An Exhaustive Survey

Bhupesh Rawat Ankur Singh Bist Untung Rahardja
Computer Sc. & Engg Computer Sc. & Engg Master of Information Technology
Graphic Era Hill University Graphic Era Hill University University of Raharja
Uttarakhand, India Uttarakhand, India Tangerang, Indonesia
[email protected] [email protected] [email protected]

Qurotul Aini Yulia Putri Ayu Sanjaya

Master of Information Technology Retail Management
University of Raharja University of Raharja
2022 10th International Conference on Cyber and IT Service Management (CITSM) | 978-1-6654-6074-3/22/$31.00 ©2022 IEEE | DOI: 10.1109/CITSM56380.2022.9935858

Tangerang, Indonesia Tangerang, Indonesia

[email protected] [email protected]

Abstract—Chatbots are becoming popular as many suggestions are provided on how to develop a good chatbot
applications need it. E-commerce, aviation, supply chain which includes qualities such as having human touch,
management companies etc. are using AI based chatbots to making things easier for user, easy to use, having high level
attract customers. Deep learning is another vertical impacting of precision, motivates user to talk, learn from the
chatbots' behavior by bringing it close to human physiology. conversation, among others [6].
Ways of interaction directly enhance the business productivity
that is the main reason for the rise of chatbots. Chatbots are II. OVERVIEW OF CHATBOT
evolving with the use of emerging techniques. In this paper we
are making a detailed study of recent deep learning techniques
for chatbots. It will bring understanding of developers closer
towards good chatbot design. We will cover current gaps and
solutions to catalyze current research.

Keywords—Chatbots, Deep learning, Customer engagement,

AI.

I. INTRODUCTION
With the advancement of emerging technologies, it is be-
coming increasingly simple for people to interact with ma-
chines in an effective manner. There are many applications
called assistant through which a user can instruct a computer
to do a specific task. This is especially beneficial for those Fig. 1. RASA framework technical architecture
users who find it difficult to give commands in writing such
as in google. Although there exist several apps, chatbot is There are various open source platforms with extended
one of the software which is used widely by users. The paid services. One of the frameworks is RASA i.e. used by
chatbot can be defined as software which help humans to various developers to build the chatbot application Fig 1 [2].
make coherent conversation with machines using natural There are various advantages of this framework. Easy to plug
language like English, etc [1]. The conversation can be in, flexible and customizable, up to date configuration, easy
engaging at times depending upon the size of vocabularies. to add in favorite platform, flawless intent support, strong
In recent times the use of deep learning techniques has learning, deployment support, analytics support etc [7].
increased significantly by researchers and scientists to find makes it useful in all current scenarios [8]. Just like RASA
new ways to solve current problems [2]. Deep learning there is another framework mentioned in Fig 2. Thirty-three
techniques have been used in a wide variety of applications Pre-built agents, Reusability and allotment texture, Simple
including chatbot. The chatbot is being used in several fields import and dig out processes, Multiple language support,
such as healthcare, education and E-service and assistant. Integration with discussion channel, Software development
Among other challenges in the development of a good kit for different programming languages, speech to text
chatbot the main challenge is to build a conversation chatbot. support, simple interface, easy to train makes it better for
Though the field of natural language processing and deep various development procedures [9].
learning has matured to some extent, a good conversation
agent is still a challenge for the scientist and researchers [3]. A. Issues with chatbot framework
In this paper, we discuss recently developed deep • It is extremely difficult to manage if the identical
learning based natural language processing techniques for the intent are there in the different perspective
development of the chatbot [4]. This exhaustive survey also • Hard to control the stream of the discussion and the
focuses on how the advancement in AI, in particular deep bot tends to misinterpret the user needs
learning, has deeply impacted the behavior of a chatbot and
brought it closer to human physiology [5]. In the end

Authorized licensed use limited to: Somaiya University. Downloaded on September 02,2024 at 15:47:46 UTC from IEEE Xplore. Restrictions apply.
The 10th International Conference on Cyber and IT Service Management (CITSM 2022)
Yogyakarta, September 20-21, 2022

• Restrictions of interpreting the synonyms and are useful for involving a person in informal open-domain
hyponyms conversations [22].
C. Recurrent Neural Networks
The most striking feature of recurrent neural networks is
its ability to consider previous conversations in the current
context to comprehend the user’s query in an effective
manner [23]. In other words, we can say the information is
passed from one layer of the network to another layer [24].
D. Sequence to Sequence Model
The perfect exam of sequence-to-sequence model is
gener- ative based model [25]. It works by creating a target
sequence model using the source sequence model [26]. In
Fig. 2. Emerging deep learning NLP techniques for chatbot design this context, the source sequence model is the user’s input,
and the target sequence model is the response generated by a
Deep learning is a branch of artificial intelligence which
chatbot. Multiple recurrent neural networks can be employed
has been used in several areas. In recent times deep learning
which is the most basic version of the model [27].
techniques have begun to use in the development of chatbot
design [10]. In this section we discuss several prominent E. Deep Seq2seq Model
emerging deep learning techniques to build smart chatbot This model is generally used to create chatbots which
[11]. perform like humans and are closer to human conversation
III. NATURAL LANGUAGE PROCESSING (NLP) [28]. The unique feature of such a model is that they utilized
more parameters for example having multiple layers of
Natural language processing is a field of artificial LSTM networks [29].
intelligence widely used for processing human language to
machine understandable language [12]. It has also begun to IV. DEEP LEARNING IMPACT ON CHATBOT DESIGN
be used for the development of chatbot design in recent AI techniques involve Machine learning and Deep
years. NLP is used to examine and interpret voice, learning, impact of Deep learning techniques in the domain
handwritten text and speech by computer. Relevant of NLP is very huge and evolving with time [30]. There are
information is collected regarding comprehension and usage so networks that, as mentioned the previous section, created
of human language to build programs that can carry out a a wide difference in conversation patterns by learning the
variety of tasks [13]. In this context, it is necessary to behavior of human conversation [31]. Role of data is also
mention that the majority of NLP techniques rely on machine significant and various open source data sets are also
learning to perform their specific tasks [14]. available if anyone wants to develop a prototype Fig 3 [32].
A. Natural Language Understanding Some algorithms like BERT are gaining popularity in the
domain and impacting NLP activities [33].
In majority of chatbots natural language understanding is
used to comprehend the context from the given user question
in a language that can be easily understood by human beings
and respond based on current user need [15]. The three major
problems raised during the NLU process are the mechanisms
of thought, the interpretation, and the general knowledge of
the user. Intent classification and entity extraction are two
major components of natural language processing. The entity
can be categorized as user defined or system defined [16].
On the other hand, context is the strings which are stored by
objects and refers to user input. For the building of intent
classification machine learning algorithms such as support
vector machines can be effectively used [17].
B. Artificial Neural Network Artificial
Neural networks play a key role in the design of various
types of chatbot [18]. For example, retrieval and generative
chatbots use ANN. It is worth mentioning the working of
such a system briefly. The system takes the user input,
transforms it into vector representation and provides it to the Fig. 3. Flow of AI impacting Chatbots Design
neural network and generates the response [19]. This process
of converting words into vectors is called embedding and A. BERT
deep learning techniques are generally used in this context BERT Base contains twelve layers i.e. transformer
such as Word to Vec [20]. Retrieval based technique works blocks, twelve attention heads and one hundred ten million
by considering input vectors and intent and outputs the parameters [34]. BERT Large contains twenty four layers
probability of every intent [21]. Generative-based chatbots and sixteen attention heads with three hundred forty million

Authorized licensed use limited to: Somaiya University. Downloaded on September 02,2024 at 15:47:46 UTC from IEEE Xplore. Restrictions apply.
 The 10th International Conference on Cyber and IT Service Management (CITSM 2022)
Yogyakarta, September 20-21, 2022

parameters [35]. Similar networks are emerging to make V. CONCLUSIONS

chatbot activities robust Fig 4 [36]. In this paper the impact of deep learning techniques for
chatbots are discussed. After analyzing the correlation of
deep learning and chatbots, it has been concluded that deep
learning is drafting major improvements in conversational
bots [49]. Still there are many challenges related to context
and semantics in real time conversations. Research is going
on to fill the gap in linguistic and AI domain. Global quest of
finding human-like chatbots may be fulfilled in future with
all these advancements [15]. Deep learning and Virtual
reality will play an important role to take chatbots to the next
level where human-like interaction is possible [50].
ACKNOWLEDGMENT
The authors would like to thank University of Raharja,
Graphic Era Hill University for their support in providing a
Fig. 4. BERT Algorithm Variants place for this research through analysis of research systems.
This research received no external funding.
B. Views on current state of Chatbot
The numerous advantages of chatbot to user such as 24*7 REFERENCES
online support on a wide range of services, availability
[1] G. Sperl´ı, “A cultural heritage framework using a deep learning
among others make it a perfect online assistant [37]. based chatbot for supporting tourist journey,” Expert Systems with
According to statistics reported in various reliable internet Applica- tions, vol. 183, p. 115277, 2021.
sources it is found that today 27% of consumers are [2] M. Dhyani and R. Kumar, “An intelligent chatbot using deep learning
interested in artificial intelligence support tools such as with bidirectional rnn and attention model,” Materials today: proceed-
conversation agent, voice assistant, chatbot [38]. in yet ings, vol. 34, pp. 817–824, 2021.
another report it is found that 1.4 billion people are using [3] S. Mathur and D. Lopez, “A scaled-down neural conversational
chatbot in some way or other to solve their problem [39]. model for chatbots,” Concurrency and Computation: Practice and
Experience, vol. 31, no. 10, p. e4761, 2019. C. Zuo, J. Qian, S. Feng,
According to a survey a chatbot can answer 80% of the W. Yin, Y. Li, P. Fan, J. Han, K. Qian, and
standard questions. In 2017, 34% of customers preferred to [4] Q. Chen, “Deep learning in optical metrology: a review,” Light:
communicate with chatbots in online retail. 67% of users Science & Applications, vol. 11, no. 1, pp. 1–54, 2022.
have used chatbots in the past year [40]. In yet another [5] L. V. Jospin, H. Laga, F. Boussaid, W. Buntine, and M. Bennamoun,
survey people said that the most striking feature of chatbot is “Hands-on bayesian neural networks—a tutorial for deep learning
24-hour service. 37% of people takes the help of chatbot in users,” IEEE Computational Intelligence Magazine, vol. 17, no. 2, pp.
an emergency to get a quick answer [41]. 29–48, 2022.
[6] V. Shankar and S. Parsana, “An overview and empirical comparison
According to experts it would be hugely beneficial for a of natural language processing (nlp) models and an introduction to
company to invest in AI based products such as chatbot to and empirical application of autoencoder models in marketing,”
enhance user experience [42]. It is estimated that around Journal of the Academy of Marketing Science, pp. 1–27, 2022.
50% of organizations are planning to invest in chatbot [7] S. Saha, D. Roy, B. Y. Goud, C. S. Reddy, and T. Basu, “Nlp-
development for customer support service and the remaining iiserb@ simpletext2022: To explore the performance of bm25 and
transformer based frameworks for automatic simplification of
40% are going to invest in virtual assistants. Chatbots will be scientific texts,” Proceedings of the Working Notes of CLEF, 2022.
a driving force for business communications with a [8] R. Sonbol, G. Rebdawi, and N. Ghneim, “The use of nlp-based text
worldwide market size of more than $1.3 billion by 2024 representation techniques to support requirement engineering tasks: A
[43]. systematic mapping review,” arXiv preprint arXiv:2206.00421, 2022.
[9] C. Toraman, O. Ozcelik, F. S¸ ahinuc¸, and U. Sahin, “Arc-nlp at
The following statistics throw more light on the checkthat! 2022: Contradiction for harmful tweet detection,” Working
significance of chatbots in the future. 69% of users Notes of CLEF, 2022.
recommend chatbot due to their ability to provide quick [10] B. Rawat, A. S. Bist, U. Rahardja, C. Lukita, and D. Apriliasari, “The
responses as compared to human beings [44]. 95% of users impact of online system on health during covid 19: A comprehensive
will be benefit much from the chatbots. One of the main study,” ADI Journal on Recent Innovation, vol. 3, no. 2, pp. 195–201,
features of chatbot is that they create a link between a 2022.
company and its customer which is also referred to as a [11] B. Luo, R. Y. Lau, C. Li, and Y.-W. Si, “A critical review of state-of-
the- art chatbot designs and applications,” Wiley Interdisciplinary
technological link [45]. However, in a survey it was revealed Reviews: Data Mining and Knowledge Discovery, vol. 12, no. 1, p.
that chatbots were found to be ineffective, in the sense that e1434, 2022.
they provide bad suggestions, are creepy, unhelpful [12] M. W. Ashfaque, “Analysis of different trends in chatbot designing
responses, unable to understand accents in some instances and development: A review,” ECS Transactions, vol. 107, no. 1, p.
[46]. In the most recent report, it is found that chatbots help 7215, 2022.
an organization to streamline their internal workflow [47]. In [13] A. J. Trappey, A. P. Lin, K. Y. Hsu, C. V. Trappey, and K. L. Tu,
other words they help to answer questions, communicate “Development of an empathy-centric counseling chatbot system
capable of sentimental dialogue analysis,” Processes, vol. 10, no. 5, p.
with employee and help in completing HR related 930, 2022.
transactional functions. Chatbots will tremendously help
[14] A. Alm and L. M. Nkomo, “Chatbot experiences of informal language
businesses to increase their sale and as a result grow their learners: A sentiment analysis,” in Research Anthology on
revenue [48].

Authorized licensed use limited to: Somaiya University. Downloaded on September 02,2024 at 15:47:46 UTC from IEEE Xplore. Restrictions apply.
The 10th International Conference on Cyber and IT Service Management (CITSM 2022)
Yogyakarta, September 20-21, 2022

Implementing Sentiment Analysis Across Multiple Disciplines. IGI [34] A. Y. Pratama, “Estimasi efisiensi teknis perbankan indonesia
Global, 2022, pp. 933–948. berbasis stochastic frontier analysis,” Technomedia Journal, vol. 7,
[15] B. K. Bintaro, P. Sokibi, I. Amsyar, and Y. P. A. Sanjaya, “Utilizing no. 2 Oktober, pp. 13–32, 2022.
digital marketing as a business strategy: Utilizing digital marketing as [35] S. Kosasi, U. Rahardja, N. Lutfiani, E. P. Harahap, and S. N. Sari,
a business strategy,” Startupreneur Bisnis Digital, vol. 1, no. 1 April, “Blockchain technology-emerging research themes opportunities in
pp. 63–71, 2022. higher education,” in 2022 International Conference on Science and
[16] B. Rawat, N. Mehra, A. S. Bist, M. Yusup, and Y. P. A. Sanjaya, Technology (ICOSTECH). IEEE, 2022, pp. 1–8.
“Quantum computing and ai: Impacts & possibilities,” ADI Journal [36] B. Rawat, A. S. Bist, N. Mehra, M. F. Fazri, and Y. A. Terah, “Study
on Recent Innovation, vol. 3, no. 2, pp. 202–207, 2022. of kumaon language for natural language processing in end- to-end
[17] U. Rahardja, “Application of the c4. 5 algorithm for identifying conversation scenario,” IAIC Transactions on Sustainable Digital
regional zone status using a decision tree in the covid-19 series,” Innovation (ITSDI), vol. 3, no. 2, pp. 143–149, 2022.
Aptisi Trans- actions on Technopreneurship (ATT), vol. 4, no. 2, pp. [37] M. R. R. Adam, T. Handra, and M. Annas, “Pengaruh celebrity en-
164–173, 2022. dorser dan periklanan terhadap brand image (peran digital
[18] A. S. Anwar, U. Rahardja, A. G. Prawiyogi, N. P. L. Santoso et al., marketing),” Technomedia Journal, vol. 7, no. 2 Oktober, pp. 53–65,
“ilearning model approach in creating blockchain based higher 2022.
education trust,” International Journal of Artificial Intelligence [38] L. Honesti, Q. Aini, M. I. Setiawan, N. P. L. Santoso, and W. Y.
Research, vol. 6, no. 1, 2022. Prihastiwi, “Smart contract-based gamification scheme for college in
[19] N. K. A. Dwijendra, I. Vaslavskaya, N. V. Skvortsova, T. P. Rakhlis, higher education,” APTISI Transactions on Management (ATM), vol.
U. Rahardja, M. H. Ali, A. H. Iswanto, L. Thangavelu, and M. M. 6, no. 2, pp. 102–111, 2022.
Kadhim, “Application of experimental design in optimizing fuel sta- [39] Q. Aini, W. Febriani, C. Lukita, S. Kosasi, and U. Rahardja, “New
tion queuing system,” Industrial Engineering & Management normal regulation with face recognition technology using attendx for
Systems, vol. 21, no. 2, pp. 381–389, 2022. student attendance algorithm,” in 2022 International Conference on
[20] W. Sejati, D. P. AH, F. Khansa, A. S. Maulana, and D. Julianingsih, Science and Technology (ICOSTECH). IEEE, 2022, pp. 1–7.
“Flood disaster mitigation using the hec-ras application to determine [40] U. Rahardja, Q. Aini, A. Khairunisa, and S. Millah, “Implementation
river water levels in the old city area of jakarta,” Aptisi Transactions of blockchain technology in learning management system (lms),”
on Technopreneurship (ATT), vol. 4, no. 2, pp. 121–134, 2022. APTISI Transactions on Management (ATM), vol. 6, no. 2, pp. 112–
[21] F. A. Rahardja, S.-C. Chen, and U. Rahardja, “Review of behavioral 120, 2022.
psy- chology in transition to solar photovoltaics for low-income [41] I. Restiaty, Z. Maharani, R. Rojali, W. Darmawan, and B. Y. D.
individuals,” Sustainability, vol. 14, no. 3, p. 1537, 2022. Yanti, “Relationship of water temperature and air humidity with aedes
[22] Y. Shino, H. Kenta, and I. K. Mertayasa, “Media promotional for art sp. manggarai tebet village south jakarta in 2022,” ADI Journal on
in tangerang city with audio visual adobe creative,” Aptisi Recent Innovation, vol. 4, no. 1, pp. 102–109, 2022.
Transactions on Technopreneurship (ATT), vol. 4, no. 2, pp. 192– [42] N. L. W. S. R. Ginantra, I. M. D. P. Asana, W. G. S. Parwita, and I.
204, 2022. W. E. Eriana, “Mobile-based customers management system in
[23] D. P. Lazirkha et al., “The impact of artificial intelligence in smart ayunadi supermarket,” ADI Journal on Recent Innovation, vol. 4, no.
city air purifier systems,” Aptisi Transactions on Technopreneurship 1, pp. 86–101, 2022.
(ATT), vol. 4, no. 2, pp. 205–214, 2022 [43] M. Azmi, M. S. Shihab, D. Rustiana, D. P. Lazirkha et al., “The effect
[24] A. Adiyanto and R. Febrianto, “Authentication of transaction process of advertising, sales promotion, and brand image on repurchasing
in e-marketplace based on blockchain?? technology,” Aptisi intention (study on shopee users),” IAIC Transactions on Sustainable
Transactions On Technopreneurship (ATT), vol. 2, no. 1, pp. 68–74, Digital Innovation (ITSDI), vol. 3, no. 2, pp. 76–85, 2022.
2020. [44] J. Artanti, P. M. Agustini, A. Saptono, G. K. Hanum, and D. Regina,
[25] J. Heikal, V. Rialialie, D. Rivelino, and I. A. Supriyono, “Hybrid “Analysis of virtual product marketing strategies to increase customer
model of structural equation modeling pls and rfm (recency, satisfaction (case study on bukalapak partners),” IAIC Transactions
frequency and monetary) model to improve bank average balance,” on Sustainable Digital Innovation (ITSDI), vol. 3, no. 2, pp. 86–109,
Aptisi Transactions on Technopreneurship (ATT), vol. 4, no. 1, pp. 1– 2022.
8, 2022. [45] U. Rahardja, “Meningkatkan kualitas sumber daya manusia dengan
[26] R. Widayanti, Q. Aini, H. Haryani, N. Lutfiani, and D. Apriliasari, sis- tem pengembangan fundamental agile,” ADI Bisnis Digital
“Decentralized electronic vote based on blockchain p2p,” in 2021 9th Interdisiplin Jurnal, vol. 3, no. 1, pp. 63–68, 2022.
International Conference on Cyber and IT Service Management [46] R. Widayanti, I. N. Hapsari, G. Firmansyah, M. A. Nurbayin et al.,
(CITSM). IEEE, 2021, pp. 1–7. “Kolaborasi dalam metode problem base learning dengan aplikasi
[27] M. I. Sanni, D. Apriliasari et al., “Blockchain technology application: trello studi kasus matakuliah e-busniness program kampus merdeka,”
Authentication system in digital education,” Aptisi Transactions on ADI Pengabdian Kepada Masyarakat, vol. 3, no. 1, pp. 8–15, 2022.
Technopreneurship (ATT), vol. 3, no. 2, pp. 151–163, 2021. [47] S. Kosasi, I. D. A. E. Yuliani, U. Rahardja et al., “Boosting e-service
[28] D. Mohammed, N. Aisha, A. Himki, A. Dithi, and A. Y. Ardianto, quality of online product businesses through it leadership,” in 2022
“Blockchain is top skill for 2020,” Aptisi Transactions on Techno- International Conference on Science and Technology (ICOSTECH).
preneurship (ATT), vol. 2, no. 2, pp. 180–185, 2020. IEEE, 2022, pp. 1–10.
[29] J. Hom, B. Anong, K. B. Rii, L. K. Choi, and K. Zelina, “The octave [48] S. Rahayu, N. I. Alzaytun et al., “Standar pelayanan publik keca-
allegro method in risk management assessment of educational matan pasar kemis kabupaten tangerang,” ADI Pengabdian Kepada
institutions,” Aptisi Transactions on Technopreneurship (ATT), vol. Masyarakat, vol. 3, no. 1, pp. 1–7, 2022.
2, no. 2, pp. 167–179, 2020. [49] Y. P. A. Sanjaya and M. A. Akhyar, “Blockchain and smart contract
[30] S. A. Yakan, “Analysis of development of artificial intelligence in the applications can be a support for msme supply chain finance based on
game industry,” International Journal of Cyber and IT Service sharia crowdfunding,” Blockchain Frontier Technology, vol. 2, no. 1,
Management, vol. 2, no. 2, pp. 111–116, 2022. pp. 44–49, 2022.
[31] B. Rawat, A. S. Bist, D. Supriyanti, V. Elmanda, and S. N. Sari, “Ai [50] T. Ramadhan and W. N. Wahid, “New authoritative changes with
and nanotechnology for healthcare: A survey,” APTISI Transactions blockchain an emphasis production network,” Blockchain Frontier
on Management (ATM), vol. 7, no. 1, pp. 86–91, 2023. Tech- nology, vol. 2, no. 1, pp. 24–35, 2022.
[32] S. Maesaroh, H. Gunawan, A. Lestari, M. S. A. Tsaurie, and M. Fauji, A. Himki, T. Ramadhan, Y. Durachman, and E. S. Pramono, “Digital
“Query optimization in mysql database using index,” International business entrepreneurship decisions: An e-business analysis (a study
Journal of Cyber and IT Service Management, vol. 2, no. 2, pp. 104– literature review),” Startupreneur Bisnis Digital, vol. 1, no. 1 April,
110, 2022. pp. 107–113, 2022
[33] R. Mulyana, N. A. Achsani, T. Andati, T. N. A. Maulana, and

Authorized licensed use limited to: Somaiya University. Downloaded on September 02,2024 at 15:47:46 UTC from IEEE Xplore. Restrictions apply.