A Review on Basic Deep Learning

Technologies and Applications

Tejashri Patil, Sweta Pandey, and Kajal Visrani

Abstract Deep learning is a rapidly developing area in data science research. Deep
learning is basically a mix of machine learning and artificial intelligence. It proved
to be more versatile, inspired by brain neurons, and creates more accurate models
compared to machine learning. Yet, due to many aspects, making theoretical designs
and conducting necessary experiments are quite difficult. Deep learning methods play
an important role in automated systems of perception, falling within the framework of
artificial intelligence. Deep learning techniques are used in IOT applications such as
smart cities, image recognition, object detection, text recognition, bioinformatics, and
pattern recognition. Neural networks are used for decision making in both machine
learning and deep learning, but the deep learning framework here is quite different,
using several nonlinear layers that generate complexity to obtain more precision,
whereas a machine learning system is implemented linearly. In the present paper,
those technologies were explored in order to provide researchers with a clear vision
in the field of deep learning for future research.

Keywords Deep learning · Neural network · Activation function · Accuracy ·

Loss function · Weight · Machine learning

1 Introduction

A new field has arisen over the past couple of years and has demonstrated its promise
in many existing technologies. Often known as deep neural network, deep learning
consists of many layers with a number of neurons in each layer. Such layers may
range from a few to thousands, and each layer may contain thousands of neurons
(processing unit) in addition. Multiplying the input values with the allocated weight

T. Patil (B) · S. Pandey · K. Visrani

SSBT’s College of Engineering and Technology, Jalgaon, Maharashtra, India
e-mail: [email protected]
S. Pandey
e-mail: [email protected]

© The Editor(s) (if applicable) and The Author(s), under exclusive license 565
to Springer Nature Singapore Pte Ltd. 2021
K. Kotecha et al. (eds.), Data Science and Intelligent Applications,
Lecture Notes on Data Engineering and Communications Technologies 52,
https://doi.org/10.1007/978-981-15-4474-3_61
566 T. Patil et al.

Fig. 1 Structure of deep learning model

to each input and summing up the result are the simplest process in a neuron. This
result will be further scrutinized by the activation function. It improves the precision
of the deep learning model. Figure 1 illustrates the structure of the deep learning
model.
The deep learning model generates results by multiplying the weights of the data,
thereby summarizing all the values

Y = (weight ∗ input) + bias (1)

where Y is the performance of the model and bias is a constant chosen to optimize
the model according to the requirement.
Deep learning has applications in numerous areas such as image detection,
speech recognition, computer vision, natural language processing, bioinformatics,
advertising, e-commerce, digital marketing, robot learning, and many more [1, 2].

2 Literature Survey

Du et al. [3] showed some advanced neural networks of deep learning and their
implementation. It also addresses the drawbacks and opportunities of deep learning.
Zhou et al. [4] introduced the importance of in-depth learning technology, imple-
mentations, and the impact on in-depth learning of the dataset by using new datasets
more quickly. This deals with computer vision area, mainly applying deep learning
to object detection tasks. Discussed description of the widely used datasets in com-
puter applications and deep learning algorithms. On the other hand, a new dataset is
designed according to the widely used datasets and select one of the networks called
faster RCNN to operate on this new dataset. Zhou et al. [5] performed a far-reaching
research study using in-depth computer well-being learning, including a basic inves-
tigation into the relative validity and possible drawbacks of the methodology and, in
addition, its future perspective. The paper focuses primarily on major deep learning
A Review on Basic Deep Learning Technologies and Applications 567

applications in the fields of translational bioinformatics, medicinal imaging, eventual

prediction, restorative computer science, and general well-being. The paper focuses
primarily on major deep learning applications in the fields of translational bioinfor-
matics, medicinal imaging, eventual prediction, restorative computer science, and
general well-being. Comprehensive empirical evidence of these residual networks
being easier to refine and being able to gain accuracy from significantly increased
size. Ioffe et al. [6] introduced a new method to dramatically speed up deep device
development. It depends on the implementation that covariate movement, known to
complicate the preparation of machine learning frameworks, often applies to sub-
systems and layers, and expelling it from the system’s internal activation will aid
in the preparation. The suggested approach derives its potential from standardiz-
ing initiations and integrating this standardization into the device engineering itself.
This means that due care is taken for any programming technique that is used to
improve the software. Karpathy et al. [2] showed that CNN models are capable of
capturing incredible highlights from weakly labeled knowledge far outperforming
well-based execution approaches and surprisingly robust to the subtleties of timely
systemic availability. There are interpretable flaws in subjective analysis of system
yields and disarray lattices. Lin et al. [7] provided another dataset with the intention
of propelling object identification representation by putting the object identification
question in relation to the more comprehensive scene understanding inquiry. This
is done on social occasions by pictures of complex everyday scenes of common
objects in their specific environment. Objects are defined using segmentation, for
example, to assist with the exact location of objects. Eventually, categorical specifics
are defined by the Gage. Zhou et al. [8] provided millions of scene images for another
scene-driven database and proposed new approaches to determine the density and
varied variety of image datasets to show that locations are as dense as other scene
datasets and are more varied.

2.1 Number of Inputs to be Considered and Finding

Noncontributing Columns

Because a dataset can contain a number of attributes, it is basically a good idea

to remove the unnecessary attributes when constructing a deep learning model. In
addition, removing one class column from the dataset is also necessary. This can
be done with the number array in dataset, but choosing the useful attributes is a
challenging job.
568 T. Patil et al.

2.2 Number of Hidden Layers

In a deep learning model, two hidden layers are absolutely essential. A single output
layer is used to combine the output of one or more hidden layers. It provides a deeper
theoretical model using more hidden layers on the one side, but each additional layer
adds computational complexity on the other. In addition, higher numbers of neurons
inserted in each layer will also increase the computational cost.

2.3 Gradient Descent Optimizers

The gradient downward tends to minimize the model’s expense. The chosen gradient
descent shifts weights to reduce the process’s cost. Errors are observed based on
observations of the input and weight combination. It is also advisable to take reason-
able long-term steps when going down as a big step could lead to a situation where
global minima could be missed. In deep learning, the main task of a model is to
assign weights to different inputs in order to optimize the model based on the inputs
given. Weight loss should not be excessive; otherwise, local maxima, depression,
and other related issues can arise. In order to optimize the model, various optimizers
are programmed to assign weights to the inputs. Choosing an optimizer is a daunting
and brainstorming task, however, as upgrading the algorithm with different weights
increases the cost of the model and takes more time when training it for large datasets
[3, 6].

2.4 Weights

Random weights are randomly choosing eight can be a good choice in which the
input values add different weights to achieve good results. In this way, the proper
coordination of weight and input can be established. Nonetheless, initializing random
weights with low values say 0.1 is a good idea. If the weights of the descent are at
zero, in this case, the weights of the corresponding input will never change and
the same weight will be repeated, so it is not desirable to choose the weight of
the descent as zero. And it has chosen to choose weights with random values. To
speed up the model’s learning process and overall performance, pick weights very
carefully. However, when performing deep learning modeling such as zero, one,
random, constant, matrix of defined weights, orthogonal matrix weights, and different
possibilities could be available in keras method.
A Review on Basic Deep Learning Technologies and Applications 569

2.5 Loss Function

Loss function usually says the discrepancy that is nothing but the error between the
actual output and the planned output. The formula calculated is called the function
of loss. F (loss) = Expected Output − Actual Output. The real and predicted perfor-
mance difference can now be calculated in many respects. There are different loss
functions to do this. Choosing an acceptable role to lose deep learning is a challenging
task.
In deep learning, loss functions are basically convex function to find the n-
dimensional interval downward on convex surface to minimize cost while simul-
taneously finding global minima in learning states. This is because the model will
work for the classification of test data with minimal costs associated with it.
Activation Function: There are many activation mechanisms, but they do not
produce similar results due to different statistical architecture. Usually, it has been
found that the sigmoid function can be used in the output activation function, and
the question of binary classification provides the best results. Softmax may be a
preferred choice where there is an issue with multi-label classification, but it needs
to be avoided for binary classification. There are many activation mechanisms, but
they do not produce similar results due to different statistical architecture. Usually,
it has been found that the sigmoid function can be used in the output activation
function, and the question of binary classification provides the best results.
Type of Network: Dense Network: The network is used to a great extent. In this,
each neuron layer is connected to the next neuron layer. Even though it seems to be
complicated, it is successful.
LSTM Network: Wide short-term memory network is a technique used over a long
period of time to prevent the memory-related issues. In a general neural network,
except from the output layer, each layer has the same structure and activation mecha-
nism. Moreover, if different layers have different structures, the LSTM network may
be an option as shown in Fig. 1.

3 Methodology

Various numbers of techniques and algorithms are used in deep learning. Some of
deep learning techniques are as follows [9].
(i) Recurrent neural networks (RNNs)
(ii) Long short-term memory (LSTM)
(iii) Convolution neural networks (CNNs)
(iv) Deep belief networks (DBNs)
(v) Deep stacking networks (DSNs)
Recurrent Neural Networks: The recurrent neural network is basic network struc-
ture; it helps to develop other deep learning structures. Basic multiple layer neural
570 T. Patil et al.

Fig. 2 Recurrent neural

networks (RNNs) [15]

network has completely feed-forward links, whereas in case of a recurrent network,

it has feedback links connected to the previous layers.
This mechanism enables recurrent neural networks to remember the previous
inputs values so that easy process within specific duration.
Figure 2 displays input steps in the current moment showing the example of input,
and feedback layers reflect the previous moment’s output. Closed-loop feedback
related to their past decisions ingests their own outputs as input moment by moment.
Inserting memory into neural networks to provide information in the sequence itself
and in feed-forward style networks, this is not the case. Back-propagation or back-
propagation by time algorithms is used to train recurrent neural networks. Mostly
used in speech recognition, recurrent neural networks.
Long Short-term Memory (LSTM): Long short-term memory was earlier struc-
tured yet it prominent as of late as a RNN engineering for different applications. In
different items will discover LSTMs that utilization consistently, android devices,
voice recognition, text captioning etc. rather presented the idea of a memory section.
It can hold its value as an element of its information sources for a long or short
period of time, allowing the location to recollect what is critical and not just its last
registered esteem. In reality, long-term short-term memory network is RNN made
of LSTM units. An output input way and a cell for the short term. The cell collects
interim values over arbitrary duration and passes the data stream to and from the
location of the memory. When a current data is skipped, the short-term cell IS used
to handle, allowing the cell to recall new information. In conclusion, the output way
handles as data stored in short memory location is utilize in output.
LSTM systems are applicable for preparing, categorizing, and forecasting
depending on event sequence information (Fig. 3).
A Review on Basic Deep Learning Technologies and Applications 571

Fig. 3 LSTM networks memory cell arrangement [15]

(i) Convolution neural network (CNN)

Convolutionary neural system is mainly used for image identification, image cate-
gorization, and biometric recognition. CNN picture identification accepts a picture
as an input and prepares it for groupings of particles (i.e., vehicles, pets, toys) [2, 9].
In Fig. 4, convolution neural network accepts information in the form of an image;
in the subsequent stage, convolution layer is used to extract features from an input
[9]. Convolution of a picture into extricate features stage performs various activities
such as border detection and eliminating noise from a picture by applying filters [10].
Deep Belief Network (DBN): DBN is a class of deep learning system with both
directed and undirected edges that involves different layers. It consists of various
layers of shrouded units, where each layer is in any case connected to each other, and
units are not connected [21]. To learn deep belief networks need to have comprehend
two imperative techniques of DBN [9, 21].
Belief Network: It includes stochastic binary unit layers with a certain weight for
each associated layer. In belief networks, stochastic binary units have a low (0) or
high (1) condition, and the probability and likelihood of becoming 1 are regulated
by a tendency and weighted input from other units [21].

Fig. 4 Basic steps of convolution neural network

572 T. Patil et al.

Table 1 Techniques and its some application area [4, 15, 16]
Techniques Area of application
Recurrent neural networks Voice recognition, text identification
Long short-term memory In NLP data compression, signature and text identification,
voice recognition, posture identification, text captioning for
pictures
Convolutional neural networks Image categorization, visual identification, NLP, behavior
recognition
Deep belief networks Image categorization, search engine, language interpretation,
failure forecasting
Deep stacking networks Search engine, conversation voice identification

Restricted Boltzmann Machine: RBM was implemented as a veiled layer unit

that has a minimal association with each hide unit so easy to know RBM. Deep
belief systems consist of different layers of RBMs to the already trained layer [21]
following the fine-tune feed-forward scheme. Deep belief network learn features
from the exposed device in the first level, learn features in a second hide layer in the
next step. The entire deep belief network is educated after learning the final layer
[11].
Deep Stacking Network (DSN)
The deep stacking network (DSN) is a deep architecture that can be adapted to
parallel weight learning [12]. It is prepared in a supervised, block-wise manner,
with no back-propagation requirement across all blocks, as it is common in other
prominent deep models [13]. The DSN blocks are stacked to form the deep system
[14], each consisting of a basic, easy-to-learn module (Table 1).

4 Conclusion

Different techniques and structures available in deep learning help to use in variety
of different application areas. Several layers of neural networks in these techniques
allow categorization of data by extracting distinct features. Huge amount of dataset
is trained in deep learning so that accuracy gets improved.

References

1. Kaiming H, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In:
Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
2. Karpathy A, Toderici G, Shetty S, Leung T, Sukthankar R, Fei-Fei L (2014) Large-scale video
classification with convolutional neural networks. In: Proceedings of the IEEE conference on
computer vision and pattern recognition, pp 1725–1732
A Review on Basic Deep Learning Technologies and Applications 573

3. Du X, Cai Y, Wang S, Zhang L (2016) Overview of deep learning. In: 31st Youth academic
annual conference of chinese association of automation (YAC). IEEE, pp 159–164
4. Zhou X, Gong W, Fu W, Du F (2017) Application of deep learning in object detection. In:
IEEE/ACIS 16th international conference on computer and information science. IEEE, pp
631–634
5. Ravì D, Wong C, Deligianni F, Berthelot B, Andreu-Perez J, Lo B, Yang G-Z (2016) Deep
learning for health informatics. IEEE J Biomed Health Inform 21(1):4–21
6. Ioffe S, Szegedy C (2015) Batch normalization: accelerating deep network training by reducing
internal covariate shift
7. Lin T-Y, Maire M, Belongie S, Bourdev L, Girshick R, Hays J, Perona P, Ramanan D, Lawrence
Zitnick C, Dollar P (2015) Microsoft COCO: common objects in context
8. Zhou B, Lapedriza A, Xiao J, Torralba A, Oliva A (2014) Learning deep features for scene
recognition using places database. In: Advances in neural information processing systems, pp
487–495
9. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 14539, 52128:436–444
10. Hariharan B, Arbeláez P, Girshick R, Malik J (2014) Simultaneous detection and segmentation
11. Rubi CR (2015) A review: speech recognition with deep learning methods. Inter J Comput Sci
Mob Comput (IJCSMC) 4(5):1017–1024
12. Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2(1):1–127
13. Hutchinson B, Deng L, Yu D (2012) Tensor deep stacking networks. IEEE Trans Pattern
Analysis Mach Intell (Special issue in learning deep architectures), 1–14
14. Makhlysheva A, Budrionis A, Chomutare T, Nordsletta AT, Bakkevoll PA, Henriksen TD,
Hurley JS (2018) Health analytics. Norwegian Center for E-health Research
15. Understanding of Convolutional Neural Network (CNN)—Deep learning. https://medium.
com/@RaghavPrabhu/understanding-of-convolutional-neural-network-cnn-deep-learning-
99760835f148
16. Deng L, Yu D (2011) Deep convex net: a scalable architecture for speech pattern classification.
In: Twelth annual conference of the international speech communication association
17. Gong Y, Wang L, Guo R, Lazebnik S Multi-scale orderless pooling of deep convolutional
activation features (published in Illinois)
18. Liu Y, Liu S, Zhao X (2017) Intrusion detection algorithm based on convolutional neural
network. In: 4th International conference on engineering technology and application, pp 9–13
19. Tim Jones M (2017) Deep learning architectures. Artificial Intelligence
20. Mahmud M, Kaiser MS, Hussain A, Vassanelli S (2018) Applications of deep learning and
reinforcement learning to biological data. IEEE Trans Neural Netw Learn Syst, 1–17