American Journal of Neural Networks and Applications

2019; 5(1): 7-11

http://www.sciencepublishinggroup.com/j/ajnna
doi: 10.11648/j.ajnna.20190501.12
ISSN: 2469-7400 (Print); ISSN: 2469-7419 (Online)

An Overview of Neural Network

Mohaiminul Islam, Guorong Chen, Shangzhu Jin
School of Intelligent Technology and Engineering, Chongqing University of Science and Technology, Chongqing, China

Email address:

To cite this article:

Mohaiminul Islam, Guorong Chen, Shangzhu Jin. An Overview of Neural Network. American Journal of Neural Networks and Applications.
Vol. 5, No. 1, 2019, pp. 7-11. doi: 10.11648/j.ajnna.20190501.12

Received: May 8, 2019; Accepted: June 17, 2019; Published: June 29, 2019

Abstract: Neural networks represent a brain metaphor for information processing. These models are biologically inspired
rather than an exact replica of how the brain actually functions. Neural networks have been shown to be very promising systems
in many forecasting applications and business classification applications due to their ability to learn from the data. This article
aims to provide a brief overview of artificial neural network. The artificial neural network learns by updating the network
architecture and connection weights so that the network can efficiently perform a task. It can learn either from available training
patterns or automatically learn from examples or input-output relations. Neural network-based models continue to achieve
impressive results on longstanding machine learning problems, but establishing their capacity to reason about abstract concepts
has proven difficult. Building on previous efforts to solve this important feature of general-purpose learning systems, our latest
paper sets out an approach for measuring abstract reasoning in learning machines, and reveals some important insights about the
nature of generalization itself. Artificial neural networks can learn by example like the way humans do. An artificial neural net
is configured for a specific application like pattern recognition through a learning process. Learning in biological systems
consists of adjustments to the synaptic connections that exist between neurons. This is true of artificial neural networks as well.
Artificial neural networks can be applied to an increasing number of real-world problems of considerable complexity. They are
used for solving problems that are too complex for conventional technologies or those types of problems that do not have an
algorithmic solution.

Keywords: Artificial Intelligence, Neural Network, Sigmoid Function, Neurons, Nodes

recognize are numerical, contained in vectors, into which all

1. Introduction real-world data, be it images, sound, text or time series, must
Neural network is a type of artificial intelligence that be translated.
attempts to imitate the way a human brain works. Rather than In its simplest form, a biological brain is a huge collection
using a digital model, in which all computations manipulate of neurons. Each neuron takes electrical and chemical signals
zeros and ones, a neural network works by creating as inputs through its many dendrites and transmits the output
connections between processing elements, the computer signals through its axon (in a more specialized context, there
equivalent of neurons. The organization and weights of the are exceptions to this behavior like with multipolar neurons).
connections determine the output [1]. Axons make contact with other neurons at specialized
Neural networks are particularly effective for predicting junctions called synapses where they pass on their output
events when the networks have a large database of prior signals to other neurons to repeat the same process over and
examples to draw on. Strictly speaking, a neural network over millions and millions of times [2].
implies a non-digital computer, but neural networks can be Taking inspiration from the brain, an artificial neural
simulated on digital computers. network is a collection of connected units, also called neurons.
Neural networks are a set of algorithms, modeled loosely The connection between the neurons can carry signals
after the human brain, that are designed to recognize patterns. between them. Each connection carries a real number value
They interpret sensory data through a kind of machine which determines the weight/strength of the signal [3].
perception, labeling or clustering raw input. The patterns they
8 Mohaiminul Islam et al.: An Overview of Neural Network

Figure 1. Neuron.

two or more images based on the edges.

2. Types of Artificial Neural Network 2.2. Radial Basis Function Neural Network
Artificial neural networks are computational models which Radial basic functions consider the distance of a point with
work similar to the functioning of a human nervous system. respect to the center. RBF functions have two layers, first
There are several kinds of artificial neural networks. These where the features are combined with the Radial Basis
type of networks are implemented based on the mathematical Function in the inner layer and then the output of these
operations and a set of parameters required to determine the features are taken into consideration while computing the
output [4]. same output in the next time-step which is basically a memory.
2.1. Feed Forward Neural Network Artificial Neuron Below is a diagram which represents the distance
calculating from the center to a point in the plane similar to a
This neural network is one of the simplest form of ANN, radius of the circle. Here, the distance measure used in
where the data or the input travels in one direction. The data euclidean, other distance measures can also be used. The
passes through the input nodes and exit on the output nodes. model depends on the maximum reach or the radius of the
This neural network may or may not have the hidden layers. In circle in classifying the points into different categories. If the
simple words, it has a front propagated wave and no back point is in or around the radius, the likelihood of the new point
propagation by using a classifying activation function usually. begin classified into that class is high. There can be a
Below is a Single layer feed forward network. Here, the transition while changing from one region to another and this
sum of the products of inputs and weights are calculated and can be controlled by the beta function.
fed to the output. The output is considered if it is above a This neural network has been applied in Power Restoration
certain value i.e threshold (usually 0) and the neuron fires with Systems. Power systems have increased in size and
an activated output (usually 1) and if it does not fire, the complexity. Both factors increase the risk of major power
deactivated value is emitted (usually -1) [5]. outages. After a blackout, power needs to be restored as
quickly and reliably as possible. This paper how RBFnn has
been implemented in this domain [6].
2.3. Kohonen Self Organizing Neural Network

The objective of a Kohonen map is to input vectors of

arbitrary dimension to discrete map comprised of neurons.
The map needs to me trained to create its own organization of
the training data. It comprises of either one or two dimensions.
When training the map the location of the neuron remains
Figure 2. ANN. constant but the weights differ depending on the value. This
self organization process has different parts, in the first phase
Application of Feed forward neural networks are found in every neuron value is initialized with a small weight and the
computer vision and speech recognition where classifying the input vector. In the second phase, the neuron closest to the
target classes are complicated. These kinds of Neural point is the ‘winning neuron’ and the neurons connected to the
Networks are responsive to noisy data and easy to maintain. winning neuron will also move towards the point like in the
This paper explains the usage of Feed Forward Neural graphic below. The distance between the point and the neurons
Network. The X-Ray image fusion is a process of overlaying
 American Journal of Neural Networks and Applications 2019; 5(1): 7-11 9

is calculated by the euclidean distance, the neuron with the networks with each other, which in turn will increase the
least distance wins. Through the iterations, all the points are computation speed. However, the processing time will depend
clustered and each neuron represents each kind of cluster. on the number of neurons and their involvement in computing
Kohonen Neural Network is used to recognize patterns in the results.
the data. Its application can be found in medical analysis to
cluster data into different categories. Kohonen map was able
to classify patients having glomerular or tubular with an high 3. Artificial Neural Network (ANN)
accuracy. Here is a detailed explanation of how it is Algorithm Work
categorized mathematically using the euclidean distance
algorithm. Below is an image displaying a comparison A typical neural network has anything from a few dozen to
between a healthy and a diseased glomerular [7]. hundreds, thousands, or even millions of artificial neurons
called units arranged in a series of layers, each of which
2.4. Recurrent Neural Network (RNN) – Long Short Term connects to the layers on either side. Some of them, known as
Memory input units, are designed to receive various forms of
information from the outside world that the network will
The Recurrent Neural Network works on the principle of attempt to learn about, recognize, or otherwise process. Other
saving the output of a layer and feeding this back to the input units sit on the opposite side of the network and signal how it
to help in predicting the outcome of the layer. responds to the information it's learned; those are known as
Here, the first layer is formed similar to the feed forward output units. In between the input units and output units are
neural network with the product of the sum of the weights and one or more layers of hidden units, which, together, form the
the features. The recurrent neural network process starts once majority of the artificial brain. Most neural networks are fully
this is computed, this means that from one time step to the next connected, which means each hidden unit and each output unit
each neuron will remember some information it had in the is connected to every unit in the layers either side. The
previous time-step. This makes each neuron act like a memory connections between one unit and another are represented by a
cell in performing computations. In this process, we need to let number called a weight, which can be either positive (if one
the neural network to work on the front propagation and unit excites another) or negative (if one unit suppresses or
remember what information it needs for later use. Here, if the inhibits another). The higher the weight, the more influence
prediction is wrong we use the learning rate or error correction one unit has on another. (This corresponds to the way actual
to make small changes so that it will gradually work towards brain cells trigger one another across tiny gaps called synapses
making the right prediction during the back propagation. [9].
2.5. Convolutional Neural Network 3.1. Formulation of Neural Network: A Simple Neural
Convolutional neural networks are similar to feed forward Network can Be Represented as Shown in the Figure
neural networks, where the neurons have learn-able weights Below
and biases. Its application has been in signal and image
processing which takes over OpenCV in field of computer
vision.
ConvNet are applied in techniques like signal processing
and image classification techniques. Computer vision
techniques are dominated by convolutional neural networks
because of their accuracy in image classification. The
technique of image analysis and recognition, where the
agriculture and weather features are extracted from the open
source satellites like LSAT to predict the future growth and
yield of a particular land are being implemented [8]. Figure 3. Input - Hidden layer – Output.

2.6. Modular Neural Network The linkages between nodes are the most crucial finding in
Modular Neural Networks have a collection of different an ANN. We will get back to “how to find the weight of each
networks working independently and contributing towards the linkage” after discussing the broad framework. The only
output. Each neural network has a set of inputs which are known values in the above diagram are the inputs. Lets call the
unique compared to other networks constructing and inputs as I1, I2 and I3, Hidden states as H1, H2, H3 and H4,
performing sub-tasks. These networks do not interact or signal Outputs as O1 and O2 [7]. The weights of the linkages can be
each other in accomplishing the tasks. The advantage of a denoted with following notation:
modular neural network is that it breakdowns a large W(I1H1) is the weight of linkage between I1 and H1 nodes.
computational process into smaller components decreasing Following is the framework in which artificial neural
the complexity. This breakdown will help in decreasing the networks (ANN) work:
number of connections and negates the interaction of these
10 Mohaiminul Islam et al.: An Overview of Neural Network

Figure 4. Work process.

3.2. Few Statistical Details about the Framework linkage between hidden nodes and the input nodes in a similar
fashion. Imagine, that this calculation is done multiple times
Every linkage calculation in an Artificial Neural Network for each of the observation in the training set [10].
(ANN) is similar. In general, we assume a sigmoid
relationship between the input variables and the activation rate
of hidden nodes or between the hidden nodes and the 4. Backpropagation
activation rate of output nodes. Let’s prepare the equation to Back propagation is a method used in artificial neural
find activation rate of H1. networks to calculate a gradient that is needed in the
Logit (H1) = W(I1H1) * I1 + W(I2H1) * I2 + W(I3H1) * I3 + calculation of the weights to be used in the network. Back
Constant = f propagation is shorthand for "the backward propagation of
errors," since an error is computed at the output and
= > P(H1) = 1/(1+e^(-f)) distributed backwards throughout the network’s layers. It is
commonly used to train deep neural networks.
Following is how the sigmoid relationship looks like:
Back propagation is a generalization of the delta rule to
muti-layered feed forward networks, made possible by using
the chain rule to alliterative compute gradients for each layer.
It is closely related to the Gauss–Newton algorithm and is part
of continuing research in neural back propagation.
Back propagation is a special case of a more general
technique called automatic differentiation. In the context of
learning, back propagation is commonly used by the gradient
descent optimization algorithm to adjust the weight of neurons
by calculating the gradient of the loss function [11].
4.1. ANN and DNN Concepts Relevant to Backpropagation
Figure 5. Sigmoid function”. Here are several neural network concepts that are important
to know before learning about backpropagation:
3.3. The Weights re-Calibrated Inputs
Source data fed into the neural network, with the goal of
Re-calibration of weights is an easy, but a lengthy process.
making a decision or prediction about the data. The data is
The only nodes where we know the error rate are the output
broken down into binary signals, to allow it to be processed by
nodes. Re-calibration of weights on the linkage between
single neurons—for example an image is input as individual
hidden node and output node is a function of this error rate on
pixels.
output nodes. It can be statistically proved that:
Error @ H1 = W(H1O1)*Error@O1 + W(H1O2)*Error@O2 4.2. Training Set

Using these errors we can re-calibrate the weights of A set of outputs for which the correct outputs are known, it
 American Journal of Neural Networks and Applications 2019; 5(1): 7-11 11

can be used to train the neural networks. mechanisms of the brain.

Outputs Perhaps the most exciting aspect of neural networks is the
The output of the neural network can be a real value possibility that some day 'consious' networks might be
between 0 and 1, a boolean, or a discrete value (for example, a produced. There is a number of scientists arguing that
category ID). conciousness is a 'mechanical' property and that 'consious'
neural networks are a realistic possibility [15].
4.3. Activation Function In future, If anyone want to develop this research, he can
Each neuron accepts part of the input and passes it through choose a topic from here. This paper is an overview of
the activation function. Commonly used functions are the artificial neural network. It’s better to choose a small topic and
sigmoid function, tanh and ReLu. Modern activation functions do research on it.
normalize the output to a given range, to ensure the model has
stable convergence [12].
References
4.4. Weight Space
[1] Neural Networks at Pacific Northwest National Laboratory
Each neuron is given a numeric weight. The weights, applied http://www.emsl.pnl.gov:2080/docs/cie/neural/neural.homepa
to the activation function, determine each neuron’s output. In ge.html
training of a deep learning model, the objective is to discover [2] Klimasauskas, CC. (1989). The 1989 Neuro Computing
the weights that can generate the most accurate output. Bibliography. Hammerstrom, D. (1986). A
Connectionist/Neural Network Bibliography.
4.5. Initialization
[3] N. Murata, S. Yoshizawa, and S. Amari, ―Learning curves,
Setting the weights at the beginning, before the model model selection and complexity of neural networks,ǁ in
trained, a typical strategy in neural networks is to initialize the Advances in Neural Information Processing Systems 5, S.
Jose Hanson, J. D. Cowan, and C. Lee Giles, ed. San Mateo,
weights randomly, and then start optimizing from there. CA: Morgan Kaufmann, 1993, pp. 607-614.
Xavier optimization is another approach which makes sure
weights are “just right” to ensure enough signal passes [4] Bradshaw, J. A., Carden, K. J., Riordan, D., 1991.
through all layers of the network13. Ecological ―Applications Using a Novel Expert System
Shellǁ. Comp. Appl. Biosci. 7, 79–83.
4.6. Forward Pass [5] Lippmann, R. P., 1987. An introduction to computing with
neural nets. IEEE Accost. Speech Signal Process. Mag., April:
The forward pass tries out the model by taking the inputs, 4-22.
passing them through the network and allowing each neuron to
react to a fraction of the input, and eventually generating an output. [6] Murphy, K. P. Machine Learning: A Probabilistic Perspective.
Cambridge, Massachusetts: The MIT Press, 2012.
4.7. Gradient Descent
[7] Hubel, H. D. and Wiesel, T. N. '' Receptive Fields of Single
A mathematical technique that modifies the parameters of a neurones in the Cat’s Striate Cortex.'' Journal of Physiology.
Vol 148, pp. 574-591, 1959.
function to descend from a high value of a function to a low
value, by looking at the derivatives of the function with [8] Jeannette Lawrence, Data Preparation for a Neural Network,
respect to each of its parameters, and seeing which step, via Neural NetworkSpecial Report: A Miller Freeman Publication,
which parameter, is the next best step to minimize the function. 1992.
Applying gradient descent to the error function helps find [9] Gene Bylinsky, Computers That Learn By Doing, Fortune,
weights that achieve lower and lower error values, making the September 6, 1993.
model gradually more accurate [14].
[10] Dennis Collins, Brain Maker: Strange, Captivating, Easy to
Use, California Computer News, July, 1990.
5. Conclusion [11] 1993 Readers' Choice Awards Winner: Brain Maker
The computing world has a lot to gain from neural networks. Professional, Software: Artificial Intelligence, Technical
Analysis of Stocks & Commodities, Bonus Issue 1994.
Their ability to learn by example makes them very flexible and
powerful. Furthermore there is no need to devise an algorithm [12] Carlton F. Vogt Jr., Brain Maker: This Will Teach Your
in order to perform a specific task; i.e. there is no need to Computer to Think, Design News, December 3, 1990.
understand the internal mechanisms of that task. They are also [13] John Vester, Artificial Intelligence and Real Life, Computor
very well suited for real time systems because of their fast Edge, October 22, 1993.
respond computational times which are due to their parallel
architecture. [14] George W. Dombi and Jeannette Lawrence, Analysis of protein
transmembrane helicalregions by a neural network, Protein
Neural networks also contribute to other areas of research Science (1994), 3:557-566.
such as neurology and psychology. They are regularly used to
model parts of living organisms and to investigate the internal [15] Henrik Lundstedt, Neural Networks and Predictions of
Solar-terrestrial Effects, Lund Observatory, Lund, Sweden, 1990.