BASICS IN NEURAL NETWORKS

WHAT IS ARTIFICIAL NEURAL NETWORK?

An Artificial Neural Network (ANN) is a mathematical model that tries to simulate the structure and
functionalities of biological neural networks. Basic building block of every artificial neural network is
artificial neuron, that is, a simple mathematical model (function). Such a model has three simple sets of
rules: multiplication, summation and activation. At the entrance of artificial neuron the inputs are
weighted what means that every input value is multiplied with individual weight. In the middle section of
artificial neuron is sum function that sums all weighted inputs and bias. At the exit of artificial neuron
the sum of previously weighted inputs and bias is passing through activation function that is also called

transfer function.
BIOLOGICAL NEURON STRUCTURE AND FUNCTIONS.

A neuron, or nerve cell, is an electrically excitable cell that communicates with other cells via
specialized connections called synapses. It is the main component of nervous tissue. Neurons are
typically classified into three types based on their function. Sensory neurons respond to stimuli such as
touch, sound, or light that affect the cells of the sensory organs, and they send signals to the spinal cord
or brain. Motor neurons receive signals from the brain and spinal cord to control everything from muscle
contractions to glandular output. Interneurons connect neurons to other neurons within the same region
of the brain or spinal cord. A group of connected neurons is called a neural circuit.

A typical neuron consists of a cell body (soma), dendrites, and a single axon. The soma is usually
compact. The axon and dendrites are filaments that extrude from it. Dendrites typically branch profusely
and extend a few hundred micrometers from the soma. The axon leaves the soma at a swelling called the
axon hillock, and travels for as far as 1 meter in humans or more in other species. It branches but usually
maintains a constant diameter. At the farthest tip of the axon's branches are axon terminals, where the
neuron can transmit a signal across the synapse to another cell. Neurons may lack dendrites or have no
axon. The term neurite is used to describe either a dendrite or an axon, particularly when the cell is
undifferentiated.

The soma is the body of the neuron. As it contains the nucleus, most protein synthesis occurs here. The
nucleus can range from 3 to 18 micrometers in diameter.

The dendrites of a neuron are cellular extensions with many branches. This overall shape and structure is
referred to metaphorically as a dendritic tree. This is where the majority of input to the neuron occurs
via the dendritic spine.

The axon is a finer, cable-like projection that can extend tens, hundreds, or even tens of thousands of
times the diameter of the soma in length. The axon primarily carries nerve signals away from the soma,
and carries some types of information back to it. Many neurons have only one axon, but this axon
may—and usually will—undergo extensive branching, enabling communication with many target cells.
The part of the axon where it emerges from the soma is called the axon hillock. Besides being an
anatomical structure, the axon hillock also has the greatest density of voltage-dependent sodium
channels. This makes it the most easily excited part of the neuron and the spike initiation zone for the
axon. In electrophysiological terms, it has the most negative threshold potential.

While the axon and axon hillock are generally involved in information outflow, this region can also
receive input from other neurons.

The axon terminal is found at the end of the axon farthest from the soma and contains synapses.
Synaptic boutons are specialized structures where neurotransmitter chemicals are released to
communicate with target neurons. In addition to synaptic boutons at the axon terminal, a neuron may
have en passant boutons, which are located along the length of the axon.

Most neurons receive signals via the dendrites and soma and send out signals down the axon. At the
majority of synapses, signals cross from the axon of one neuron to a dendrite of another. However,
synapses can connect an axon to another axon or a dendrite to another dendrite. The signaling process is
partly electrical and partly chemical. Neurons are electrically excitable, due to maintenance of voltage
gradients across their membranes. If the voltage changes by a large amount over a short interval, the
neuron generates an all-or-nothing electrochemical pulse called an action potential. This potential
travels
rapidly along the axon, and activates synaptic connections as it reaches them. Synaptic signals may be
excitatory or inhibitory, increasing or reducing the net voltage that reaches the soma.

In most cases, neurons are generated by neural stem cells during brain development and childhood.
Neurogenesis largely ceases during adulthood in most areas of the brain. However, strong evidence
supports generation of substantial numbers of new neurons in the hippocampus and olfactory bulb.
STRUCTURE AND FUNCTIONS OF ARTIFICIAL NEURON.

An artificial neuron is a mathematical function conceived as a model of biological neurons, a neural

network. Artificial neurons are elementary units in an artificial neural network. The artificial neuron
receives one or more inputs (representing excitatory postsynaptic potentials and inhibitory postsynaptic
potentials at neural dendrites) and sums them to produce an output (or activation, representing a neuron's
action potential which is transmitted along its axon). Usually each input is separately weighted, and the
sum is passed through a non-linear function known as an activation function or transfer function. The
transfer functions usually have a sigmoid shape, but they may also take the form of other non-linear
functions, piecewise linear functions, or step functions. They are also often monotonically increasing,
continuous, differentiable and bounded. The thresholding function has inspired building logic gates
referred to as threshold logic; applicable to building logic circuits resembling brain processing. For
example, new devices such as memristors have been extensively used to develop such logic in recent
times.
STATE THE MAJOR DIFFERENCES BETWEEN BIOLOGICAL AND ARTIFICIANEURAL
NETWORKS

1. Size: Our brain contains about 86 billion neurons and more than a 100 synapses (connections). The
number of “neurons” in artificial networks is much less than that.

2. Signal transport and processing: The human brain works asynchronously, ANNs work synchronously.

3. Processing speed: Single biological neurons are slow, while standard neurons in ANNs are fast.

4. Topology: Biological neural networks have complicated topologies, while ANNs are often in a tree
structure.

5. Speed: certain biological neurons can fire around 200 times a second on average. Signals travel at
different speeds depending on the type of the nerve impulse, ranging from 0.61 m/s up to 119 m/s.
Signal travel speeds also vary from person to person depending on their sex, age, height, temperature,
medical condition, lack of sleep etc. Information in artificial neurons is carried over by the continuous,
floating point number values of synaptic weights. There are no refractory periods for artificial neural
networks (periods while it is impossible to send another action potential, due to the sodium channels
being lock shut) and artificial neurons do not experience “fatigue”: they are functions that can be
calculated as many times and as fast as the computer architecture would allow.

6. Fault-tolerance: biological neuron networks due to their topology are also fault-tolerant. Artificial
neural networks are not modeled for fault tolerance or self regeneration (similarly to fatigue, these ideas
are not applicable to matrix operations), though recovery is possible by saving the current state (weight
values) of the model and continuing the training from that save state.

7. Power consumption: the brain consumes about 20% of all the human body’s energy — despite it’s
large cut, an adult brain operates on about 20 watts (barely enough to dimly light a bulb) being extremely
efficient. Taking into account how humans can still operate for a while, when only given some c-
vitamin rich lemon juice and beef tallow, this is quite remarkable. For benchmark: a single Nvidia
GeForce Titan X GPU runs on 250 watts alone, and requires a power supply. Our machines are way less
efficient than biological systems. Computers also generate a lot of heat when used, with consumer GPUs
operating safely between 50–80°Celsius instead of 36.5–37.5 °C.

8. Learning: we still do not understand how brains learn, or how redundant connections store and recall
information. By learning, we are building on information that is already stored in the brain. Our
knowledge deepens by repetition and during sleep, and tasks that once required a focus can be executed
automatically once mastered. Artificial neural networks in the other hand, have a predefined model,
where no further neurons or connections can be added or removed. Only the weights of the connections
(and biases representing thresholds) can change during training. The networks start with random weight
values and will slowly try to reach a point where further changes in the weights would no longer
improve performance. Biological networks usually don't stop / start learning. ANNs have different
fitting (train) and prediction (evaluate) phases.

9. Field of application: ANNs are specialized. They can perform one task. They might be perfect at
playing chess, but they fail at playing go (or vice versa). Biological neural networks can learn completely
new tasks.

10. Training algorithm: ANNs use Gradient Descent for learning. Human brains use something
different (but we don't know what).
 BRIEFLY EXPLAIN THE BASIC BUILDING BLOCKS OF ARTIFICIAL NEURAL NETWORKS.

Processing of ANN depends upon the following three building blocks:

1. Network Topology
2. Adjustments of Weights or Learning
3. Activation Functions

1. Network Topology: A network topology is the arrangement of a network along with its nodes
and connecting lines. According to the topology, ANN can be classified as the following kinds:

A. Feed forward Network: It is a non-recurrent network having processing units/nodes in layers

and all the nodes in a layer are connected with the nodes of the previous layers. The connection
has different weights upon them. There is no feedback loop means the signal can only flow in
one direction, from input to output. It may be divided into the following two types:

 Single layer feed forward network: The concept is of feed forward ANN having only
one weighted layer. In other words, we can say the input layer is fully connected to the
output layer.

 Multilayer feed forward network: The concept is of feed forward ANN having more
than one weighted layer. As this network has one or more layers between the input and the
output layer, it is called hidden layers.
 B. Feedback Network: As the name suggests, a feedback network has feedback paths, which means
the signal can flow in both directions using loops. This makes it a non-linear dynamic system,
which changes continuously until it reaches a state of equilibrium. It may be divided into the
following types:

 Recurrent networks: They are feedback networks with closed loops. Following are the two
types of recurrent networks.
 Fully recurrent network: It is the simplest neural network architecture because all nodes
are connected to all other nodes and each node works as both input and output.

 Jordan network − It is a closed loop network in which the output will go to the input again
as feedback as shown in the following diagram.

2. Adjustments of Weights or Learning: Learning, in artificial neural network, is the method of

modifying the weights of connections between the neurons of a specified network. Learning in ANN can
be classified into three categories namely supervised learning, unsupervised learning, and reinforcement
learning.
 Supervised Learning: As the name suggests, this type of learning is done under the supervision of a
teacher. This learning process is dependent. During the training of ANN under supervised learning, the
input vector is presented to the network, which will give an output vector. This output vector is
compared with the desired output vector. An error signal is generated, if there is a difference between
the actual output and the desired output vector. On the basis of this error signal, the weights are
adjusted until the actual output is matched with the desired output.

Unsupervised Learning: As the name suggests, this type of learning is done without the supervision of
a teacher. This learning process is independent. During the training of ANN under unsupervised
learning, the input vectors of similar type are combined to form clusters. When a new input pattern is
applied, then the neural network gives an output response indicating the class to which the input pattern
belongs. There is no feedback from the environment as to what should be the desired output and if it is
correct or incorrect. Hence, in this type of learning, the network itself must discover the patterns and
features from the input data, and the relation for the input data over the output.

Reinforcement Learning: As the name suggests, this type of learning is used to reinforce or strengthen
the network over some critic information. This learning process is similar to supervised learning,
however we might have very less information. During the training of network under reinforcement
learning, the network receives some feedback from the environment. This makes it somewhat similar to
supervised learning. However, the feedback obtained here is evaluative not instructive, which means
there is no teacher as in supervised learning. After receiving the feedback, the network performs
adjustments of the weights to get better critic information in future.

3. Activation Functions: An activation function is a mathematical equation that determines the output of
each element (perceptron or neuron) in the neural network. It takes in the input from each neuron and
transforms it into an output, usually between one and zero or between -1 and one. It may be defined as
the extra force or effort applied over the input to obtain an exact output. In ANN, we can also apply
activation functions over the input to get the exact output. Followings are some activation functions of
interest:
i) Linear Activation Function: It is also called the identity function as it performs no input editing. It
can be defined as: F(x) = x

ii) Sigmoid Activation Function: It is of two type as follows −

 Binary sigmoidal function: This activation function performs input editing between 0 and 1. It
is positive in nature. It is always bounded, which means its output cannot be less than 0 and
more than 1. It is also strictly increasing in nature, which means more the input higher would be
the output. It can be defined as

F(x)=sigm(x)=11+exp(−x)F(x)=sigm(x)=11+exp(−x)
 Bipolar sigmoidal function: This activation function performs input editing between -1 and 1.
It can be positive or negative in nature. It is always bounded, which means its output cannot be
less than -1 and more than 1. It is also strictly increasing in nature like sigmoid function. It can
be defined as

F(x)=sigm(x)=21+exp(−x)−1=1−exp(x)1+exp(x)

WHAT IS A NEURAL NETWORK ACTIVATION FUNCTION?

In a neural network, inputs, which are typically real values, are fed into the neurons in the network. Each
neuron has a weight, and the inputs are multiplied by the weight and fed into the activation
function. Each neuron’s output is the input of the neurons in the next layer of the network, and so the
inputs cascade through multiple activation functions until eventually, the output layer generates a
prediction. Neural networks rely on nonlinear activation functions—the derivative of the activation
function helps the network learn through the backpropagation process.
SOME COMMON ACTIVATION FUNCTIONS INCLUDE THE FOLLOWING:

1. The sigmoid function has a smooth gradient and outputs values between zero and one. For
very high or low values of the input parameters, the network can be very slow to reach a
prediction, called the vanishing gradient problem.
2. The TanH function is zero-centered making it easier to model inputs that are strongly
negative strongly positive or neutral.
3. The ReLu function is highly computationally efficient but is not able to process inputs
that approach zero or negative.
4. The Leaky ReLu function has a small positive slope in its negative area, enabling it to process
zero or negative values.
5. The Parametric ReLu function allows the negative slope to be learned,
performing backpropagation to learn the most effective slope for zero and
negative input values.
6. Softmax is a special activation function use for output neurons. It normalizes outputs for each
class between 0 and 1, and returns the probability that the input belongs to a specific class.
7. Swish is a new activation function discovered by Google researchers. It performs better than
ReLu with a similar level of computational efficiency.

APPLICATIONS OF ANN

1. Data Mining: Discovery of meaningful patterns (knowledge) from large volumes of data.
2. Expert Systems: A computer program for decision making that simulates thought process
of a human expert.
3. Fuzzy Logic: Theory of approximate reasoning.
4. Artificial Life: Evolutionary Computation, Swarm Intelligence.
5. Artificial Immune System: A computer program based on the biological immune system.
6. Medical: At the moment, the research is mostly on modelling parts of the human body and
recognizing diseases from various scans (e.g. cardiograms, CAT scans, ultrasonic scans,
etc.).Neural networks are ideal in recognizing diseases using scans since there is no need to
provide a specific algorithm on how to identify the disease. Neural networks learn by example so
the details of how to recognize the disease are not needed. What is needed is a set of examples
that are representative of all the variations of the disease. The quantity of examples is not as
important as the 'quantity'. The examples need to be selected very carefully if the system is to
perform reliably and efficiently.
 7. Computer Science: Researchers in quest of artificial intelligence have created spin offs like
dynamic programming, object oriented programming, symbolic programming, intelligent storage
management systems and many more such tools. The primary goal of creating an artificial
intelligence still remains a distant dream but people are getting an idea of the ultimate path,
which could lead to it.
8. Aviation: Airlines use expert systems in planes to monitor atmospheric conditions and system
status. The plane can be put on autopilot once a course is set for the destination.
9. Weather Forecast: Neural networks are used for predicting weather conditions. Previous data is
fed to a neural network, which learns the pattern and uses that knowledge to predict weather
patterns.
10. Neural Networks in business: Business is a diverted field with several general areas of
specialization such as accounting or financial analysis. Almost any neural network application
would fit into one business area or financial analysis.
11. There is some potential for using neural networks for business purposes, including resource
allocation and scheduling.
12. There is also a strong potential for using neural networks for database mining, which is,
searching for patterns implicit within the explicitly stored information in databases. Most of the
funded work in this area is classified as proprietary. Thus, it is not possible to report on the full
extent of the work going on. Most work is applying neural networks, such as the Hopfield-Tank
network for optimization and scheduling.
13. Marketing: There is a marketing application which has been integrated with a neural network
system. The Airline Marketing Tactician (a trademark abbreviated as AMT) is a computer
system made of various intelligent technologies including expert systems. A feed forward neural
network is integrated with the AMT and was trained using back-propagation to assist the
marketing control of airline seat allocations. The adaptive neural approach was amenable to rule
expression. Additionally, the application's environment changed rapidly and constantly, which
required a continuously adaptive solution.
14. Credit Evaluation: The HNC company, founded by Robert Hecht-Nielsen, has developed several
neural network applications. One of them is the Credit Scoring system which increases the
profitability of the existing model up to 27%. The HNC neural systems were also applied to
mortgage screening. A neural network automated mortgage insurance under writing system was
developed by the Nestor Company. This system was trained with 5048 applications of which
2597 were certified. The data related to property and borrower qualifications. In a conservative
mode the system agreed on the under writers on 97% of the cases. In the liberal model the system
agreed 84% of the cases. This is system run on an Apollo DN3000 and used 250K memory while
processing a case file in approximately 1 sec.

ADVANTAGES OF ANN

1. Adaptive learning: An ability to learn how to do tasks based on the data given for training or
initial experience.
2. Self-Organisation: An ANN can create its own organisation or representation of the
information it receives during learning time.
3. Real Time Operation: ANN computations may be carried out in parallel, and special hardware devices
are being designed and manufactured which take advantage of this capability.
4. Pattern recognition: is a powerful technique for harnessing the information in the data and
generalizing about it. Neural nets learn to recognize the patterns which exist in the data set.
5. The system is developed through learning rather than programming.. Neural nets teach themselves
the patterns in the data freeing the analyst for more interesting work.
6. Neural networks are flexible in a changing environment. Although neural networks may take some
time to learn a sudden drastic change they are excellent at adapting to constantly changing information.
7. Neural networks can build informative models whenever conventional approaches fail. Because
neural networks can handle very complex interactions they can easily model data which is too difficult
to model with traditional approaches such as inferential statistics or programming logic.
8. Performance of neural networks is at least as good as classical statistical modelling, and better on
most problems. The neural networks build models that are more reflective of the structure of the data in
significantly less time.

LIMITATIONS OF ANN

In this technological era everything has Merits and some Demerits in others words there is a Limitation
with every system which makes this ANN technology weak in some points. The various Limitations of
ANN are:-

1) ANN is not a daily life general purpose problem solver.

2) There is no structured methodology available in ANN.
3) There is no single standardized paradigm for ANN development.
4) The Output Quality of an ANN may be unpredictable.
5) Many ANN Systems does not describe how they solve problems.
6) Black box Nature
7) Greater computational burden.
8) Proneness to over fitting.
9) Empirical nature of model development.

ARTIFICIAL NEURAL NETWORK CONCEPTS/TERMINOLOGY

Here is a glossary of basic terms you should be familiar with before learning the details of neural networks.

Inputs: Source data fed into the neural network, with the goal of making a decision or prediction
about the data. Inputs to a neural network are typically a set of real values; each value is fed into one
of the neurons in the input layer.
Training Set: A set of inputs for which the correct outputs are known, used to train the neural network.
Outputs : Neural networks generate their predictions in the form of a set of real values or boolean
decisions. Each output value is generated by one of the neurons in the output layer.
Neuron/perceptron: The basic unit of the neural network. Accepts an input and generates a prediction.
Each neuron accepts part of the input and passes it through the activation function. Common
activation functions are sigmoid, TanH and ReLu. Activation functions help generate output values
within an acceptable range, and their non-linear form is crucial for training the network.

Weight Space: Each neuron is given a numeric weight. The weights, together with the activation
function, define each neuron’s output. Neural networks are trained by fine-tuning weights, to
discover the optimal set of weights that generates the most accurate prediction.
Forward Pass: The forward pass takes the inputs, passes them through the network and allows each
neuron to react to a fraction of the input. Neurons generate their outputs and pass them on to the next
layer, until eventually the network generates an output.
Error Function: Defines how far the actual output of the current model is from the correct output.
When training the model, the objective is to minimize the error function and bring output as close as
possible to the correct value.
Backpropagation: In order to discover the optimal weights for the neurons, we perform a backward
pass, moving back from the network’s prediction to the neurons that generated that prediction. This
is called backpropagation. Backpropagation tracks the derivatives of the activation functions in each
successive neuron, to find weights that bring the loss function to a minimum, which will generate the
best prediction. This is a mathematical process called gradient descent.
Bias and Variance: When training neural networks, like in other machine learning techniques, we try
to balance between bias and variance. Bias measures how well the model fits the training set—able to
correctly predict the known outputs of the training examples. Variance measures how well the model
works with unknown inputs that were not available during training. Another meaning of bias is a “
bias neuron” which is used in every layer of the neural network. The bias neuron holds the number 1,
and makes it possible to move the activation function up, down, left and right on the number graph.
Hyperparameters: A hyper parameter is a setting that affects the structure or operation of the neural
network. In real deep learning projects, tuning hyper parameters is the primary way to build a network
that provides accurate predictions for a certain problem. Common hyper parameters include the
number of hidden layers, the activation function, and how many times (epochs) training should be
repeated.