DATA STRUCTURES

AND
ALGORITHMS
SWASTIK RAJ
22BCE0411
SLOT :- C2+TC2
FACULTY:- DR.
PARVEEN
SULTANA H
UNVIELIN
G THE
POWER OF
NEURAL
NETWORKS
IN THE
REAL
WORLD
OBJECTIVE

• The objective of this digital assignment is to provide a comprehensive

understanding of neural networks.
• It will cover the basic concepts, architecture, training, and applications of
neural networks.
 PART-1

INTRODUCTION TO NEURAL
NETWORKS
DEFINITION AND OVERVIEW
• A neural network is a series of algorithms that tries to recognize underlying
relationships in a set of data through a process that mimics the way the
human brain operates.
• In this sense, neural networks refer to systems of neurons, either organic or
artificial in nature.
• The concept of neural networks, which has its roots in artificial intelligence,
is swiftly gaining popularity in the development of trading system.
• Neural networks, also known as artificial neural
networks (ANNs) or simulated neural networks
(SNNs), are a subset of M and are at the heart of
deep learning algorithms.
• Their name and structure are inspired by the
human brain, mimicking the way that biological
neurons signal to one another.
HISTORY:-
• The simplest kind of feedforward neural network (FNN) is a
linear network, which consists of a single layer of output nodes;
the inputs are fed directly to the outputs via a series of weights.
• The sum of the products of the weights and the inputs is
calculated in each node.
• The mean squared errors between these calculated outputs and a
given target values are minimized by creating an adjustment to
the weights.
• This technique has been known for over two centuries as
the method of least squares or linear regression.
• It was used as a means of finding a good rough linear fit to a set
of points by Legendre (1805) and Gauss (1795) for the
prediction of planetary movement.
• Though the concept of integrated machines that can think has existed for centuries, there
have been the largest strides in neural networks in the past 100 years.
• In 1943, Warren McCulloch and Walter Pitts from the University of Illinois and the
University of Chicago published "A Logical Calculus of the Ideas Immanent in Nervous
Activity".
• The research analyzed how the brain could produce complex patterns and could be
simplified down to a binary logic structure with only true/false connections.
• Frank Rosenblatt from the Cornell Aeronautical Labratory was credited with the
development of perceptron in 1958.
• His research introduced weights to McColloch's and Pitt's work, and Rosenblatt leveraged
his work to demonstrate how a computer could use neural networks to detect imagines
and make inferences.
• Wilhelm Lenz and Ernst Ising created and analyzed the Ising
model (1925) which is essentially a non-learning artificial recurrent neural
network (RNN) consisting of neuron-like threshold elements.
• In 1972, Shun'ichi Amari made this architecture adaptive.
• His learning RNN was popularised by John Hopfield in 1982.
 PART II

ARCHITECTURE &
STRUCTURE
STRUCTURE :-
NEURON
 PROPOGATION FUNCTION
 BIAS
IMPORTANT CONCEPT TO UNDERSTAND:-
MULTI – LAYERED PERCEPTRON
• In a multi-layered perceptron (MLP), perceptrons are arranged in interconnected layers. The
input layer collects input patterns. The output layer has classifications or output signals to
which input patterns may map. For instance, the patterns may comprise a list of quantities
for technical indicators about a security; potential outputs could be “buy,” “hold” or “sell.”
• Hidden layers fine-tune the input weightings until the neural network’s margin of error is
minimal. It is hypothesized that hidden layers extrapolate salient features in the input data
that have predictive power regarding the outputs. This describes feature extraction, which
accomplishes a utility similar to statistical techniques such as principal component analysis.
 TYPES OF
NEURAL NETWORKS
ON THE BASIS OF
ARCHITECTURE
 FEED FORWARD NEURAL NETWORKS

• Feed-forward neural networks are one of the more simple types of neural networks.
• It conveys information in one direction through input nodes; this information continues to be processed in
this single direction until it reaches the output mode.
• Feed-forward neural networks may have hidden layers for functionality, and this type of most often used
for facial recognition technologies.
• In this network, the information moves in only one direction—forward—from the input nodes, through
the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network.
 RECURRENT NEURAL NETWORKS

• A more complex type of neural network, recurrent neural networks take the output of a processing
node and transmit the information back into the network. This results in theoretical "learning" and
improvement of the network. Each node stores historical processes, and these historical processes are
reused in the future during processing.
• This becomes especially critical for networks in which the prediction is incorrect; the system
will attempt to learn why the correct outcome occurred and adjust accordingly. This type of neural
network is often used in text-to-speech applications.
 CONVOLUTIONAL NEURAL NETWORKS

• Convolutional neural networks, also called ConvNets or CNNs, have several layers in which data is
sorted into categories.
• These networks have an input layer, an output layer, and a hidden multitude of convolutional layers in
between.
• The layers create feature maps that record areas of an image that are broken down further until they
generate valuable outputs.
• These layers can be pooled or entirely connected, and these networks are especially beneficial for image
recognition applications.
 DECONVOLUTIONAL NEURAL NETWORKS

• Deconvolutional neural networks simply work in reverse of convolutional neural

networks.
• The application of the network is to detect items that might have been
recognized as important under a convolutional neural network.
• These items would likely have been discarded during the convolutional neural
network execution process.
• This type of neural network is also widely used for image analysis or processing.
 MODULAR NEURAL NETWORKS

• Modular neural networks contain several networks that work independently from one
another.
• These networks do not interact with each other during an analysis process.
• Instead, these processes are done to allow complex, elaborate computing processes to
be done more efficiently.
• Similar to other modular industries such as modular real estate, the goal of the
network independence is to have each module responsible for a particular part of an
overall bigger picture.
 PART III
CODES AND PROGRAMS
RELATED
TO NEURAL NETWORKS
• Neural networks can be interpreted in two ways. The “graph” way and the
“matrix” way.

• That’s how neural networks are usually represented, vertices(neurons) +

oriented edges(dendrites), a directed graph. But in practice they’re used with
matrices because computations are much faster on GPUs that way.
• Note: - We will use c++ programming language here because in the case of
machine learning tools and algorithms, c++, java and python are considered
much useful than any other language.
• The easy version of how a neural network does the forward pass is as follows:
1. create a neural network following the above architecture
2. put values in the input neurons (exp: 3 neurons with value 1, 5 and 3)
3. trigger the input layer. The values are multiplied with a weight linking each
input neuron to each neuron of the 2nd layer
(3neurons*4neurons=12scalars/weights/edges). Then sum the 3 inputs*weights
for each of the 4 neurons of the 2nd layer. (exp: 1*a1+5*b1+3*c1 for 1st neuron
of 2nd layer, then 1*a2+5*b2+3*c2 etc.)
4. trigger the 2nd layer and do 3. between the 3rd layer and the 2nd layer (the
same way it was done between the 2nd and the 1st layer)
5. read the two values accumulated by the 3rd/output layer.
• The forward pass is easy. The same computations can be done with matrices.
Framework Design
1. Folders
• Neural: Contains classes related to the neural network.
• Dataset: Contains one simple class for reading a dataset.
• Optimizer: Contains a generic optimizer class, a classical backpropagation
optimizer and an exotic optimizer.
• Misc: Contains activation functions and their derivatives.
2. Neural Classes
• NeuralNetwork.h/.cpp: the main class containing a model. A
NeuralNetwork is composed of multiple layers.
• Layer.h/.cpp: A Layer is composed of multiple neurons. It has a type
(INPUT/STANDARD/OUTPUT) because it doesn’t work the same way
depending on its type and it has an activation function.
• Neuron.h/.cpp: A Neuron has a parent layer. A neuron accumulates the
output of the edges connected to it (_accumulated), it outputs that input to its
edges after processing it with its _activation_function (which you can change
on a per-neuron basis). A neuron knows its _next and _previous edges.
• Edge.h/.cpp: An Edge knows its next neuron (_n), the neuron its coming
from (_nb), it has a weight (_w), it can memorize how much its weight was
shift (_last_shift) last time it was, and it has a backpropagation memory so
that it can retain a part of the chain rule.
3. Dataset / Optimizer
• Dataset.h/.cpp: It gives the data. One input/output is just a vector (again, no
matrices, no images). So the list of all inputs is a vector of vector. The
Dataset class is also responsible for doing the train/test split.
• Optimizer.h/.cpp: Abstract class that links the network and the dataset
together. Child classes must implement a “minimize” method that is called
during the optimization process to minimize the loss.
• Backpropagation.h/.cpp: Implementation of the backpropagation optimizer
(though the backpropagation computations are in the neuron class for
simplification). This class is mostly there to call these methods in the neuron
class and to apply the new computed weights. It uses a batch size, the
learning rate is a global variable because it’s common to all implemented
optimizers and it’s easier to access it from anywhere.
BUILDING A NEURAL NETWORK

• Instanciate the class.

• Add an input layer, specify the number of neurons (size).
• Then add hidden layers (standard), specify the number of neurons (size=5 neurons) and an
activation function (sigmoid).
• Finally, add an output layer, its size (1 output value) and an activation function (sigmoid).
• The “autogenerate” adds the edges (connectComplete) so that the layers are fullyconnected,
and it also randomizes all the weights:
PREDICTING A VALUE
• The inputs are set up in the _accumulated variable of the input neurons, then
trigger() calls each layer to fire successively, it computes the output of that
layer and put it in the input of the neurons in the next layer.
• Finally the output() just returns the activation_function(accumulated) for each
neuron of the last/output layer.
BACKPROPOGATION
• You need the target (the values we’re trying to learn) to do the computations for the output layer
(the “if” block), then you need to keep in memory a part of the computations to apply the chain
rule in all other layers (the “else” block).
• dw stands for delta_weight, so for each neuron you’re essentially computing a delta to apply to
each of its _previous edges, it’s backpropagation starting from the output layer.
• The chain rule puts Δerr/Δw in three operations which gives the three d1, d2 and (d3) the previous
output you have in the code (the line in if section with dw[i] = …). Basically dw[i] = Δerr/Δw.
• Functions are named so that they should be easy to read and to understand.
• That’s basic backpropagation. No momentum, no tricks, nothing but gradients. The learning rate
applies when we add the delta_weight to the weight (shiftBackWeights method of Neuron calls
shiftWeight of Edge).
ACTIVATION FUNCTION
• Scalar activationFunction(Scalar x)
•{
• return tanhf(x);
•}

• Scalar activationFunctionDerivative(Scalar x)
•{
• return 1 - tanhf(x) * tanhf(x);
•}
TRAINING NEURAL NETWORKS
• void NeuralNetwork::train(std::vector<RowVector*> input_data, std::vector<RowVector*>
output_data)
•{
• for (uint i = 0; i < input_data.size(); i++) {
• std::cout << "Input to neural network is : " << *input_data[i] << std::endl;
• propagateForward(*input_data[i]);
• std::cout << "Expected output is : " << *output_data[i] << std::endl;
• std::cout << "Output produced is : " << *neuronLayers.back() << std::endl;
• propagateBackward(*output_data[i]);
• std::cout << "MSE : " << std::sqrt((*deltas.back()).dot((*deltas.back())) /
deltas.back()->size()) << std::endl;
• }
•}
 IMPORTANT CONCEPTUAL FACT
• Data Structure Aspect: Neural networks rely on various data structures to efficiently store and manipulate
information during computation. The fundamental data structure in a neural network is the artificial neuron or
perceptron. Neurons are interconnected in layers, forming a network architecture. Each neuron typically holds a
set of weights and biases that are adjusted during the learning process. The network structure can be represented
using data structures like arrays, matrices, or graphs, depending on the specific type of neural network.
• Algorithms Aspect: Neural networks employ algorithms for training and inference. Two key algorithmic
components of neural networks are forward propagation and backpropagation.
1. Forward Propagation: In forward propagation, input data flows through the network, layer by layer, from the
input layer to the output layer. The neurons in each layer perform computations using activation functions and
the weighted inputs from the previous layer. This process continues until the output layer produces a prediction
or an output value.
2. Backpropagation: Backpropagation is an algorithm used to train neural networks by adjusting the weights and
biases of the neurons. It works by propagating the error or the difference between the predicted output and the
desired output, backward through the network. The gradients of the error with respect to the weights and biases
are computed and used to update the parameters through optimization techniques like gradient descent.
• Additionally, various optimization algorithms, such as stochastic gradient descent, Adam, or RMSprop, are used
to efficiently update the weights and biases during the training process. These algorithms leverage concepts from
numerical optimization and iterative methods to find the optimal values for the network parameters.
• Neural Networks
Pros
• Can often work more efficiently and for longer than humans
• Can be programmed to learn from prior outcomes to strive to make smarter future
calculations
• Often leverage online services that reduce (but do not eliminate) systematic risk
• Are continually being expanded in new fields with more difficult problems
Cons
• Still rely on hardware that may require labor and expertise to maintain
• May take long periods of time to develop the code and algorithms
• May be difficult to assess errors or adaptions to the assumptions if the system is self-
learning but lacks transparency
• Usually report an estimated range or estimated amount that may not actualize
REAL LIFE APPLICATIONS
TOP 27 ARTIFICIAL NEURAL NETWORK
SOFTWARES