 UNDERSTANDING NEURAL NETWORKS

WHAT IS A NEURAL NETWORK:

 A neural network is a mathematical function that takes in inputs and
produces outputs.
 A neural network is a computational graph through which multidimensional
arrays flow.
 A neural network is made up of layers, each of which can be thought of as
having a number of “neurons.”
 A neural network is a universal function approximator that can in theory
represent the solution to any supervised learning problem.

TRAINING SET VS TESTING SET

 If any dataset received as being a sample from a population.
 Our goal is always to find a model that uncovers relationships in the
population.
 There is always a danger that we build a model that picks up relationships
that exist in the sample but not in the population.
 This is a problem known as overfitting.
 How can we detect whether a model structure we use is likely to have this
problem?
 The solution is to split our sample into a training set and a testing set. We
use the training data to train the model
 Then we evaluate the model on the testing set to estimate its performance
 CONVOLUTIONAL NEURAL NETWORKS
  CNNs are the standard neural network architecture used for prediction
when the input observations are images.
 The filters are applied to the input image to extract features such as edges,
textures, and shapes.
 The output of the pooling layers is then passed through one or more fully
connected layers, which are used to make a prediction or classify the
image.
 CNN are very satisfactory at picking up on design in the input image, such
as lines, gradients, circles, or even eyes and faces.

Representation of image in the form of an array of pixel values.

 REPRESENTATION LEARNING
 Neural networks can automatically discover combinations of the original
features that are important through their training process.
 The process starts by creating initially random combinations of the original
features via multiplication by a random weight matrix
 From the training, the neural network learns to refine combinations that are
helpful and discard those that aren’t helpful combinations.
 This process of learning which combinations of features are important is known
as representation learning.
  THE CONVOLUTION OPERATION:
Here are the three elements that enter into the convolution operation
 Input image
 Feature detector
 Feature map
 You place it over the input image beginning from the top-left corner within
the borders you see demarcated above, and then you count the number of
cells in which the feature detector matches the input image.
 The number of matching cells is then inserted in the top-left cell of the
feature map.
 You then move the feature detector one cell to the right and do the same
thing. This movement is called a and since we are moving the feature
detector one cell at time, that would be called a stride of one pixel.
 What you will find in this example is that the feature detector's middle-left
cell with the number 1 inside it matches the cell that it is standing over
inside the input image. That's the only matching cell, and so you write “1”
in the next cell in the feature map, and so on and so forth.  After you
have gone through the whole first row, you can then move it over to the
next row and go through the same process.
  THE MULTICHANNEL CONVOLUTION OPERATION
 What actually happens in a convolutional Layer in a neural network goes
one step further
 we’ll create f sets of n features, each with a corresponding (initially
random) set of weights defining a visual pattern
 It detection at each location in the input image will be captured in the
feature map.
 These f feature maps will be created via f convolution operations.

 CONVOLUTIONAL LAYERS
A convolution neural network has multiple hidden layers that help in
extracting information from an image.
The four important layers in CNN are:
 Convolution layer
 ReLU layer
 Pooling layer
 Fully connected layer

CONVOLUTION LAYER:
 This is the first step in the process of extracting valuable features from an
image.
 A convolution layer has several filters that perform the convolution
operation.
 Every image is considered as a matrix of pixel values.
 Consider the following 5x5 image whose pixel values are either 0 or 1.
 There’s also a filter matrix with a dimension of 3x3.
 Slide the filter matrix over the image and compute
 the dot product to get the convolved feature matrix

ReLU layer:
 ReLU stands for the rectified linear unit.
 The feature maps are extracted, the next step is to move them to a ReLU
layer.
 ReLU performs an element-wise operation and sets all the negative pixels to
0.
 It introduces non-linearity to the network, and the generated output is
a rectified feature map.

 POOLING LAYER

 Pooling is a down-sampling operation that reduces the dimensionality of

the feature map.
 The rectified feature map now goes through a pooling layer to generate a
pooled feature map

 RECURRENT NEURAL NETWORKS :

 Recurrent neural networks (RNNs) are a class of neural network that are
helpful in modeling sequence data
 Derived from feedforward networks, RNNs exhibit similar behavior to how
human brains function.
 Simply put: recurrent neural networks produce predictive results in
sequential data that other algorithms can’t.
TYPES OF RECURRENT NEURAL NETWORKS:
 ONE TO ONE
 ONE TO MANY
 MANY TO ONE
 MANY TO MANY

 PyTorch Tensors
 A Pytorch Tensor is basically the same as a NumPy array
 PyTorch does not know anything about deep learning or computational
graphs or gradients and is just a generic n-dimensional array to be used
for arbitrary numeric computation.
 The biggest difference between a NumPy array and a PyTorch Tensor is
that a PyTorch Tensor can run on either CPU or GPU.
How to create a Tensor
import torch
V_data = [1, 2, 3, 4, 5]
V = torch.tensor(V_data)
print(V)
Output:
tensor([1, 2, 3, 4, 5])

You can also create a tensor of random data with a given dimensionality
import torch
x = torch.randn((3, 4, 5))
print(x)
 tensor([[[ 0.8332, -0.2102, 0.0213, 0.4375, -0.9506],
[ 0.0877, -1.5845, -0.1520, 0.3944, -0.7282],
[-0.6923, 0.0332, -0.4628, -0.9127, -1.4349],
[-0.3641, -0.5880, -0.5963, -1.4126, 0.5308]],

[[ 0.4492, -1.2030, 2.5985, 0.8966, 0.4876],

[ 0.5083, 1.4515, 0.6496, 0.3407, 0.0093],
[ 0.1237, 0.3783, -0.7969, 1.4019, 0.0633],
[ 0.4399, 0.3827, 1.2231, -0.0674, -1.0158]],

[[-0.2490, -0.5475, 0.6201, -2.2092, 0.8405],

[ 0.1684, -1.0118, 0.7414, -3.3518, -0.3209],
[ 0.6543, 0.1956, -0.2954, 0.1055, 1.6523],
[-0.9872, -2.0118, -1.6609, 1.4072, 0.0632]]])

You can also create tensors using the following functions:

• torch.zeros(): Creates a new Tensor with all elements, initialized as zeros.

import torch
z= torch.zeros([3,3], dtype=torch.int32)
print(z)
Output:
tensor([[0, 0, 0],
[0, 0, 0],
[0, 0, 0]], dtype=torch.int32

• torch.ones(): Creates a new Tensor with all elements, initialized as ones.

import torch
z = torch.ones([3,3])
print(z)
Output:
tensor([[1., 1., 1.],
[1., 1., 1.],
[1., 1., 1.]])
 torch.full() and torch.full_like(): These functions return a Tensor of the
required size filled with required fill_value provided. The complete
prototype for torch.full() is:
import torch
 newTensor= torch.full((4, 3), 3.14,dtype= torch.float32)
print(newTensor)
Output:
tensor([[3.1400, 3.1400, 3.1400],
[3.1400, 3.1400, 3.1400],
[3.1400, 3.1400, 3.1400],
[3.1400, 3.1400, 3.1400]])

import torch
# Example for torch.full_like()
x = torch.full_like(newTensor,3.24, dtype=None )
print(x)
Output:
tensor([[3.2400, 3.2400, 3.2400],
[3.2400, 3.2400, 3.2400],
[3.2400, 3.2400, 3.2400],
[3.2400, 3.2400, 3.2400]])

TENSOR ATTRIBUTES:
Each tensor( torch.Tensor ) has a torch.dtype, torch.device,
and torch.layout attributes.
• torch.dtype: A torch.dtype is an object that represents the data type
of torch.Tensor. PyTorch has twelve different data types.
• torch.device: A torch.device is an object representing the device on
which a torch.Tensor is or will be allocated. The torch.device contains a
device type (‘cpu’ or ‘cuda’) and an optional device ordinal for the device
type.
• torch.layout: A torch.layout is an object that represents the memory
layout of a torch.Tensor. Currently, the torch supports two types of
memory layout.
 DEEP LEARNING WITH PYTORCH
Deep learning models have several elements that work together to produce
a trained model:
• A Model, which contains Layers
• An Optimizer
• A Loss
 • A Trainer
It turns out that with PyTorch, the Optimizer and the Loss are one-liners,
and the Model and Layers are straightforward.
A key feature of PyTorch is the ability to define models and layers as easy-
to-use objects that handle sending gradients backward and storing
parameters automatically, simply by having them inherit from
the torch.nn.Module class

PyTorchModel can also be written this way:

class PyTorchModel(nn.Module):

def __init__(self) -> None: super().__init__()

def forward(self, x: Tensor, inference: bool = False) -> Tensor:

o raise NotImplementedError()
 CONVOLUTIONAL NEURAL NETWORKS IN PYTORCH
The multichannel convolution operation had the following shapes for its
two inputs and its output:
•The data input shape [batch_size, in_channels, image_height,
image_width]
•The parameters input shape [in_channels, out_channels, filter_size,
filter_size]
•The output shape [batch_size, out_channels, image_height, image_width]

we have to do is define a PyTorchModel with its operations in

the __init__ function and the sequence of operations
defined in the forward function to begin to train.
A convolutional layer that transforms the input from 1 “channel” to 16
channels
• Another layer that transforms these 16 channels into 8 (with each
channel still containing 28 × 28 neurons)
• Two fully connected layers
The pattern of several convolutional layers followed by a smaller number of
fully connected layers is common for convolutional architectures; here, we just
use two of each:
class MNIST_ConvNet(PyTorchModel): def __init__(self): super().__init__()
self.conv1 = ConvLayer(1, 16, 5, activation=nn.Tanh(), dropout=0.8)
self.conv2 = ConvLayer(16, 8, 5, activation=nn.Tanh(), flatten=True,
dropout=0.8)
self.dense1 = DenseLayer(28 * 28 * 8, 32, activation=nn.Tanh(), dropout=0.8)
self.dense2 = DenseLayer(32, 10)
def forward(self, x: Tensor) -> Tensor: assert_dim(x, 4) x = self.conv1(x) x =
self.conv2(x) x = self.dense1(x) x = self.dense2(x) return x