Neural Networks 1 Artificial Neural Networks • Vaguely inspired by the biological neural networks

that constitute animal brains 2 • The human brain is made of close to 100 billions of neurons
interconnected by synapses. • A neuron processes and transmits information through electrical and
chemical signals that are carried via the synapses • Neurons can connect to each other to form
neural networks – each neuron can be connected with about 5,000 other neurons 2 Artificial NNs Vs
Biological NN • Size: our brain contains about 86 billion neurons and more than a 100 trillion (or
according to some estimates 1000 trillion) synapses (connections). The number of “neurons” in
artificial networks is much less than that (usually in the ballpark of 10–1000) but comparing their
numbers this way is misleading. 3 ANN BNN Size 10-1000 Neurons & 1000’s of synapses 86 Billion
Neurons & >100 Trillion synapses Topology Usually Feed-forward, that is computed Layer by Layer
Complex Network that is computed asynchronously Calculation Speeds nano seconds milli seconds
Power 100 watts (~100Watts) 20 watts Others Not Fault Tolerant, Learning Fault Tolerant, Learning?
3 Applications • Neural Nets have done exceptionally well at tasks like • Image Recognition,
Character Recognition, Face Recognition • Feature Extractions, Finger print processing, Signature
matching • Speech Recognition • Other modest successes in: Stock market predictions,
Combinatorial optimization, Medicine etc. 4 The Perceptron • Perceptron: The main building block 5
Activation function Activation function Bias The Artificial Neural Net 6 • Number of Layers • Single Vs
• Multi Layer • Number of Nodes in each Layer • Weights/connections • Activation or Transfer
function 6 Examples of Activation functions • ReLu (with Softmax/Linear) • Sigmoid (Logistic) •
Hyperbolic Tangent (tanh) • Step function (Heaviside) • Softmax (Generalized Logistic) • Linear •
Which one do we use? • There is no set procedure/Rule • ReLu has become very popular 7 An
Example 8 Learning • How to determine the weights? • Start with guess values for weights •
Calculate outputs from inputs • Compare outputs to desired outputs: Calculating errors • Training
algorithms update the weights in a way to minimize the errors (cost functions) • Cost (loss)
functions: measures how close an output is to the desired output. Preferably: • Non-negativity •
Globally continuous and differentiable 9 Training Also: Back Propagation • Back propagation of
errors, is a common algorithm to train artificial neural networks used in conjunction with an
optimization method such as gradient descent. • The method calculates the gradient of a loss
function with respect to all the weights in the network. • The gradient is fed to the optimization
method which in turn uses it to update the weights, in an attempt to minimize the loss function. 10
Gradient Descent • A first-order optimization algorithm. • Essentially equivalent to falling down on a
slope to eventually find the minimum (lowest point in the valley). • To find a minimum (valley) of a
function, take a small step along the steepest direction. And keep iterating. • For finding maximums
we’d do gradient ascent. 11 Learning rate • Choosing the Learning rate ( ) • Too small, we will need
too many iterations for convergence • Too large, we may skip the optimal solution • Adaptive
Learning Rate : start with high learning rate and gradually reduce the learning rate with each
iteration • Can also be chosen by Trial & Error – Use a range of learning rate (1, 0.1, 0.001, 0.0001)
and use the results as a guide 12 Epochs, Batch size, Iterations • When dataset is too large, passing
all the data through a Neural net before we make weight updates is computationally expensive •
Instead we would create data batches with smaller batch size. • After each batch is passed and
weights updated, we will count it as one iteration. • When an entire dataset is passed forward and
backward (weights updated) through the neural network, we will count it as one epoch. • Too few
epochs: under fitting, Too many: overfitting • Batch training: All of the training samples pass through
the neural net, before weights are updated • Sequential training: Weights are updated after each
training vector is passed through the neural net. 13 Scaling • Scaling the Variables • The non-
linearities in the activation function and the numerical rounding errors make input scaling quite
important • Scaling can accelerate learning and improve performance 14 Overfitting • Neural
Network Models are susceptible to overfitting • Large number of weights and biases • Excessive
learning (Epocs) on training data • Ways to avoid Overfitting • Increase sample size • Early stopping
• Reduce Network Size • Regularization 15 Regularization (weight decay) • Regularization is a
technique used to avoid this overfitting problem. • The idea behind regularization is that models that
overfit the data are complex models that have for example too many parameters. • Regularization
penalizes the usual loss function by adding a complexity term that would give a bigger loss for more
complex models. • Types of Regularization • LASSO • Ridge • Optimal value of λ, the decay rate or
penalty coefficient, is determined through cross-validation 16 Hyper parameters and tuning • Hyper
parameters are the variables which determines the network structure and the variables which
determine how the network is trained • Number of hidden layers • Number of neurons in hidden
layers • Decay factor • Number of Epoch • Learning Rate • The Activation Function • Tuning these to
ensure that you don’t overfit is an art. 17