Unit 1: Introduction to Neural Networks and Deep Learning

Unit 1: Introduction to Neural Networks and Deep Learning

1. Artificial Neural Networks (ANNs):

- Definition: Inspired by biological neural networks, ANNs are computing systems composed of

layers of interconnected nodes (neurons).

- Structure:

- Input Layer: Receives raw data.

- Hidden Layer(s): Performs transformations and identifies patterns.

- Output Layer: Produces the final predictions or outputs.

- Computation: Each neuron processes a weighted sum of inputs, adds a bias, and passes it

through an activation function.

Advantages:

1. Parallel Processing: Ability to perform multiple tasks simultaneously.

2. Fault Tolerance: Can still function even with partial damage to the network.

3. Memory Distribution: Data is stored across the network, not centrally.

4. Handles Incomplete Data: Can make predictions with missing information.

Disadvantages:

1. Uncertain Structure: No fixed rules for designing network architectures.

2. Lack of Transparency: Limited insight into the internal workings ("black box").

3. Hardware Dependence: Requires high computational power.

4. Time-consuming Training: Requires substantial time to train complex networks.

- Definition: A specialized branch of machine learning using neural networks with multiple hidden

layers to learn complex features and patterns.

- Characteristics:

- Automatically extracts features from raw data.

- Can handle large datasets with high-dimensional features.

- Applications:

- Image recognition, speech processing, natural language understanding, and more.

Architectures:

1. Deep Neural Networks (DNNs): Networks with multiple hidden layers for complex non-linear

relationships.

2. Deep Belief Networks (DBNs): Consist of stacked layers of Restricted Boltzmann Machines.

3. Recurrent Neural Networks (RNNs): Suitable for sequential data like time-series and text.

3. Components of Neural Networks:

1. Weights and Biases: Weights determine the strength of the connection. Bias shifts the activation

function's output.

2. Activation Functions:

- Linear: Simple linear relationships, not suitable for complex patterns.

- ReLU: Rectified Linear Unit, widely used in hidden layers.

- Sigmoid: Produces outputs between 0 and 1, used for binary classification.

- Tanh: Produces outputs between -1 and 1, often used in hidden layers.

- Softmax: Used for multi-class classification.

3. Error and Loss Functions: Measure the deviation between predicted and actual outcomes.

4. Learning Rate (alpha): Determines how quickly the network updates weights during training.
4. Learning Rules in ANNs:

1. Hebbian Learning Rule: Increases the connection strength between neurons that activate

together.

2. Delta Rule: Minimizes error through gradient descent.

3. Perceptron Rule: Adjusts weights for binary classifications.

4. Competitive Learning: Nodes compete to represent the input, and the winner is activated.

5. Out-Star Learning: Adjusts weights to match target outputs in layered networks.

5. Perceptron and Backpropagation:

1. Perceptron:

- A single-layer neural network for binary classification.

- Steps:

1. Calculate weighted sum.

2. Apply an activation function.

- Limitations: Can only classify linearly separable data.

2. Backpropagation:

- A method to train multi-layer networks using gradient descent.

- Two stages:

- Forward Pass: Compute outputs using current weights.

- Backward Pass: Adjust weights based on error gradients.

- Iterative process until the error converges to a minimum.

6. Applications of Neural Networks:

1. Facial Recognition: Using Convolutional Neural Networks (CNNs) for robust surveillance and

authentication.

2. Stock Market Prediction: Multilayer Perceptrons analyze historical stock data for forecasts.
3. Healthcare:

- CNNs for medical image analysis.

- Recurrent Neural Networks (RNNs) for voice and patient data recognition.

- Generative networks for drug discovery.

4. Weather Forecasting: Combines CNNs and RNNs to predict weather patterns.

5. Aerospace and Defense:

- Fault detection, autopilot systems, and underwater mine detection.

6. Social Media: Analyzing user behavior and preferences for targeted content.

7. Signature Verification: Ensuring document authenticity through ANN analysis.

7. Activation and Loss Functions:

- Activation Functions: Add non-linearity to model outputs and allow networks to learn complex

mappings.

- Loss Functions: Define objectives to minimize errors (e.g., MSE, Binary Cross-Entropy, and

Categorical Cross-Entropy).

8. Summary of Key Takeaways:

- Neural networks mimic the brain's functioning to solve problems.

- Deep learning extends traditional ANNs with additional layers and capabilities.

- Training involves defining architectures, updating weights, and minimizing errors.

- Applications are vast, spanning industries like healthcare, defense, and finance.