Soft Computing

Soft Computing
MCA20401 (Elective-III)

4th Semester MCA (AR-20)

1 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Lecture -1

Outline of the Lecture

 About the Course Syllabus, CEO, CO & Assessment Methods.

 Lesson Plan

 Course Overview & Objectives

 Prerequisites

 Introduction to Soft Computing (1st Topic)

2 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing About the Course Syllabus, CEO, CO & Assessment Methods.

SYLLABUS
UNIT:1 (12 Hours)
Introduction to Soft Computing, Fundamentals of ANN, Basic Model of an
Artificial Neuron, NN Architecture, Learning Methods, Terminology of
ANN, Hebb Network, ADALINE & MADALINE, Perceptron, MLP, Back
Propagation Network (BPN)- Architecture, Back Propagation, Learning
Effect of Tuning Parameters of the BPNN, Back Propagation- Algorithms.
Associative Memory: Auto-correlators & Hetero-correlators, Linear
Associative Memory, Applications, Adaptive Resonance Theory (ART),
ART1, ART2 & Applications

3 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing About the Course Syllabus, CEO, CO & Assessment Methods.

UNIT:2 (10 Hours)

Fuzzy Set Theory: Crisp Sets, Fuzzy Sets, Crisp Relations, Fuzzy Relations,
Fuzzy Systems, Crisp Logic, Predicate Logic, Fuzzy Logic (FL), Fuzzy Rule-
Based Systems (FRBS), Defuzzification Methods (Centroid & COS Method),
MOM Method
UNIT:3 (10 Hours)
Genetic Algorithm (GA): GA Fundamentals, Encoding Techniques, Fitness
functions, Reproduction, Genetic Modelling: Crossover, inversion, deletion,
Mutation Operators, Bitwise operators, Convergence of GA & Applications,
Real-life Problems
UNIT:4 (08 Hours)
Hybrid Systems: Introduction – Hybrid System, NN, Fuzzy Logic & Genetic
Algorithm Hybrids, GA-based BPN, GA-based Weight Determination
Applications, FL Controlled GA, Soft Computing Tools, Applications

4 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

Soft Computing About the Course Syllabus, CEO, CO & Assessment Methods.

Books
Text Books:
1) S. N. Sivanandam, S. N. Deepa “Principles of Soft Computing”. Wiley
India (P) Ltd.
2) S. Rajasekaran, G.A. Vijayalakshmi Pai – “Neural Networks, Fuzzy Logic
and Genetic Algorithms” - PHI Private Limited, New Delhi
Reference Books:
1) J. S. R. Jang. C. T. SUN and E. Mizutani, “Neuro-fuzzy and soft-
computing”. PHI Pvt. Ltd., New Delhi.
2) Fredric M. Ham and Ivica Kostanic, “Principle of Neuro Computing for
Science and Engineering”, Tata McGraw Hill.
3) S. Haykins, “Neural networks: a comprehensive foundation”. Pearson
Education, India.
4) V. Keeman, “Learning and Soft Computing”, Pearson Education, India.
5) R. C. Eberhart and Y. Shi, “Computational Intelligence Concepts to
Implementation”. Morgan Kaufmann Publishers (Indian Reprint).

5 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing About the Course Syllabus, CEO, CO & Assessment Methods.

Course Educational Objectives (CEO)

To familiarize with soft computing concepts.
To introduce the ideas of Neural networks, fuzzy logic and use of heuristics based
on human experience.
To introduce the concepts of Genetic algorithm and its applications to soft
computing using some applications.

Course Outcomes (CO)

Upon successful completion of this course, students should be able to:
CO1 Learn about soft computing techniques and their applications and Analyze
various neural network architectures
CO2 Apply fuzzy logic and reasoning to handle uncertainty and solve engineering
problems
CO3 Apply genetic algorithms to combinatorial optimization problems
CO4 Evaluate and compare solutions by various Hybrid soft computing
approaches for a given problem

6 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

Soft Computing About the Course Syllabus, CEO, CO & Assessment Methods.

Course Assessment Methods

1. Two Cycle Test

2. End Semester Examination

3. Online Quizzes, Assignments & Seminars

7 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

Soft Computing About the Course Syllabus, CEO, CO & Assessment Methods.

Cycle Test Pattern

Duration of Examination: 90 Minutes
Total Marks : 0 Marks

Question Paper Pattern of Cycle Test

2 Short Answer Questions: 5 Questions carrying 2 marks each

3 Long Answer Questions: 2 Questions, out of which 4 are to be
answered carrying 10 marks each (may
contain sub-questions).

8 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

Soft Computing About the Course Syllabus, CEO, CO & Assessment Methods.

End Semester Examination Pattern

Duration of Examination: 3 Hours
Total Marks : 70 Marks

Question Paper Pattern of Cycle Test

2 Short answer Questions: 10 questions carrying 2 marks each

3 Long Answer Questions: Answer any FIVE questions out of 7 Questions

9 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Lesson Plan
Name Satya Narayan Das Designation Associate Professor
Department CSA Semester & Section 4th Sem & Sec-A to F
Subject Name Soft Computing Subject Code MCA20401 (PE-III)
Pre requisite(s) Algorithms Academic Regulation AR-20
Total Hrs Planned 40 No. of Hrs/week 04 (L-3 & T-1)
Starting Date 02-02-2024 Total Credit 3
References
Sl. No.

Unit-I

Detail Description of Mode of

Hour

Date COs &POs Text Book

Topics/Subtopics Lecture
Page No.
1 1.1 Introduction: What is Soft Online [CO1][ PO1] TB1:11,
Computing? Brief descriptions
of different components of
soft computing, Difference
between Hard and soft
computing,
2 1.2 Requirement of Soft Online [CO1][ PO1] TB1:
computing, Major Areas of
Soft Computing, Applications
of Soft Computing.
10 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Lesson Plan

References
Sl. No.

Unit-I
Hour
Detail Description of Mode of
Date COs &POs Text Book
Topics/Subtopics Lecture
Page No.
3 1.3 Neural Networks: Basic Online [CO1][ PO1] TB1:11-13
Concept of Neural Networks
4 1.4 Models of an artificial Online [CO1][ PO1] TB1:13-16
Neuron, Various activation
functions
5 1.5 Neural Network Architecture Online [CO1][ PO1] TB1:16-19
& Characteristics, Different
Learning methods
6 1.6 Introduction to Early ANN Online [CO1][ PO1] TB1:19-20
architectures (basics only)
McCulloch & Pitts Model
7 1.7 Perceptron, ADALINE, Online [CO1][ PO1] TB1:22-24
MADALINE
8 1.8 Back Propagation Neural Online [CO1][ PO1] TB1:25-27
Network

11 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Lesson Plan

References
Sl. No.

Unit-I
Hour
Detail Description of Mode of
Date COs &POs Text Book
Topics/Subtopics Lecture
Page No.
9 1.9 Back propagation Learning Online [CO1][ PO1] TB1:28-30
Algorithm
10 1.10 Example of BPN , Effect of Online [CO1][ PO1] TB1:
Tuning parameters of the
BPN Network,
11 1.11 Associative Memory: Online [CO1][ PO1] TB1:
Autocorrelators,
Hetrocorrelators
12 1.12 Energy function for BAM, Online [CO1][ PO1] TB1:
Exponential BAM.

12 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Lesson Plan

References
Sl. No.

Unit-II
Detail Description of Mode of

Hour
Date COs &POs Text Book
Topics/Subtopics Lecture
Page No.
13 2.1 Fuzzy set Vs Crisp sets Online [CO2][ PO1] TB1:
14 2.2 Fuzzy Sets – Properties Online [CO2][ PO1] TB1:
15 2.3 Fuzzy Membership Online [CO2][ PO1] TB1:
Functions
16 2.4 Fuzzy Set Operations Online [CO2][ PO1] TB1:
17 2.5 Crisp and Fuzzy Relations Online [CO2][ PO1] TB1:
18 2.6 Fuzzy Relation Operations Online [CO2][ PO1] TB1:
19 2.7 Crisp logic, Propositional Online [CO2][ PO1] TB1:
logic, Predicate Logic
20 2.8 Fuzzy logic – Quantifier – Online [CO2][ PO1] TB1:
Inference
21 2.9 Fuzzy Rule based system Online [CO2][ PO1] TB1:
22 2.10 Defuzzification Methods Online [CO2][ PO1] TB1:

13 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Lesson Plan

References

Unit-III
Sl. No.

Detail Description of Mode of

Hour
Date COs &POs Text Book
Topics/Subtopics Lecture
Page No.
23 3.1 Fundamentals of genetic Online [CO1][ PO1] TB1:
algorithms
24 3.2 Encoding, Fitness functions Online [CO1][ PO1] TB1:
25 3.3 Reproduction Online [CO1][ PO1] TB1:
26 3.4 Genetic Modeling: Cross Online [CO1][ PO1] TB1:
over
27 3.5 Different Cross overs Online [CO1][ PO1] TB1:
28 3.6 Inversion and deletion Online [CO1][ PO1] TB1:
29 3.7 Mutation operator Online [CO1][ PO1] TB1:
30 3.8 Bit-wise operators & its Online [CO1][ PO1] TB1:
uses in GA.
31 3.9 Convergence of Genetic Online [CO1][ PO1] TB1:
algorithm
32 3.10 Applications, Real life Online [CO1][ PO1] TB1:
Problems.

14 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Lesson Plan

References

Unit-IV
Sl. No.

Detail Description of Mode of

Hour
Date COs &POs Text Book
Topics/Subtopics Lecture
Page No.
33 4.1 Hybrid system Online [CO1][ PO1] TB1:
34 4.2 Neural Networks Hybrid Online [CO1][ PO1] TB1:
system
35 4.3 Fuzzy Hybrid system Online [CO1][ PO1] TB1:
36 4.4 Genetic Algorithm Hybrid Online [CO1][ PO1] TB1:
system
37 4.5 Genetic Algorithm based Online [CO1][ PO1] TB1:
Back propagation Networks
38 4.6 GA based weight Online [CO1][ PO1] TB1:
determination
39 4.7 Fuzzy Back Propagation Online [CO1][ PO1] TB1:
Networks
40 4.8 Fuzzy logic controller Online [CO1][ PO1] TB1:

15 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Course Overview & Objectives

This course will cover fundamental concepts used in Soft

computing. The Artificial Neural Networks (ANNs) will be
covered first followed by concepts of Fuzzy logic (FL) and
optimization techniques using Genetic Algorithm (GA).
Applications of different Hybrid Soft Computing techniques to
solve a number of real life problems will be covered to have hands
on practices. In summary, this course will provide exposure to
different soft computing techniques
After completing this course, you will be able to learn:
• Artificial Neural Networks and its Applications.
• Fuzzy Logic and its applications.
• Solving single-objective optimization problems using GAs.
• Solving multi-objective optimization problems using
Evolutionary algorithms (MOEAs).
• Applications of Soft computing to solve problems in
varieties of application domains.
16 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Prerequisites

To extract the maximum from the course, the following

prerequisites are required.

 A strong mathematical background.

 Proficiency with algorithms.

 Programming skills in C, C++, Python or Java.

 Critical thinking and problem solving skills.

17 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Introduction to Soft Computing

 Concept of Computation
 Soft Computing Techniques
 Hard computing
 Soft computing
 How soft computing?
 Hard computing vs. Soft computing

18 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing CONCEPT OF COMPUTATION

Computing
Antecedent Consequent
y = f(x)

Control Action
Figure: Basic of computing
Antecedent: A thing that existed before or logically precedes another.
y = f(x), f is a mapping function, f is also called formal methods
or an Algorithm to solve the problem
Control Action should an unambiguous and accurate.
Consequent: Following as a result or effect.
It should provide precise solution.
It is suitable for problem, which is easy to model mathematically.
19 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Soft Computing Techniques

Neural
Neuro Fuzzy Network Neuro Genetic

Neuro Fuzzy
Fuzzy Genetic Genetic
Logic Algorithm

Fuzzy Genetic

20 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Soft Computing Techniques

Soft Computing refers to a consortium of computational

methodologies. Some of its principal components include Neural
Networks (NN), Fuzzy Logic (FL) and Genetic Algorithms (GA), all
having their roots in Artificial Intelligence (Al).

Also, a combination of one or more of the methodologies mentioned

termed hybrid systems has resulted in the emergence of a new class
of systems such as Neuro-Fuzzy, Fuzzy-Genetic, Neuro-Genetic and
Neuro-Fuzzy-Genetic systems.

“Artificial Intelligence (AI) is an area of computer science

concerned with designing intelligent computer systems” that is,
systems that exhibit the characteristics we associate with intelligence
in human behavior. “Al is a branch of computer science that is
concerned with the automation of intelligent behavior”.

21 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing

22 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Hard Computing

• In 1996, L. A. Zade (LAZ) introduced the term Hard

Computing.
• According to LAZ: We term a computing as Hard
computing, if
 Precise result is guaranteed.
 Control action is Unambiguous.
 Control action is formally defined (i.e., with
mathematical model or algorithm).

23 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Examples of Hard Computing

• Solving numerical problems (e.g., roots of polynomials,

integration, etc.).

• Searching and sorting techniques.

• Solving computational geometry problems (e.g., shortest tour

in a graph, finding closet pair of points given a set of points,
etc.) many more…

24 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Soft Computing
The term soft computing was proposed by the inventor of fuzzy
logic, Lotfi A. Zadeh. He describes it as follows.
Soft computing is a collection of methodologies that aim to exploit
the tolerance for imprecision and uncertainty to achieve tractability,
robustness, and low solution cost. Its principal constituents are fuzzy
logic, neuro-computing, and probabilistic reasoning. The role model
for soft computing is the human mind.

• It does not require any mathematical modeling of problem

solving.
• It may not yield the precise solution.
• Algorithms are adaptive (i.e., it can adjust to the change of
dynamic environment).
• Use some biological inspired methodologies such as genetics,
evolution, Ant’s behaviors, particles swarming, human nervous
system, etc.).
25 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing

Soft computing

Example: Hand written character recognition

(Artificial Neural Networks)

26 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing

Example: Robot movement

(Fuzzy Logic)
27 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing

Bank with
Soft computing maximum
return

Example: Money allocation problem

(Evolutionary Computing)

28 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing How soft computing?

 How a student learns from his/her teacher?

• Teacher asks questions and tell the answers then.
• Teacher puts questions and hints answers and asks
whether the answers are correct or not.
• Student thus learn a topic and store in his memory.
• Based on the knowledge he solves new problems.

 This is the way how human brain works.

 Based on this concept Artificial Neural Network is used to

solve the problems.

29 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing How soft computing?

 How a doctor treats his patient?

• Doctor asks the patient about suffering.
• Doctor find the symptoms of diseases.
• Doctor prescribed tests and medicines.

 This is exactly the way Fuzzy Logic works.

• Symptoms are correlated with diseases with uncertainty .
• Doctor prescribes tests/medicines fuzzily.

30 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing How soft computing?

 How world selects the best?

• It starts with a population (random).
• Reproduces another population (next generation).
• Rank the population and selects the superior
individuals.

 Genetic algorithm is based on this natural phenomena.

• Population is synonymous to solutions.
• Selection of superior solution is synonymous to
exploring the optimal solution.

31 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Hard computing vs. Soft computing

Hard computing Soft computing

 It requires a precisely stated  It is tolerant of imprecision,
analytical model and often a uncertainty, partial truth, and
lot of computation time. approximation.
 It is based on binary logic,  It is based on fuzzy logic,
crisp systems, numerical neural nets and probabilistic
analysis and crisp software. reasoning.
 It has the characteristics of  It has the characteristics of
precision and categoricity. approximation &
dispositionality.
 It is deterministic.  It incorporates stochasticity.

 It is strictly sequential.  It allows parallel

computations.
32 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Hard computing vs. Soft computing

Hard computing Soft computing

 It requires exact input data.  It can deal with ambiguous
and noisy data.
 It produces precise answers.  It can yield approximate
answers
 It requires programs to be  It can evolve its own
written programs

33 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Question Answers

1. Core components of soft Computing are

a) Neural Network, Fuzzy Logic and Genetic Algorithms

b) Fuzzy Networks and Artificial Intelligence
c) Artificial Intelligence and Neural Science
d) Neural Science and Genetic Science

Answer: a

34 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Question Answers

2. Who initiated the idea of Soft Computing

a) Charles Darwin
b) Lofti A Zadeh
c) Rechenberg
d) Mc_Culloch

Answer: b

35 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Question Answers

3. Hand written character recognition can use which

technique?

a) Neuron Network
b) Fuzzy Logic
c) Genetic Algorithm
d) Evolutionary Computing

Answer: a

36 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Topic for the Next Class

Brief descriptions of different components of soft

computing.
 Neural Network
Fuzzy Logic
Genetic Algorithm (Evaluation Computing)

37 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Lecture -2

Outline of the Lecture

 Quick overview of previous Class

 Brief descriptions of different components of soft computing.

 Neural Network

 Fuzzy Logic

 Genetic Algorithm (Evaluation Computing)

38 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Quick overview of previous Class

 Concept of Computation
 Soft Computing Techniques
 Hard computing
 Soft computing
 How soft computing?
 Hard computing vs. Soft computing

39 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Brief descriptions of different components of soft computing.

Neural
Neuro Fuzzy Network Neuro Genetic

Neuro Fuzzy
Fuzzy Genetic Genetic
Logic Algorithm

Fuzzy Genetic

Different components of soft computing.

40 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Neural Network

 As we earlier discussed about computing methods such as

hard computing and soft computing by highlighting
advantages of both the techniques along with their detailed
classification.
 It justifies the needs of soft computing methods for
designing an intelligent systems.
 Soft Computing refers to a consortium of computational
methodologies. Some of its principal components include
Neural Networks (NN), Fuzzy Logic (FL) and Genetic
Algorithms (GA), all having their roots in Artificial
Intelligence (Al).
 Also, a combination of one or more of the methodologies
mentioned termed hybrid systems has resulted in the
emergence of a new class of systems such as Neuro-Fuzzy,
Fuzzy-Genetic, Neuro-Genetic and Neuro-Fuzzy-Genetic
hybrid systems.
41 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Neural Network

 “Artificial Intelligence (AI) is an area of computer science

concerned with designing intelligent computer systems”
that is, systems that exhibit the characteristics we associate
with intelligence in human behavior.
 “Al is a branch of computer science that is concerned with
the automation of intelligent behavior”.
 Neural network being a simplified model of biological
neuron system is a massively parallel distributed
processing system made up of highly interconnected neural
computing elements that have an ability to learn and thereby
acquire knowledge and make it available for use.

42 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Neural Network

Human Brain consists of approximately 1011 no of Neurons

43 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Biological Neuron

44 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Human Brain

45 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Biological Neuron & Artificial Neuron

46 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Artificial Neuron

Bias
Inputs Weights
b
x1 w1
x2 w2
y
n Output

x3 w3 x w
i 1
i i
....

Activation
....

Sumation Function
xn wn Unit

An Artificial Neuron

47 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Introduction to Fuzzy Logic

 The term fuzzy refers to things which are not clear or are
vague. In the real world many times we encounter a situation
when we can’t determine whether the state is true or false,
their fuzzy logic provides a very valuable flexibility for
reasoning. In this way, we can consider the inaccuracies and
uncertainties of any situation.
 In Boolean system truth value, 1.0 represents absolute truth
value and 0.0 represents absolute false value. But in the fuzzy
system, there is no logic for absolute truth and absolute false
value. But in fuzzy logic, there is intermediate value too
present which is partially true and partially false.
 Fuzzy logic is a form of many-valued logic in which the truth
values of variables may be any real number between 0 and 1
both inclusive.

48 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Boolean Logic Vs Fuzzy Logic

Yes / 1

Is it Cold? Boolean Logic

No / 0

Very Cold / 1.0

Is it Cold? Not Quite Cold / 0.5 Fuzzy Logic

Not Cold / 0.0

49 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Genetic Algorithm

 Genetic algorithms are randomized search algorithms that have

been developed in an effort to imitate the mechanics of natural
selection and natural genetics. Genetic algorithms operate on
string structures, like biological structures, which are evolving in
time according to the rule of survival of the fittest by using a
randomized yet structured information exchange. Thus, in every
generation, a new set of strings is created, using parts of the fittest
members of the old set. The main characteristics of a genetic
algorithm are as follows:
 The genetic algorithm works with a coding of the parameter set,
not the parameters themselves.
 The genetic algorithm initiates its search from a population of
points, not a single point.
 The genetic algorithm uses payoff information, not derivatives.
 The genetic algorithm uses probabilistic transition rules, not
deterministic ones.
50 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Genetic Algorithm

GA is based on the concept

of ‘Survival of the Fittest’.
GA represents the solutions
in the form of chromosomes
and the fitness of the
chromosomes is evaluated.
The more fit solutions are
selected for the reproduction
using the crossover operator.
The mutation operator is
used to maintain the
diversity the population.

51 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Different components of soft computing Overview

Artificial Neural Networks: Fuzzy logic System: Genetic Algorithm:

Neural Networks in Soft Fuzzy Logic in Soft Genetic Algorithm in Soft
Computing – Human brains in Computing – Fuzzy Computing – Nature is and
a way describe the real world logic is a will always be an amazing
conditions, which computers mathematical logic, source of inspiration for all
cannot. In order to solve this which attempts to of mankind. Genetic
issue, for the first time, neural solve problems with algorithms (GA) take all
networks were developed in an open, imprecise their inspiration from
the 1950s. An artificial neural spectrum of data that nature, and there are no less
network is an attempt to makes it possible to genetic algorithms based on
emulate a network of neurons get an array of precise search-based algorithms
that make a human brain so findings. Fuzzy logic that find its roots in natural
that computers can be able to is designed to be selection and concepts of
learn things and make considered the best genetics. The genetic
decisions in a human way. possible decision by algorithm is also a subset of
ANN is made by regular considering all a large branch of
computer programming as if available information computation (also called
they are mutually associated and looking an input. evolutionary computation).
with brain cells.

52 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Major Area of Soft Computing

Some stunning Application areas of Soft Computing are:

 Actuarial Science
 Agricultural Production Engineering
 Medicine and Biology Application
 Construction and Design Engineering
 Computer Engineering
 Sin Forecasting
 Computational Process
 Natural Environmental Engineering
 Fault-Tolerance
 Machine Learning
 Signal processing
 Mechanical engineering
 Materials Engineering
 Disease diagnosis
 Nano Technology
 Pattern Recognition and Many more………

53 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Application of Soft Computing

There are various Applications of Soft Computing are:

 Consumer appliance like AC, Refrigerator, Heaters, Washing
machine.
 Robotic works in the form of Emotional Pet robots.
 Food preparation devices are Microwave and Rice cookers.
 For amusing gaming playing product like Checker and Poker
etc.
 Recognition for Handwriting.
 Data compression / Image Processing
 For Different Architecture
 Decision-support System
 Applications in Agricultural and Biological Engineering
 Natural Language Processing
 Data Mining / Sentiment Mining
 And so on…………..

54 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Advantages of Soft Computing

These are many Advantages of Soft Computing are:

 Work as human being reasoning

 Nearest human thinking
 Biological inspiration
 Tolerance to imprecision
 Can be captured uncertainty and vagueness values
 Perceive Linguistic Variables
 Work in equations and conditions

55 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Question Answers

1. Why do we need biological neural networks?

a) To solve tasks like machine vision & natural language

processing
b) To apply heuristic search methods to find solutions of problem
c) To make smart human interactive & user friendly system
d) All of the mentioned

Answer: d

Explanation: These are the basic aims that a neural network

achieves.

56 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Question Answers

2. What is the trend in software nowadays?

a) To bring computer more & more closer to user

b) To solve complex problems
c) To be task specific
d) To be versatile

Answer: a

Explanation: Software should be more interactive to the user, so

that it can understand its problem in a better way.

57 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Question Answers

3. What’s the main point of difference between human &

machine intelligence?

a) Human perceive everything as a pattern while machine

perceive it merely as data
b) Human have emotions
c) Human have more IQ & intellect
d) Human have sense organs

Answer: a

Explanation: Humans have emotions & thus form different patterns

on that basis, while a machine (say computer) is
dumb & everything is just a data for him.

58 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Topic for the Next Class

Details about the neural Network

Model of an ANN
Various Activation Function

59 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Lecture -3

Outline of the Lecture

 Quick overview of previous Class

 Basic Concept of Neural Networks

 Neural Network Architecture

 ANN Model

 Various activation functions

60 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Quick overview of previous Class

 Soft Computing Techniques

 Brief descriptions of different components of soft
computing.
 Neural Network
 Fuzzy Logic
 Genetic Algorithm (Evaluation Computing)

61 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Brief descriptions of different components of soft computing.

Neural
Neuro Fuzzy Network Neuro Genetic

Neuro Fuzzy
Fuzzy Genetic Genetic
Logic Algorithm

Fuzzy Genetic

Different components of soft computing.

62 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Outlines of lecture - 3

 Basic concept of Neural Network

 Biological Neuron & Artificial Neuron
 Basic Model of an Artificial Neuron
 Activation Function and
 Different Activation Function

63 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Basic Concept of Neural Networks

 Neural networks are simplified models of the biological nervous system

and therefore have drawn their motivation from the kind of computing
performed by a human brain.
 An NN, in general, is a highly interconnected network of a large
number of processing elements called neurons in an architecture
inspired by the brain. An NN can be massively parallel and therefore is
said to exhibit parallel distributed processing.
 Neural networks exhibit characteristics such as mapping capabilities or
pattern association, generalization, robustness, fault tolerance, and
parallel and high speed information processing.
 Neural networks learn by examples. They can therefore be trained with
known examples of a problem to acquire knowledge about it. Once
appropriately trained, the network can be put to effective use in solving
‘unknown’ or ‘untrained’ instances of the problem.
 Neural networks have been successfully applied to problems in the
fields of pattern recognition, image processing, data compression,
forecasting, and optimization to quote a few.

64 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Basic Concept of Neural Networks

 Neural Networks, which are simplified models of the biological neuron

system, is a massively parallel distributed processing system made up of
highly interconnected neural computing elements that have the ability to
learn and thereby acquire knowledge and make it available for use.
 Various learning mechanisms exist to enable the NN acquire
knowledge. NN architectures have been classified into various types
based on their learning mechanisms and other features. Some classes of
NN refer to this learning, process as training and the ability to solve a
problem using the knowledge acquired as inference.
 NNs are simplified imitations of the central nervous system, and
obviously therefore, have been motivated by the kind of computing
performed by the human brain. The structural constituents of a human
brain termed neurons are the entities, which perform computations such
as cognition, logical inference, pattern recognition and so on.

65 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Human Brain

The human brain is one of the most complicated things which, on the
whole, has been poorly understood. However, the concept of neurons
as the fundamental constituent of the brain, attributed to Ramon Y.
Cajal (1911), has made the study of its functioning comparatively
easier. Figure illustrates the physical structure of the human brain.

66 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Biological Neuron

67 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Basic Model of an Artificial Neuron

 As mentioned earlier, the human brain no doubt is a highly

complex structure, viewed as a massive, highly interconnected
network of simple processing elements called neurons. However,
the behavior of a neuron can be captured by a simple model as
shown in previous Figure. Every component of the model bears a
direct analogy to the actual constituents of a biological neuron and
hence is termed as artificial neuron. It is this model which forms
the basis of Artificial Neural Networks.
 Here, x1, x2, x3………xn are the n inputs to the artificial neuron. w1,
w2……..wn, are the weights attached to the input links. Recollect
that a biological neuron receives all inputs through the dendrites,
sums them and produces an output if the sum is greater than a
threshold value. The input signals are passed on to the cell body
through the synapse which may accelerate or retard an arriving
signal.

68 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Model of an Artificial Neuron

Bias
Inputs Weights
b
x1 w1
x2 w2
y
n Output

x3 w3 x w
i 1
i i
....

Activation
....

Sumation Function
xn wn Unit

An Artificial Neuron

69 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Model of an Artificial Neuron

It is this acceleration or retardation of the input signals that is

modeled by the weights. An effective synapse which transmits a
stronger signal will have a correspondingly larger weight while a
weak synapse will have smaller weights. Thus, weights here are
multiplicative factors of the inputs to account for the strength of the
synapse. Hence, the total input I received by the soma of the
artificial neuron is
I = w1x1 + w2x2 + ….. + wnxn
n
= w x
i 1
i i

To generate the final output y, the sum is passed on to a non-linear

filter  called Activation function, or Transfer function, or Squash
function which releases the output.
y =  (I)

70 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Model of an Artificial Neuron
A very commonly used Activation function is the Threshold
function. In this, the sum is compared with a threshold value . If the
value of I is greater than , then the output is 1 else it is 0.
 n 
y =    wi xi   
 i1 
Where  is the step function known as Heaviside function and is
such that 1, I > 0
 (I) =
0, I ≤ 0

This is convenient in the sense that the

output signal is either 1 or 0 resulting
in the neuron being on or off.

71 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Activation Function

 An Activation Function decides whether a neuron should be

activated or not. This means that it will decide whether the
neuron’s input to the network is important or not in the process of
prediction using simpler mathematical operations.
 The role of the Activation Function is to derive output from a set
of input values fed to a node (or a layer).
 The activation function is a mathematical “gate” in between the
input feeding the current neuron and its output going to the next
layer. It can be as simple as a step function that turns the neuron
output on and off, depending on a rule or threshold. Or it can be a
transformation that maps the input signals into output signals that
are needed for the neural network to function.
 Increasingly, neural networks use non-linear activation functions,
which can help the network learn complex data, compute and
learn almost any function representing a question, and provide
accurate predictions.
72 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Activation Function

Depending on the nature and intensity of these input signals,

the brain processes them and decides whether the neuron
should be activated (“fired”) or not.
The purpose of an activation function is to add non-linearity
to the neural network.
73 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Classification of Activation Function

The Activation Functions can be divided into 3 types-

 Binary Step Function
 Linear or Identity Activation Function
 Non-linear Activation Functions
1. Binary Step Function
The binary step function depends on a
threshold value that decides whether a
neuron should be activated or not.
The input fed to the activation function
is compared to a certain threshold; if the
input is greater than it, then the neuron
is activated, else it is deactivated,
meaning that its output is not passed on
to the next hidden layer.

74 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Linear Activation Function

2. Linear or Identity Activation Function

It takes the inputs, multiplied by the weights for each neuron, and
creates an output signal proportional to the input.
Equation: f(x) = x
Derivative: f’(x) = 1
Range: (-∞, +∞)

75 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Linear Activation Function

Two major problems with Linear Activation Function:

1) Back-propagation is not possible — The derivative of the

function is a constant, and has no relation to the input, X. So it’s
not possible to go back and understand which weights in the
input neurons can provide a better prediction.

2) All layers of the neural network collapse into one — with

linear activation functions, no matter how many layers in the
neural network, the last layer will be a linear function of the first
layer

76 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Non-Linear Activation Function

3. Non-Linear Activation Functions

The linear activation function shown above is simply a linear
regression model.
Because of its limited power, this does not allow the model to create
complex mappings between the network’s inputs and outputs.
Non-linear activation functions solve the following limitations of
linear activation functions:
 They allow backpropagation because now the derivative function
would be related to the input, and it’s possible to go back and
understand which weights in the input neurons can provide a
better prediction.
 They allow the stacking of multiple layers of neurons as the
output would now be a non-linear combination of input passed
through multiple layers. Any output can be represented as a
functional computation in a neural network.

77 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Non-Linear Activation Function

 Modern neural network models use non-linear activation

functions. They allow the model to create complex mappings
between the network’s inputs and outputs, which are essential for
learning and modeling complex data, such as images, video,
audio, and data sets which are non-linear or have high
dimensionality.
 Almost any process imaginable can be represented as a functional
computation in a neural network, provided that the activation
function is non-linear.
 Non-linear functions address the problems of a linear activation
function:
 They allow back-propagation because they have a derivative
function which is related to the inputs.
 They allow “stacking” of multiple layers of neurons to create a
deep neural network. Multiple hidden layers of neurons are
needed to learn complex data sets with high levels of accuracy.
78 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Types of Non-Linear Activation Function

Now, let’s have a look at ten different non-linear neural networks

activation functions and their characteristics.
1. Sigmoid / Logistic Activation Function
2. Tanh Function (Hyperbolic Tangent)
3. Rectified Linear Unit (ReLU) Function
4. Leaky ReLU Function
5. Parametric ReLU Function
6. Exponential Linear Units (ELUs) Function
7. Softmax Function
8. Swish
9. Gaussian Error Linear Unit (GELU) Function
10. Scaled Exponential Linear Unit (SELU) Function

79 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Non-Linear Activation Function

1. Sigmoid / Logistic Activation Function

This function takes any real value as input and outputs values in the
range of 0 to 1.
The larger the input (more positive), the closer the output value will
be to 1.0, whereas the smaller the input (more negative), the closer
the output will be to 0.0, as shown below.

Mathematically it can
be represented as:

80 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Non-Linear Activation Function

Some commonly used non-linear activation functions are —

1) Sigmoid / Logistic

Sigmoid function gives an ‘S’ shaped curve. In order to map

predicted values to probabilities, we use the sigmoid function. The
function maps any real value into another value between 0 and 1.

• Equation: f(x) = s= 1/(1+e⁻ˣ)

• Derivative: f’(x) = s*(1-s)
• Range: (0,1)

81 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Non-Linear Activation Function

Why the sigmoid/logistic activation function is one of the most

widely used functions?

It is commonly used for models where we have to predict the

probability as an output. Since the probability of anything exists only
between the range of 0 and 1, sigmoid is the right choice because of
its range.
The function is differentiable and provides a smooth gradient, i.e.,
preventing jumps in output values. This is represented by an S-shape
of the sigmoid activation function.
The limitations of the sigmoid function are discussed below:

The derivative of the function is f'(x) = sigmoid(x)*(1-sigmoid(x)).

82 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing
Signum function

Also known as the Quantizer function, the function  is defined as

+1, I > 
 (I) =
-1, I ≤ 

Illustrates the Signum function.

83 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing

Sigmoidal function
This function is a continuous function that varies gradually between
the asymptotic values 0 and 1 or -1 and +1 and is given by
1
 (I) = 1  e I

where,  is the slope

parameter, which adjusts the
abruptness of the function as
it changes between the two
asymptotic values. Sigmoidal
functions are differentiable,
which is an important feature
of NN theory. Figure
illustrates the sigmoidal
function.
84 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Sigmoid / Logistic Activation Function

Advantages:
 The function is differentiable. That means, we can find the slope
of the sigmoid curve at any two points.
 Output values bound between 0 and 1, normalizing the output of
each neuron.
Disadvantages:
 Vanishing gradient — for very high or very low values of X, there
is almost no change to the prediction, causing a vanishing gradient
problem.
 Due to vanishing gradient problem, sigmoid have slow
convergence.
 Outputs not zero centered.
 Computationally expensive.

85 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Hyperbolic tangent Activation Function

2) Tan-h / Hyperbolic tangent

• Equation : f(x) = a =tanh(x) =(eˣ - e⁻ˣ)/(eˣ +e⁻ˣ)
• Derivative: (1- a²)
• Range: (-1, 1)
Advantages:
1. Zero centered — making it easier
to model inputs that have strongly
negative, neutral, and strongly positive
values.
2. The function and its derivative both
are monotonic.
3. Works better than sigmoid function
Disadvantage:
1. It also suffers vanishing gradient
problem and hence slow
convergence.

86 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing ReLU Activation Function

3) ReLU (Rectified Linear Unit)

•Equation: f(x) = a =max(0,x)
•Derivative: f’(x) = { 1 ; if z>0, 0; if z<0 and undefined if z=0 }
•Range: (0, +∞)
Advantages:
1.Computationally efficient — allows
the network to converge very quickly
2.Non-linear — although it looks like a
linear function, ReLU has a derivative
function and allows for back-
propagation
Disadvantages:
1.The Dying ReLU problem — when
inputs approach zero, or are negative,
the gradient of the function becomes
zero, the network cannot perform back-
propagation and cannot learn.
87 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Leaky ReLU Activation Function

4) Leaky ReLU

•Equation: f(x)= a = max(0.01x, x)

•Derivative: f’(x) = {0.01 ; if z<0, 1 ; otherwise}
•Range: (0.01, +∞)

88 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Leaky ReLU Activation Function

Advantage:
1. Prevents dying ReLU problem — this variation of ReLU has a
small positive slope in the negative area, so it does enable back-
propagation, even for negative input values
Disadvantage:
1. Results not consistent — leaky ReLU does not provide
consistent predictions for negative input values.
2. During the front propagation if the learning rate is set very high it
will overshoot killing the neuron.
The idea of leaky ReLU can be extended even further. Instead of
multiplying x with a constant term we can multiply it with a hyper-
parameter which seems to work better the leaky ReLU. This
extension to leaky ReLU is known as Parametric ReLU.

89 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Softmax Activation Function
5) Softmax
Softmax function calculates the probabilities distribution of the
event over ‘n’ different events.
• Equation: f(x) = eˣᵢ / (Σⱼ₌₀ eˣᵢ)
• Probabilistic interpretation: Sⱼ = P(y=j|x)
• Range: (0, 1)
Advantages:
1. Able to handle multiple classes only one class in other activation
functions — normalizes the outputs for each class between 0 and
1, and divides by their sum, giving the probability of the input
value being in a specific class.
2. Useful for output neurons — typically Softmax is used only for
the output layer, for neural networks that need to classify inputs
into multiple categories.

90 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Swish Activation Function

6) Swish
Swish is a new, self-gated
activation function
discovered by researchers at
Google. It performs better
than ReLU with a similar
level of computational
efficiency. In experiments on
ImageNet with identical
models running ReLU and
Swish, the new function
achieved top -1 classification
accuracy 0.6-0.9% higher.

91 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Topic for the Next Class

 Neural Network Architecture

 Characteristics of Neural Network
 Various Learning Methods

92 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Lecture -4

Outline of the Lecture

 Quick overview of previous Class

 Neural Network Architecture

 Characteristics of Neural Network

 Various Learning Methods

93 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Quick overview of previous Class (Lecture – 3)

 Basic concept of Neural Network

 Biological Neuron & Artificial Neuron
 Basic Model of an Artificial Neuron
 Activation Function and
 Different Activation Function

94 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Outlines of lecture - 4

 Neural Network Architecture

 Characteristics of Neural Network

 Various Learning Methods

95 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Neural Networks Architecture

An Artificial Neural Network is defined as a data processing system

consisting of a large number of simple highly interconnected
processing elements (artificial neurons) in an architecture inspired by
the structure of the cerebral cortex of the brain.
Generally, an ANN structure can be represented using a directed
graph.
A graph G is an ordered 2-tuple (V, E) consisting of a set V of
vertices and a set E of edges.
When each edge is assigned an orientation, the graph is directed and
is called a directed graph or a digraph.
Digraphs assume significance in Neural Network theory since
signals in NN systems are restricted to flow in specific directions.
The vertices of the graph may represent neurons (input/output) and
the edges, the synaptic links. The edges are labelled by the weights
attached to the synaptic links.

96 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Neural Networks Architecture

Vertices V = {v1, v2, v3, v4, v5}

Edges E = {e1, e2, e3, e4, e5}
Figure – 1 (Digraph)

There are several classes of NN, classified according to their

learning mechanisms. However, we identify three fundamentally
different classes of Networks. All the three classes employ the
digraph structure for their representation.

97 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Single Layer Feedforward Neural Network

 This type of network comprises of two layers, namely the input

layer and the output layer.
 The input layer neurons receive the input signals and the
output layer neurons receive the output signals.
 The synaptic links carrying the weights connect every input
neuron to the output neuron but not vice-versa. Such a network
is said to be feedforward in type or acyclic in nature.
 Despite the two layers, the network is termed single layer since
it is the output layer, alone which performs computation.
 The input layer merely transmits the signals to the output layer.
 Hence, the name single layer feedforward network.
98 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Single Layer Feedforward Neural Network

xi : Input neurons
yj : Output neurons
wij : Weights

Figure – 2
Single Layer Feedforward Network.

99 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Multi Layer Feedforward Neural Network

 This network is made up of multiple layers. Thus, architectures

of this class besides possessing an input and an output layer also
have one or more intermediary layers called hidden layers.
 The computational units of the hidden layer are known as the
hidden neurons or hidden units. The hidden layer aids in
performing useful intermediary computations before directing
the input to the output layer.
 The input layer neurons are linked to the hidden layer neurons
and the weights on these links are referred to as input hidden
layer weights. Again, the hidden layer neurons are linked to the
output layer neurons and the corresponding weights are referred
to as hidden output layer weights.
 A multilayer feedforward network with l input neurons, m1,
neurons in the first hidden layer, m2, neurons in the second
hidden layer and n output neurons in the output layer is written
as (l – m1 – m2 – n).
100 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Multi Layer Feedforward Neural Network

xi Input neurons
yj Hidden neurons
zk Output neurons
vij Input hidden layer weights
wjk Output hidden layer weights

Figure – 3:
A multilayer feedforward network ( l - m - n configuration).

101 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Recurrent Neural Network

 These networks differ from feedforward network architectures in

the sense that there is at least one feedback loop.
 Thus, in these networks, for example, there Could exist one layer
with feedback connections as shown in Figure – 4.
 There could also be neurons with self-feedback links, i.e. the
output of a neuron is fed back into itself as input

Figure – 4
Recurrent Neural Network

102 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Characteristics of Neural Network

i. The NNs exhibit mapping, capabilities, that is, they can map
input patterns to their associated output patterns.
ii. The NNs learn by examples. Thus, NN architectures can be
‘trained’ with known examples of a problem before they are
tested for their ‘Inference’ capability on unknown instances of
the problem. They can, therefore, identify new objects
previously untrained.
iii. The NNs possess the capability to generalize. Thus, they can
predict new outcomes from past trends.
iv. The NNs are robust systems and are fault tolerant. They can,
therefore, recall full patterns from incomplete, partial or noisy
patterns.
v. The NNs can process information in parallel, at high speed,
and in a distributed manner.

103 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Learning Methods

Learning methods in Neural Networks can be broadly classified into

three basic types:
 Supervised Learning
 Unsupervised Learning
 Reinforced Learning.

104 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Supervised Learning Method

 As the name suggests, supervised learning takes place under

the supervision of a teacher.
 This learning process is dependent.
 During the training of ANN under supervised learning, the
input vector is presented to the network, which will produce an
output vector.
 This output vector is compared with the desired/target output
vector. An error signal is generated if there is a difference
between the actual output and the desired/target output vector.
 On the basis of this error signal, the weights would be adjusted
until the actual output is matched with the desired output.
 There are two main types of supervised learning problems:
they are classification that involves predicting a class label
and regression that involves predicting a numerical value.

105 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Unsupervised Learning Method

 As the name suggests, this type of learning is done without the

supervision of a teacher.
 This learning process is independent.
 In this learning method, the target output is not presented to the
network hence the system learns of its own by discovering and
adapting to structural features in the input patterns.
 During the training of ANN under unsupervised learning, the
input vectors of similar type are combined to form clusters.
 When a new input pattern is applied, then the neural network
gives an output response indicating the class to which input
pattern belongs.
 There are many types of unsupervised learning, although there
are two main problems: they are clustering that involves
finding groups in the data and density estimation that involves
summarizing the distribution of data.

106 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Supervised & Unsupervised Learning Method

107 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Reinforced Learning Methods

 In this method, a teacher though available, does not tell the

expected answer but only indicates if the computed output is
correct or incorrect.
 The information provided helps the network in its learning
process. A reward is given for a correct answer computed and
a penalty for wrong answer.
 But, reinforced learning is not one of the popular forms of
learning.
 Supervised and unsupervised learning methods, which are
most popular forms of learning, have found expression
through various rules.

108 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Classification of Learning Method

Neural Network
Learning Algorithm

Supervised Learning Unsupervised Reinforced Learning

(Error Based) Learning (Output Based)

Error Correction Stochastic Competitive

Hebbian
Gradient

Least Mean Backpropagation

Square

109 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Gradient Descent Learning Method

Gradient descent learning

This is based on the minimization of error E defined in terms of
weights and the activation function of the network. Also, it is
required that the activation function employed by the network is
differentiable, as the weight update is dependent on the gradient of
the error E.
Thus, if Wij is the weight update of the link connecting the ith and
jth neuron of the two neighboring layers, then Wij is defined as
E
 Wij  
Wij
Where η is the learning rate parameter and ∂E/∂Wij is the error
gradient with reference to the weight Wij.
The Widrow and Hoffs Delta rule and Backpropagation learning rule
are all examples of this type of learning mechanism.
The Least mean square and Back propagation are two variations of
this learning technique.
110 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Stochastic Learning Method

Stochastic learning
In this method, weights are adjusted in a probabilistic fashion.
An example is evident in simulated annealing-the learning
mechanism employed by Boltzmann and Cauchy machines, which
are a kind of NN systems.

111 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Hebbian Learning Method

Hebbian Learning
This rule was proposed by Hebb (1949) and is based on correlative
weight adjustment. This is the oldest learning mechanism inspired by
biology.
In this method, the input-output pattern pairs (Xi, Yi) are associated
by the weight matrix W, known as the correlation matrix. It is
computed as:
n
W= i i
T
X Y
i 1

Hear YiT is the transpose of the associated output vector Yi.

Numerous variants of the rule have been proposed (Anderson, 1983;
Kosko. 1985; Lippman, 1987; Linsker, 1988).

112 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Competitive Learning Method

Competitive learning
In this method, those neurons which respond strongly to input stimuli
have their weights updated.
When an input pattern is presented, all neurons in the layer compete
and the winning neuron undergoes weight adjustment.
Hence, it is a “Winner – takes – all” strategy.
The connections between the output neurons show the competition
between them and one of them would be ‘ON’ which means it would
be the winner and others would be ‘OFF’.

113 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Question Answers

1. The structural constitute of a human brain is known as -----

a) Neuron b) Cells c) Chromosomes d) Genes

2. Neural networks also known as -----------------------

a) Artificial Neural Network b) Artificial Neural Systems

c) Both A and B d) None of the above

3. Neurons also known as -----------------

a) Neurodes b) Processing elements c) Nodes d) All the above

4. In the neuron, attached to the soma are long irregularly

shaped filaments called--------------

a) Dendrites b) Axon c) Synapse d) Cerebellum

114 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Question Answers

5. Signum function is defined as -------------------

a) φ(I) =+1, I>0, -1, I<=0 b) φ(I)=0 c) φ(I)=+1,I>0

d) φ(I)=-1,I<=0

6. To generate the final output, the sum is passed on to a non-

linear filter φ called

a) Smash function b) Sum function

c) Activation function d) Output function

7. ---------------function is a continuous function that varies

gradually between the asymptotic values 0 and 1 or -1 and +1

a)Activation function b)Thresholding function

c)Signum function d)Sigmoidal function
115 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Topic for the Next Class

Introduction to Early ANN architectures (basics only)

McCulloch & Pitts Model
Perceptron, ADALINE, MADALINE

116 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Outlines of lecture - 7

 Introduction to early ANN Architecture

 McCulloch & Pitts Model

 Perceptron, ADALINE, MADALINE

117 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing MP Model of NN

It is very well known that the most fundamental unit of neural

networks is called an artificial neuron / Perceptron. But the very
first step towards the Perceptron we use today was taken in 1943 by
McCulloch and Pitts, by mimicking the functionality of a biological
neuron.
The first computational model of a neuron was proposed by Warren
MuCulloch (neuroscientist) and Walter Pitts (logician) in 1943.

118 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing MP Model of NN

Boolean Functions Using M-P Neuron

So far we have seen how the M-P neuron works. Now lets look at
how this very neuron can be used to represent a few Boolean
functions. Note that our inputs are all Boolean and the output is also
Boolean so essentially, the neuron is just trying to learn a Boolean
function. A lot of Boolean decision problems can be cast into this,
based on appropriate input variables
M-P Neuron: A Concise Representation
This representation just
denotes that, for the Boolean
inputs x1, x2 and x3 if
the g(x) i.e., sum ≥ Ө, the
neuron will fire otherwise, it
won’t.

119 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Perceptron

So here goes, a perceptron is not the Sigmoid neuron we use in

ANNs or any deep learning networks today.

120 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Perceptron

The perceptron model is a more general computational model than

McCulloch-Pitts neuron. It takes an input, aggregates it (weighted
sum) and returns 1 only if the aggregated sum is more than some
threshold else returns 0. Rewriting the threshold as shown above and
making it a constant input with a variable weight, we would end up
with something like the following:

121 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Perceptron

A single perceptron can only be used to implement linearly

separable functions. It takes both real and Boolean inputs and
associates a set of weights to them, along with a bias. We learn the
weights, we get the function. Let's use a perceptron to learn an OR
function.
OR Function Using A Perceptron

122 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Linearly Separable & Non Separable Function

AND, OR functions are Linearly-Separable

XOR function is Not Linearly Separable.

123 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing ADALINE Network

ADALINE which stands for Adaptive Linear Neural Element, is a

network having a single linear unit. It was developed by Widrow and
Hoff in 1960. Some important points about ADALINE are as follows
It uses bipolar activation function.
It uses delta rule for training to minimize the Mean-Squared
Error MSE between the actual output and the desired/target output.
The weights and the bias are adjustable.
The supervised learning algorithm adopted by the network is similar
to the perceptron learning algorithm. Devised by Widrow-
Hoff(1960), the learning algorithm is also known as the Least Mean
Square (LMS) or Delta rule.

124 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing ADALINE Network Architecture

The basic structure of ADALINE is similar to Perceptron having an

extra feedback loop with the help of which the actual output is
compared with the desired/target output. After comparison on the
basis of training algorithm, the weights and bias will be updated.

125 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing ADALINE Training

y=b+∑

1 If y >= 0
f(y) =
−1 If y < 0

Case 1 if y ≠ t then,
wi(new)=wi(old)+α(t−y)xi
b(new)=b(old)+α(t−y)
Case 2 if y = t then,
wi(new)=wi(old)
b(new)=b(old)
Where α is the learning rate, y is the computed output and t is the
desired/target output, xi is the input and wi is the weight.
(t−y) is the computed error.

126 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing MADALINE Network

MADALINE which stands for Multiple Adaptive Linear Neural

Elements, is a network which consists of many ADALINE in
parallel. It will have a single output unit. Some important points
about MADALINE are as follows −
 It is just like a multilayer perceptron, where Adaline will act as
a hidden unit between the input and the MADALINE layer.
 The weights and the bias between the input and Adaline layers,
as in we see in the Adaline architecture, are adjustable.
 The Adaline and MADALINE layers have fixed weights and
bias of 1.
 Training can be done with the help of Delta rule.
The architecture of MADALINE consists of “n” neurons of the
input layer, “m” neurons of the Adaline layer, and 1 neuron of the
MADALINE layer. The Adaline layer can be considered as the
hidden layer as it is between the input layer and the output layer, i.e.
the MADALINE layer.
127 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing MADALINE Network Architecture

The architecture of MADALINE consists of “n” neurons of the

input layer, “m” neurons of the Adaline layer, and 1 neuron of the
MADALINE layer. The Adaline layer can be considered as the
hidden layer as it is between the input layer and the output layer, i.e.
the MADALINE layer.

128 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Topic for the Next Class

Backpropagation Neural Network algorithm

129 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing The Concept of Learning

 The learning is an important feature of human computational

ability.
 Learning may be viewed as the change in behavior acquired due
to practice or experience, and it lasts for relatively long time.
 As it occurs, the effective coupling between the neuron is
modified.
 In case of artificial neural networks, it is a process of modifying
neural network by updating its weights, biases and other
parameters, if any.
 During the learning, the parameters of the networks are optimized
and as a result process of curve fitting.
 It is then said that the network has passed through a learning
phase.

130 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Single Layer feed forward NN training

 We know that, several neurons are arranged in one layer with

inputs and weights connect to every neuron.
 Learning in such a network occurs by adjusting the weights
associated with the inputs so that the network can classify the
input patterns.
 A single neuron in such a neural network is called perceptron.
 The algorithm to train a perceptron is stated below.
 Let there is a perceptron with (n + 1) inputs x0, x1, x2,…..,

 Let f denotes the transfer function of the neuron. Suppose, X and

xnwhere x0 = 1 is the bias input.

Ydenotes the input-output vectors as a training data set. Wdenotes

the weight matrix.
With this input-output relationship pattern and configuration of a
perceptron, the algorithm Training Perceptron to train the
perceptron is stated in the following slide.

131 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Single Layer feed forward NN training

For each input pattern x ∈ Xdo Here, x = {x0, x1, … , xn}

1. Initialize W = w0, w1, … ,wn to some random weights.

Compute I = ∑
2.

1, >0
0, ≤0
Compute observed output y = f (I) =

Y′ = Y′+ y Add y to Y′ , which is initially empty

If the desired output Y matches the observed output Y′ then
output W and exit.
3.

Otherwise, update the weight matrix W as follows :

For each output y ∈ Y′ do
4.

If the observed out y is 1 instead of 0, then wi = wi - αxi,

(i = 0, 1, 2, ... , n)
Else, if the observed out y is 0 instead of 1, then wi = wi + αxi,
(i = 0, 1, 2, ... , n)
5. Go to step 2.

132 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Single Layer feed forward NN training

In the above algorithm, α is the learning parameter and is a constant

decided by some empirical studies.
Note :
 The algorithm Training Perceptron is based on the supervised
learning technique
 ADALINE : Adaptive Linear Network Element is also an
alternative term to perceptron
 If there are 10 number of neutrons in the single layer feed
forward neural network to be trained, then we have to iterate
the algorithm for each perceptron in the network.

133 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Specification of Multi Layer feed forward NN

Like single layer feed forward neural network, supervisory training

methodology is followed to train a multilayer feed forward neural
network.
Before going to understand the training of such a neural network, we
redefine some terms involved in it.
A block diagram and its configuration for a three layer multilayer FF
NN of type l - m - n is as follows.

134 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Specification of Multi Layer feed forward NN

135 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Specification of Multi Layer feed forward NN

For simplicity, we assume that all neurons in a particular layer follow

same transfer function and different layers follows their respective
transfer functions as shown in the configuration.
Let us consider a specific neuron in each layer say i-th, j-th and k-th
neurons in the input, hidden and output layer, respectively.
Also, let us denote the weight between i-th neuron (i = 1, 2, ... , l ) in
input layer to j-th neuron (j = 1, 2, ... , m) in the hidden layer is
denoted by vij.
The weight matrix between the input to hidden layer say V is

v11 v12 … v1j … v1m

denoted as follows.

v21 v22 … v2j … v2m

… … … … … …
V= vi1 vi2 … vij … vim
… … … … … …
1 2 … ! … "

136 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Specification of Multi Layer feed forward NN

Similarly, wjk represents the connecting weights between j-th

neuron(j = 1, 2, ... ,m) in the hidden layer and k-th neuron (k = 1, 2,
... , n) in the output layer as follows:

w11 w12 … w1k … w1m

w21 w22 … w2k … w2n
… … … … … …
wj1 wj2 … wjk … wjn
W =
… … … … … …
wm1 wm2 … wmk … wmn

137 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Learning of Multi Layer feed forward NN

Whole learning method consists of the following three computations:

 Input layer computation

 Hidden layer computation

 Output layer computation

In our computation, we assume that < T0, TI > be the training set of
size |T|.

138 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Input Layer Computation

{I}I = [I11, I12, … , I1i, I1l ] where {I}I ∈ {T}I

Let us consider an input training data at any instant be

Consider the outputs of the neurons lying on input layer are the same
with the corresponding inputs to neurons in hidden layer.
That is, {O}I = {I}I
[l×1] [l×1] [Output of the input layer]
The input of the j-th neuron in the hidden layer can be calculated as
follows.
IHj = V1j .OI1 + V2j .OI2 +, … , + Vij .OIj + … + Vlj .OIl
where j = 1, 2, …, m.
[Calculation of input of each node in the hidden layer]

{I}H = [V]T × {O}I

In the matrix representation form, we can write

[m×1] [m×l] [l×1]

139 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Hidden Layer Computation

Let us consider any j-th neuron in the hidden layer.

Since the output of the input layer’s neurons are the input to the j-th
neuron and the j-th neuron follows the sigmoidal activation function,
we have
( () *+ ,-. * /-. )
O = Hj

where j = 1, 2, … , m and αH is the constant co-efficient of the

transfer function.
Note that all output of the nodes in the hidden layer can be expressed
as a one-dimensional column matrix.
….

1
….

(1 + 2 34 567 3 867 )
{O}H =
….
…. [m × 1]

140 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Output Layer Computation
Let us calculate the input to any k-th node in the output layer. Since,
output of all nodes in the hidden layer go to the k-th layer with
weights w1k , w2k , … ,wmk , we have
IOk = W1k . OH1 + W2k . OH2 + … + Wmk . Ohm where k = 1, 2, … , n.

{I}O = [W]T × {O}H

In the matrix representation, we have

[n×1] [n× "] [m×1]

Now, we estimate the output of the k-th neuron in the output layer.
We consider the sigmoidal activation function.
( ()
*+ , * / )
OOk = for k = 1, 2, … , n.
9: 9:

….

1
….

(1 + 2 34 5;< 3 8;< )
{O}O =
….
[n × 1]
….
141 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Backpropagation Algorithm

 The above discussion comprises how to calculate values of

different parameters in l - m - n multiple layer feed forward
neural network.
 Next, we will discuss how to train such a neural network.
 We consider the most popular algorithm called Back-Propagation
Algorithm, which is a supervised learning.
 The principle of the Back-Propagation algorithm is based on
the error-correction with Steepest-descent method.
 We first discuss the method of steepest descent followed by its
use in the training algorithm.

142 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Method of Steepest Descent

 Supervised learning is, in fact, error-based learning.

 In other words, with reference to an external (teacher) signal (i.e.
target output) it calculates error by comparing the target output
and computed output.
 Based on the error signal, the neural network should modify its
configuration, which includes synaptic connections, that is , the
weight matrices.
 It should try to reach to a state, which yields minimum error.
 In other words, its searches for a suitable values of parameters
minimizing error, given a training set.
 Note that, this problem turns out to be an optimization problem.

143 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Method of Steepest Descent

(a) Searching for a (b) Error surface with two

minimum error parameters V and W

144 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Method of Steepest Descent

 For simplicity, let us consider the connecting weights are the

only design parameter.
 Suppose, V and W are the wights parameters to hidden and
output layers, respectively.

E= ∑? 2 (=, >, )
 Thus, given a training set of size N, the error surface, E can be
represented as

 where Ii is the i-th input pattern in the training set and ei (…)
denotes the error computation of the i-th input.
 Now, we will discuss the steepest descent method of computing
error, given a changes in V and W matrices.

145 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Calculation of Error in Neural Network

pattern Ii ∈ TI (input in training) the target output TOk of the k-th

Let us consider any k-th neuron at the output layer. For an input

neuron be TOk .
Then, the error ek of the k-th neuron is defined corresponding to the
@
input Ii as ek = (TOk - OOk )2

For a training session with Ii ∈ TI , the error in prediction considering

where OOk denotes the observed output of the k-th neuron.

e = ∑< ek = ∑< (TOk − OOk )2

all output neurons can be given as
@
where n denotes the number of neurons at the output layer.
The total error in prediction for all output neurons can be determined

E = ∑∀ 5 ∈ B5 2 = ∑∀ C ∈DB5,B;E ∑< (TOk − OOk)2

considering all training session < TI , TO > as
@

146 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Backpropagation Algorithm

Step-1: Normalize the inputs and outputs with respect to their

maximum values. It is proved that the neural networks work better if
input and outputs lie between 0-1. for each training pair, assume there
are ‘l’ inputs given by {I}I and ‘n’ outputs {O}O in a normalized
form.
Step-2: Assume the number of Neurons in the hidden layer to lie
between l<m<2l.
Step-3: [V] represents the weights of synapses connecting input
neuron and hidden neuron and [W] represents the weights of
synapses connecting hidden neuron and output neurons. Initialize the
weights to small random values usually from -1 to +1. For general
problems, λ can be assumed as 1 and threshold value as 0.
[V]0 = [random weights]
[W]0 = [random weights]
[∆V]0 = [∆W]0 = [0]

147 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Backpropagation Algorithm

Step-4: For training data, we need to present one set of inputs and
outputs. Present the pattern as inputs to the input layer {I}I as
inputs to the input layer. By using linear activation function, the
output of the input layer may be evaluated as
{ O } I = { I }I
ℓx1 ℓx1
Step-5: Compute the inputs to the hidden layers by multiplying
corresponding weights of synapses as
{ I }H = [ V] T { O }I
[m x 1] [m x ℓ] [ℓ x 1]
Step-6: Let the hidden layer units, evaluate the output using the
sigmoidal function as ….
….
1
{O}H = (1 +….
2 356 )
…. [m × 1]
148 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Backpropagation Algorithm

Step-7: Compute the inputs to the output layers by multiplying

corresponding weights of synapses as
{ I }O = [ W ] T { O }H
[n x 1] [n x m] [m x 1]
Step-8: Let the output layer units, evaluate the output using
sigmoidal function as ….
….
1
{O}O = (1 +….
2 35;K )
…. [n × 1]
The above is the network output
Step-9: Calculate the error using the difference between the network
output and the desired output as for the jth training set as
∑ FG 3HHG I

J
EP =

149 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Backpropagation Algorithm
….
Step-10: Find {d} as ….

{d} = (Tk –OOk) OOk (1-OOk)

[n × 1]
….
….

[ Y ] = { O }H 〈 d 〉
Step-11: Find [ Y ] matrix as

mxn mx1 1xn

Step-12: Find [∆W] t +1 = α [∆W]t + η [ Y ]
mxn mxn mxn
Step-13: Find {e} = [W] {d}
….
mx1 mxn nx1 ….

{d*} = ei (OHi) (1-OHi)

[m × 1]
….
….
150 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Backpropagation Algorithm

[ X ] = { O }I 〈 d* 〉 = { I }I 〈 d* 〉
Find [ X ] matrix as

ℓxm ℓx1 1xm ℓx1 1xm

Step-14: Find [∆V] t +1 = α [∆V]t + η [ X ]
ℓxm ℓxm ℓxm

Step-15: Find [V] t +1 = [V] t + [∆V] t +1

[W] t +1 = [W] t + [∆W] t +1
Step-16: Find error rate as
∑ NO
P)C
Error Rate =
Step-17: Repeat steps 4 to 16 until the convergence in the error
rate is less than the tolerance value

151 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Topic for the Next Class

Backpropagation Neural Network algorithm Example

152 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Backpropagation Algorithm Simulation

Table : Training sets

Input Output
Sl. No.
I1 I2 O
1 0.4 -0.7 0.1
2 0.3 -0.5 0.05
3 0.6 0.1 0.3
4 0.2 0.4 0.25
5 0.1 -0.2 0.12

0.4 0.1
0.2
TO = 0.1
0.4 -0.2
-0.7
0.2 -0.5

Input Hidden Output

Layer Layer Layer
153 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Backpropagation Algorithm Simulation

0.4
−0.7
Step-1: Find {O}I = {I}I = 2x1

ℓx1 ℓx1

0.1 0.4 0.2

Step-2: Initialize the weights as

−0.2 0.2 −0.5 2x1

0
[V] = [W] 0 =
2x2

Step-3: Find {I}H = [V] T {O}I as

[m x 1] [m x ℓ] [ℓ x 1]

0.1 −0.2 0.4 0.18

0.4 0.2 −0.7 0.02
= x =

1
Step-4: Find
(1 + 2 3X. Y ) 0.5448
{O}H = 1 =
0.505
(1 + 2 3X.X@ )

154 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Backpropagation Algorithm Simulation

Step-5: Find {I}O = [W] T {O}H as

0.5448
[m x 1] [m x ℓ] [ℓ x 1]
= 0.2 − 0.5 x = −0.14354
0.505
1
Step-6: Find
{O}O = (1 + 2 X. \]^\ ) = 0.4642

Step-7: Error = (TO – OO)2 = (0.1 – 0.4642)2 = 0.13264

Step-8: Let us adjust the weights
Find d = (TO – OO1 ) ( OO1 ) (1 – OO1 )
= (0.1 – 0.4642) (0.4642) (0.5358) = –0.09058

0.5448 −0.0493
[Y] = {O}H 〈 d 〉 = x 〈–0.09058〉 =
0.505 −0.0457

155 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Backpropagation Algorithm Simulation

Step-9: Find [∆W]1 = α [∆W]0 + η [Y] (Assume η = 0.6)

−0.0493 −0.02958
−0.0457 −0.02742
= x [0.6] =
0.2 −0.018116
Step-10: Find {e}= [W] 〈d〉 = x 〈–0.09058〉 =
−0.5 −0.04529
−0.00449
Step-11: 〈d 〉 =
0.01132
*
(–0.018116) (0.5448) (1− 0.5448)
=
0.4
(0.04529) (0.505) ( 1 – 0.505)
Step-12: [X] = {O}I 〈d*〉 = x −0.00449 0.01132
−0.7
– 0.001796 0.004528
=
0.003143 –0.007924
Step-13: Find [∆V]1 = α [∆V]0 + η [X]
– 0.001077 0.002716
=
0.1 0.4
0.001885 –0.004754

−0.2 0.2
1 – 0.001077 0.002716
Step-14: Find [V] = +
0.001885 –0.004754

156 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Backpropagation Algorithm Simulation

0.2 −0.02958 −0.17042

−0.5 −0.02742 −0.52742
Step-14: Find [W]1 = + =
Step-15: With the updated weights [V] and [W], error is calculated
again and next training set is taken and the error will then get
adjusted.
Step 16 : Iterations are carried out till we get the error less than the
tolerance.
Step 17 : Once the weights are adjusted the network is ready for
inferencing new objects .

157 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Backpropagation Key Points

• Simplifies the network structure by elements weighted links that

have the least effect on the trained network
• You need to study a group of input and activation values to
develop the relationship between the input and hidden unit layers.
• It helps to assess the impact that a given input variable has on a
network output. The knowledge gained from this analysis should
be represented in rules.
• Backpropagation is especially useful for deep neural networks
working on error-prone projects, such as image or speech
recognition.
• Backpropagation takes advantage of the chain and power rules
allows backpropagation to function with any number of outputs.

158 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Effect of Tunning Parameters of the BPN

• The proper selection of tuning parameters such as Momentum

factor(α), Learning Coefficient(η), Sigmoidal Gain(λ) and
Threshold value(θ) are required for efficient learning and
designing of a stable network.
• Weight adjustment is made on the momentum method.
• The momentum factor α has a significant role in deciding the
value of learning rate that will produce rapid learning.
• It determines the step size of change in Weights or Biases.
• If the Momentum factor is 0, then the smoothening is minimum
and the entire weight adjustment comes from the newly calculated
change.
• If the Momentum factor is 1, then the new adjustment is ignored
and the previous one is repeated.
• The value of Momentum factor between 0 and 1 is region where
the weight adjustment is smoothened by an amount proportional to
the momentum factor.
159 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Effect of Tunning Parameters of the BPN

• Momentum factor of 0.9 has been found to be suitable for most of

the problems.
• The role of Momentum factor is to be increase the speed of
learning without leading oscillations.

• The choice of Learning Coefficient is a tricky task in

Backpropagation Algorithm.
• The range of Learning Coefficient that will produce rapid training
depends on the number and types of input patterns.
• An empirical formula to select Learning Coefficient has been
suggested by Eaton and Oliver in (1992) is given as
.^
(? (?@ ( …(?` )
• η = @ @ @

• Where N1 is the number of patterns of type 1 and m is the number

of different pattern types.
• The suggested optimum value of Learning rate is 0.6.

160 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Effect of Tunning Parameters of the BPN

• If the sigmoidal function is selected, the input-output relationship

of the neuron can be set as
1
• O =
(1+e-λ(I+θ))
• Where λ is a scaling factor known as sigmoidal gain.
• To get a graded output or a binary output, scaling factor can be
varied.
• The value of sigmoidal gain also affects Backpropagation.
• Improper combinations of scaling factor, learning rate and
momentum factor might lead to over correction and poor
convergence.
• To get the graded output, as the scaling factor is increased,
learning rate and momentum factor have to be decreased in order
to prevent oscillations.

161 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Effect of Tunning Parameters of the BPN

• Θ (Thita) is commonly called as threshold value of a neuron, or

the bias or the noise factor.
• A neuron fires or generates an output if the weighted sum of the
input exceeds the threshold value.
• One method is to simply assign a small value to it and not to
change it during training.
• The other method is to initially choose some random values and
change them during training.
• It is hard to say which method is more efficient.

162 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Next Class

 Associative Memory

 Auto Associative Memory

 Hetero Associative Memory

163 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Lecture -13

Outline of the Lecture

 Quick overview of previous Class

 Associative Memory

164 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Associative Memory

An associative memory is a content-addressable structure that

maps a set of input patterns to a set of output patterns.
A content-addressable structure is a type of memory that allows the
recall of data based on the degree of similarity between the input
pattern and the patterns stored in memory.
There are two types of associative memory : auto-associative and
hetero-associative.
An auto-associative memory retrieves a previously stored pattern
that most closely resembles the current pattern.
In a hetero-associative memory, the retrieved pattern is in general,
different from the input pattern not only in content but
possibly also in type and format.
Neural networks are used to implement these associative memory
models called NAM (Neural associative memory).

165 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Associative Memory

Associative Mapping of Inputs to Outputs

Input Output

Block diagram of an Associative Memory

166 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Associative Memory

An associative memory is a content-addressable structure that maps a

set of input patterns to a set of output patterns.

Input Recalled
Pattern Pattern

The working of an Associative Memory

167 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Associative Memory
 An associate memory is a store house of associated patterns which
are encoded in some form
 If the associated pattern pairs (x, y) are different and if the model
outputs y given x or vice-versa, Then it is termed as hetero
associative memory
 If , x and y refer to the same pattern , then it is termed as auto
associative memory.
 An auto associative memory retrieves a previously stored pattern
that most closely resembles the current pattern.
 In a hetero associative memory, the retrieved pattern is, in general,
different from the input pattern not only in content but possibly
also in type and format.
 Hetero associative memories are useful for the association of
patterns, auto associative memories are useful for image
refinement, that is, given a distorted of a partial pattern, the whole
pattern stored in its perfect form can be recalled.
 Auto-associative correlation Memories are known as
Autocorrelator.
168 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Associative Memory
 Auto-associative correlation Memories are known as
Autocorrelators.
 Hetero-associative correlation Memories are known as
Hetrocorrelators.

Input Recall of Presented Recall of

Pattern Associated Distorted Perfect
Presented Pattern Pattern Pattern
Hetero-Associative Memory Auto-Associative Memory

Hetero and Auto Correlators

169 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Autocorrelators

 Autocorrelator was Introduced by Donald Hebb in 1949

 It is Also known as Hopfield Associative Memory (HAM)
 A Hopfield ANN is a time iterative feedback network which
consists of N neurons. The output of each neuron is connected to
every other neuron
 The first order Autocorrelators obtain their connection weight
matrix is symmetric in nature (indicative of the association of the
pattern itself) by multiplying a pattern’s element with every other
pattern’s elements.
 A first order Autocorrelator stores M bipolar patterns (A1, A2, A3,
`
……, Am) by summing together m outer product as

a = c d a [d ]
X
 Here, T = [tij] is a (p x p) connection matrix and Ai ∈ −1, 1 g

170 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Auto-correlators

 The Autocorrelator’s recall equation is a vector-matrix

multiplication followed by a pointwise nonlinear threshold
operation.

ajnew = f(ai . tij , ajold) ∀ j = 1, 2, …, p

 The recall equation is given by

 Where Ai = (a1, a2, …, ap) and the two parameter bipolar threshold

1, h>0
function is

h, i = j i, h=0
−1, h<0

171 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Working of an Auto-correlators

 Consider the following patterns

 A1 = (-1, 1, -1, 1)
 A2 = (1, 1, 1, -1)
 A3 = (-1, -1, -1, 1)
which are to be stored as an autocorrelator.

3 1 3 −3
 The connection matrix is
]

a = c d a 4l1 d 1l4 = 1 3 1 −1
3 1 3 −3
−3 −1 −3 3

172 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Recognition of Stored patterns

 The autocorrelator is presented a stored pattern A2 = (1, 1, 1, -1).

ajnew = f(ai . tij , ajold) ∀ j = 1, 2, …, p

The computation of equation
yields,

a1new = f(3 + 1 + 3 + 3, 1) = (10,1) = 1 3 1 3 −3

1 3 1 −1
3 1 3 −3
a2new = f(1 + 3 + 1 + 1, 1) = (6, 1) = 1

−3 −1 −3 3
a3new = f(3 + 1 + 3 + 3, 1) = (10, 1) = 1
a4new = f(-3 - 1 - 3 - 3, 1) = (-10, 1) = -1

This is indeed the vector itself.

Also, in the retrieval of A3 = (-1, -1, -1, 1)

(a1new , a2new, a3new, a4new ) = (-1, -1, -1, 1)
Yielding the same vector.

173 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Recognition of Noisy patterns

 Consider a vector A' = (1, 1, 1, 1) which is a distorted presentation

of one among the stored patterns
 We proceed to find the proximity of the noisy vector to the stored
patterns using the Hamming distance measure.
 The Hamming Distance (HD) of a vector X from Y, given
X=(x1, x2, x3, …, xn) and Y=(y1, y2, y3, …, yn) is given by

Hn ,o = c| −o |

 Thus the HD of A' from each of the patterns in the stored set is as
follows:
HD (A' , A1) = 4
HD (A' , A2) = 2
HD (A' , A3) = 6

174 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Recognition of Noisy patterns

 It is evident that the vector A' is closer to A2 and therefore

resembles it, or in other words, is a noisy version of A2

ajnew = f(ai . tij , ajold) ∀ j = 1, 2, …, p

 Now the computations using
yields,

 (a1new , a2new, a3new, a4new ) = (f(4, 1), f(4, 1), f(4, 1), f(-4, 1))
= (1, 1, 1, -1) = A2
 Hence, in the case of partial vectors, an autocorrelator results in
the refinement of the pattern or removal of noise to retrieve the
closest matching stored pattern.

175 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Hetrocorrelators

The Hopfield one-layer unidirectional auto-associators have been

discussed in previous section. Kosko (1987) extended this network to
two-layer bidirectional structure called Bidirectional Associative
Memory (BAM) which can achieve hetero-association. The important
performance attributes of the BAM is its ability to recall stored
pairs particularly in the presence of noise.
Definition : If the associated pattern pairs (X, Y) are different
and if the model recalls a pattern Y given a pattern X or vice-
versa, then it is termed as hetero-associative memory.
This section illustrates the bidirectional associative memory :
Operations (retrieval, addition and deletion) ,
Energy Function (Kosko's correlation matrix, incorrect recall of
pattern),
Multiple training encoding strategy (Wang's generalized correlation
matrix)

176 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Hetrocorrelators

BAM is a two-layer nonlinear neural network. Denote one layer as

field A with elements Ai and the other layer as field B with
elements Bi. The basic coding procedure of the discrete BAM is
as follows:

Consider N training pairs {(A1, B1), (A2, B2), ..., (Ai, Bi), ..., (An, Bn)}
where Ai = (ai1 , ai2 , . . . , ain) and Bi = (bi1 , bi2 , . . . , bip)
and aij, bij are either in ON or OFF state.

In binary mode , ON = 1 and OFF = 0 and

In bipolar mode, ON = 1 and OFF = -1
We frame the correlation matrix

q = cla r

177 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Hetrocorrelators

To retrieve the nearest of (Ai , Bi) pattern pair, given any pair (α, β),
the recall equation are as follows:
starting with (α, β) as the initial condition, we determine a finite
sequence (α', β'), (α", β"), ….. until an equilibrium point (αF, βF), is
reached, where
β' = ϕ(αM)
α' = ϕ(β'MT)
ϕ(F) = G = g1, g2, …, gn
F = (f1, f2, …, fn)

1 >0
0 t uvwo , <0
s =
−1 t gx vw , <0
yw2 xz{ s , =0

178 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Addition and Deletion of Pattern Pairs

Given a set of pattern pairs (Xi , Yi), for i = 1 , 2, . . . , n and their

correlation matrix M, a new pair (X' , Y’) can be added or an
existing pair (Xj , Yj) can be deleted from the memory model.
Addition :
In case of addition, if we add a new pair (X', Y'), to the existing
correlation matrix M, then the new correlation matrix Mnew is given
by Mnew = X1T Y1 + X2T Y2 + …… + XnT Yn + X'T Y'
Deletion :
In case of deletion, if we subtract the matrix corresponding to an
existing pair (Xj , Yj) from the correlation matrix M, then the new
correlation matrix Mnew is given by Mnew = M - (XjT Yj)

Note : The addition and deletion of information is similar to the

functioning of the system as a human memory exhibiting learning
and forgetfulness.

179 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Energy Function for BAM

 A system that changes with time is a dynamic system.

 There are two types of dynamics in a neural network.
 During training phase it iteratively update weights and
 During production phase it asymptotically converges to the
solution patterns.
 State is a collection of qualitative and qualitative items that
characterize the system e.g., weights, data flows.
 The Energy function (or Lyapunov function) is a bounded
function of the system state that decreases with time and the
system solution is the minimum energy.

180 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Energy Function for BAM

 Let a pair (A , B) defines the state of a BAM.

 To store a pattern, the value of the energy function for that pattern
has to occupy a minimum point in the energy landscape.
 Also adding a new patterns must not destroy the previously stored
patterns.
 The stability of a BAM can be proved by identifying the energy
function E with each state (A , B) .
 For Auto-associative memory :
 The energy function is E(A) = - AMAT
 For bidirectional Hetero associative memory :
 The energy function is E(A, B) = - AMBT
 For a particular case A = B , it corresponds to Hopfield auto-
associative energy function.

181 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Energy Function for BAM

 We wish to retrieve the nearest of (Ai , Bi) pair, when any(α , β)

pair is presented as initial condition to BAM.
 The neurons change their states until a bidirectional stable state
(Af , Bf) is reached.
 Kosko has shown that such stable state is reached for any
matrix M when it corresponds to local minimum of the energy
function.
 Each cycle of decoding lowers the energy E if the energy
function for any point (α , β) is given by E = αMβT
 If the energy evaluated using coordinates of the pair (Ai , Bi) i.e.
E = AiMBiT does not constitute a local minimum, then the
point cannot be recalled, even though one starts with α = Ai.
 Thus Kosko's encoding method does not ensure that the stored
pairs are at a local minimum.

182 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Working with Kosko’s BAM

 The working of Kosko's BAM for retrieval of associated pair.

 Start with X3, and hope to retrieve the associated pair Y3 .

• Consider N = 3 pattern pairs (A1 , B1), (A2 , B2), (A3 , B3) given by:

A1 = (1 0 0 0 0 1) B1 = (1 1 0 0 0)
A2 = (0 1 1 0 0 0) B2 = (1 0 1 0 0)
A3 = (0 0 1 0 1 1) B3 = (0 1 1 1 0)

• Convert these 3 binary pattern to bipolar form replacing 0s by -1s.

X1 = (1 -1 -1 -1 -1 1) Y1 = (1 1 -1 -1 -1)
X2 = ( -1 1 1 -1 -1 -1) Y2 = (1 -1 1 -1 -1)
X3 = ( -1 -1 1 -1 1 1) Y3 = (-1 1 1 1 -1)

183 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Working with Kosko’s BAM

1 1 −3 −1 1
The correlation matrix M is calculated as 6x5 matrix

1 −3 1 −1 1
−1 −1 3 1 −1
q = l1ar1 + l2ar2 + l3ar3 =
−1 −1 −1 1 3
−3 1 1 3 1
−1 3 −1 1 −1
Suppose we start with α = X3, and we hope to retrieve the associated
pair Y3. The calculations for the retrieval of Y3 yield :
αM = ( -1 -1 1 -1 1 1 ) (M) = ( -6 6 6 6 -6 )
β' = ϕ (αM) = ( -1 1 1 1 -1 )
β'MT = ( -5 -5 5 -3 7 5 )
ϕ (β'MT) = ( -1 -1 1 -1 1 1) = α'
α'M = ( -1 -1 1 -1 1 1 ) (M) = ( -6 6 6 6 -6 )
ϕ(α'M) = β" = ( -1 1 1 1 -1) = β’
Here, β' is same as Y3. Hence (αF, βF) = (X3, Y3) is the desired result.

184 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Working with Kosko’s BAM

The Working of incorrect recall by Kosko's BAM.

Start with X2, and hope to retrieve the associated pair Y2 .
Consider N = 3 pattern pairs (A1 , B1) , (A2 , B2) , (A3 , B3)
given by
A1 = ( 1 0 0 1 1 1 0 0 0 ) B1 = ( 1 1 1 0 0 0 0 1 0 )
A2 = ( 0 1 1 1 0 0 1 1 1 ) B2 = ( 1 0 0 0 0 0 0 0 1 )
A3 = ( 1 0 1 0 1 1 0 1 1 ) B3 = ( 0 1 0 1 0 0 1 0 1 )
Convert these three binary pattern to bipolar form replacing 0s by -1s.
X1 = ( 1 -1 -1 1 1 1 -1 -1 -1 ) Y1 = ( 1 1 1 -1 -1 -1 -1 1 -1 )
X2 = ( -1 1 1 1 -1 -1 1 1 1 ) Y2 = ( 1 -1 -1 -1 -1 -1 -1 -1 1 )
X3 = ( 1 -1 1 -1 1 1 -1 1 1 ) Y3 = ( -1 1 -1 1 -1 -1 1 0 1 )

185 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Working with Kosko’s BAM

q = l1ar1 + l2ar2 + l3ar3

The correlation matrix M is calculated as 9x9 matrix

3 −3 1 −1 −1 1 −3 −1 −1
3 −3 1 −1 −1 1 −3 −1 −1
1 −1 −1 1 1 −1 −1 −3 1
−1 1 1 −1 −1 1 1 3 −1
q= −1 1 1 −1 −1 1 1 3 −1
−1 1 −3 −1 −1 −3 1 −1 3
3 −3 1 −1 −1 1 −3 −1 1
1 −1 −1 1 1 −1 −1 −3 1
−1 1 1 −1 −1 1 1 3 −1

186 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Working with Kosko’s BAM

Suppose we start with α = X2, and we hope to retrieve the associated

pair Y2. The calculations for the retrieval of Y2 yield :
αM = ( 13 -13 -5 1 1 -5 -13 -19 5)
β' = ϕ (αM) = ( 1 -1 -1 1 1 -1 -1 -1 1)
β'MT = ( 5 5 11 -11 -11 5 5 11 -11)
ϕ (β'MT) = ( 1 1 1 -1 -1 1 1 1 -1) = α'
α'M = ( 13 -13 -5 1 1 -5 -13 -19 5)
ϕ(α'M) = β" = ( 1 -1 -1 1 1 -1 -1 -1 1) = β'
Here, β' is same as Y2. Hence (αF, βF) = (X2, Y2) is the desired result.

187 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Exponential BAM

 The capacity N(k) of a specific BAM structure is defined to be

the maximum number of training pairs selected from a uniform
distribution which can be recalled with a minimum specified
probability P = 1- k.
 Wang and Don. (1995) proposed a BAM structure with an
exponential form and it is therefore termed eBAM. eBAM has
higher capacity for pattern pair storage than conventional
BAMs.
 The model takes advantage of the exponential nonlinearity in
the evolution equations causing a significant increase in the
signal-to-noise ratio.
 The energy, as a result decreases as the recall process is in
progress, ensuring the stability of the system.
 The increase in the signal-to-noise ratio also enhances the
capacity of BAM.

188 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing Exponential BAM
?

1 c o | }l ≥ 0
Suppose we are given N
.l
training pairs {(A1, B1),
o =
(A2 , B2),.., (An, Bn)}
where Ai = (ai1, ai2, …, ain) | ?

−1 c o | }l <0
.l
and Bi = (bi1, bi2, …, bin)
and if Xi, Yi are the bipolar
modes of the training
?
respectively, given by Xi ∈ 1 c }r ≥ 0
pattern pairs Ai and Bi
.r
|
{-1, 1}n and Yi ∈ {-1, 1}p
| = ?

−1 c | }r <0
.r
Then, we use the following
equations in the recall
process of eBAM.
Here, b is a positive number, b>1 and “.” represents the linear product

(X. Xi) = ∑7 !. !
operator of X and Xi, Y and Yi i.e. for X=(x1, x2, …, xn) and Xi =
(xi1, xi2, …, xin)
189 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024
 Soft Computing Applications

• Character Recognition
• Defect identification

190 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024

 Soft Computing ART Networks

Adaptive Resonance Theory (ART) Networks

191 Department of CSA, GIET University, Gunupur Thursday, May 2, 2024