2/25/2021

COMPUTATIONAL INTELLIGENCE - CCS619

INTRODUCTION

Feb., 2021

NASEEBAH A. MAQTARY
PhD in Computing
Head of Computer Science Dept.

Office: CS-office in Faculty building

Email: [email protected]

1
 2/25/2021

COURSE CONTENT

 Computational Intelligence: Overview, Concepts, Paradigms, Application Areas and Techniques

 Evolutionary Computation: GA and PSO

 Artificial Neural Networks: Perceptron, Hopefield and SOFM

 Fuzzy Systems: Concepts and Paradigms

 Examples and Case Studies: Presented by the Students

TEXTBOOK

Computational Intelligence:
Concepts to Implementations
R. Eberhart and Y. Shi

2007

2
 2/25/2021

ASSESSMENT

 Assignments: 20%

 Midterm Exam: 10%

 Presentation: 30%

 Final Exam: 40%

LECTURE OUTLINE

 Definitions  Adaptation

 Paradigms  Adaptation Methodologies

 Evolutionary Computation  Adaptation Space

 Artificial Neural Networks  Self Organization

 Fuzzy logic  Intelligent Systems (Adaptive Systems)
 CI Myths  Generalization
 Application Areas

3
 2/25/2021

INTELLIGENCE

 Webster’s New Collegiate Dictionary defines intelligence as “(1) : The ability to learn or understand or to
deal with new or trying situations : REASON; also: the skilled use of reason (2) : the ability to apply
knowledge to manipulate one’s environment or to think abstractly as measured by objective criteria (as
tests).”
 The capability of a system to adapt its behavior (implement decisions) to meet its goals in a range of
environments. It is a property of all purpose-driven decision makers.
 David Fogel - 1995

COMPUTATIONAL INTELLIGENCE

 Computational intelligence comprises practical adaptation and self-organization concepts,

paradigms, algorithms and implementations that enable or facilitate appropriate actions (intelligent
behavior) in complex and changing environments.
 Computational intelligence systems usually incorporate hybrids of paradigms such as artificial neural
networks, fuzzy systems, and evolutionary computation systems, augmented with knowledge elements.
 They are often designed to mimic one or more aspects of biological intelligence.

4
 2/25/2021

ARTIFICIAL NEURAL NETWORK

 An analysis paradigm very roughly modeled after the massively parallel structure of the brain.

 Simulates a highly interconnected, parallel computational structure with numerous relatively simple
individual processing elements (PEs).

FUZZINESS

 Fuzziness: non-statistical imprecision and vagueness in information and data.

 Fuzzy Sets: model the properties of imprecision, approximation or vagueness.

 Fuzzy Membership Values: reflect the membership grades in a set.

 Fuzzy Logic: is the logic of approximate reasoning. It is a generalization of conventional logic (two-
valued, crisp logic).

5
 2/25/2021

EVOLUTIONARY COMPUTATION

 Comprises machine learning optimization and classification paradigms roughly based on mechanisms of
evolution such as natural selection and biological genetics. Includes genetic algorithms, evolutionary
programming, evolution strategies and genetic programming.
 The evolutionary computation field includes genetic algorithms, evolutionary programming, genetic
programming, evolution strategies, and particle swarm optimization.
 All of these paradigms use populations of individuals (potential solutions), rather than single data points
or vectors.

COMPUTATIONAL INTELLIGENCE

 A methodology involving computing that exhibits an ability to learn and/or to deal with new situations,
such that the system is perceived to possess one or more attributes of reason, such as generalization,
discovery, association and abstraction.
 Silicon-based computational intelligence systems usually comprise hybrids of paradigms such as
artificial neural networks, fuzzy systems, and evolutionary algorithms, augmented with knowledge
elements, and are often designed to mimic one or more aspects of carbon-based biological intelligence.

6
 2/25/2021

PARADIGM

 A paradigm is a particular example of computational intelligence attributes—in the case of a neural network, the
architecture, activation and learning rules, update procedure, and so on—that exhibits a certain type of behavior.
Put another way, it is a clear and specific example of a concept. Back-propagation is one example of a neural
network paradigm because it implies a certain set of attributes, for example, the architecture and the learning
rule. A paradigm is a particular set of choices for all attributes. Development of a new paradigm involves
assembling a set of attributes that define the intended behavior of the CI tool.
 An implementation is a computer program written and compiled for a specific computer or class of computers
that implements a paradigm.

BIOLOGICAL BASIS: NEURAL NETWORK

 Neurons: nerve cells; consist of dendrites, body and an axon; signals flow through synapses.

 Some differences between biological and artificial neurons (processing elements):

 Signs of weights (+ or -): Eccles’s law
 Signals are AC (pulse frequency) in neurons, DC in PEs
 Many types of neurons in a system; usually only a few at most in neural networks (2 PEs)
 Basic cycle time for PC (~100 ns) faster than brain (10-100ms) {as far as we know!}

7
 2/25/2021

BIOLOGICAL NEURON

BIOLOGICAL BASIS- EVOLUTIONARY COMPUTATION

 Ties with genetics, “a branch of biology that deals with the heredity and variation of organisms”

 Chromosomes: structures in cell bodies that transmit genetic information; humans have 46, in 23 pairs

 Individual patterns in EC correspond to chromosomes in biological systems

 The genotype completely specifies an organism; in EC a structure specifies a system; in most EC tools,
one string specifies a structure, so structure is interchangeable with chromosome. A solution.

8
 2/25/2021

CHROMOSOMES

BIOLOGY – EC CHROMOSOME DIFFERENCE

 Artificial (EC) chromosomes all same length

 Biological: DNA (various length), EC: bits or real numbers (fixed length)

 In reproduction, biological cells divide, while EC cells copy

 Synthesis of new chromosomes: 50 percent from each biological parent, any percentage from EC
parents. Mutation not intrinsic to biological system as it is in EC.

9
 2/25/2021

FUZZY LOGIC

 FL analogous to uncertainty in human experiences (“Jim is very tall.”)

 Fuzziness is associated with nonstatistical uncertainty.

 FL thus is reflected at the behavioral level of the organism.

 Fuzziness is not resolved by observation or measurement (probability).

CI MYTHS

 The supercomputer/Nobel laureate myth

 CI implementations are faster, cheaper and better than anything else
 CI will eliminate need for programming
 CI is more important than preprocessing
 Only biology experts can use CI
 Fuzzy logic is fuzzy
 Fuzzy logic is a substitute for probability
 Optimization is possible

10
 2/25/2021

APPLICATION AREAS

 Evolutionary Computation
 Optimization: Route Finding, Scheduling
 Classification: Diagnosis (OK, minimal set covering)
 Configuration (aka, design)
 Artificial Neural Network
 Classification/Prediction
 Associative memory
 Clustering or compression
 Simulation or composition
 Control systems
 Fuzzy Logic
 Control systems: Vehicles, Home appliances
 Expert systems: Industrial processes, Diagnostics, Finance, Robotics and manufacturing

ADAPTATION

 Adaptation is any process whereby a structure is progressively modified to give better performance in its
environment.
 Holland 1992
 Adaptive processes are improvement (amelioration) processes. They are usually not really optimization
processes.

11
 2/25/2021

ADAPTATION

 Adaptation overcomes the barriers of nonlinearity and local optima.

 It involves a progressive modification of some structure or structures, and uses a set of operators acting
on the structure(s) that evolve over time.
 Adaptation is “…a fundamental process, appearing in a variety of guises but subject to unified study.”
 J. Holland

 Dynamic adaptation is the ability of a system to adapt “online,” that is, in essentially real time, in a
changing environment. In dynamic adaptation, the system adapts while it is running (online), rather
than being taken offline to be retrained.

BARRIERS TO ADAPTATION

 Large problem spaces

 Large numbers of variables

 Complex and nonlinear fitness functions

 Fitness functions that change over time and over the problem space

 Complex and changing environments

12
 2/25/2021

LAW OF SUFFICIENCY

 If a solution to a problem is:

 Good enough (it meets specs)
 Fast enough
 Cheap enough

 Then it is Sufficient (Being good enough simply means it meets specifications).

 Optimization problems

ADAPTATION VS. LEARNING

 Adaptation: 1: the act or process of adapting : the state of being adapted 2: adjustment to
environmental conditions: as a: adjustment of a sense organ to the intensity or quality of stimulation
b:modification of an organism or its parts that makes it more fit for existence under the conditions of its
environment.
 Adapt: to make fit (as for a specific or new use or situation) often by modification

 Fit: suitable, adapted so as to be capable of surviving, acceptable from a particular viewpoint

13
 2/25/2021

ADAPTATION VS. LEARNING

 Learning: knowledge or skill acquired by instruction or study syn: knowledge

 Learn: to gain knowledge or understanding of or skill in by study, instruction or experience, syn:

discover
 Learning is what an entire intelligent system does.

 Learning thus applies to the entire intelligent system, while adaptation mainly applies to the portion of
the system we address in this course—the portion where computational intelligence exists.
 “Computational intelligence and adaptation (with self-organization) are synonymous”.

SYSTEM ADAPTATION TYPES

 Types
 Supervised adaptation
 Unsupervised adaptation
 Reinforcement adaptation

 Note that in all three cases the adaptation algorithm is separated from the adaptive system.
 Offline
 Online

14
 2/25/2021

SUPERVISED ADAPTATION

 The process of adjusting (adapting) a system

so it produces specified outputs in response to
specified inputs." "Supervised” means that the
output is known for all inputs and the system
training algorithm uses the error to guide the
training. (Reed and Marks 1999)

SUPERVISED ADAPTATION

 A “teacher” provides input-output examples (the “gold standard”)

 Adaptation is carried out one iteration at a time

 Fitness is often inversely proportional to a function of the sum of errors

 Good for function approximation: mapping input vectors to output vectors

 Example: Back-propagation algorithm used to train neural networks

15
 2/25/2021

REINFORCEMENT ADAPTATION

 A "sparse reinforcement signal" grades the

system response as good or bad. A “critic”
provides heuristic reinforcement information.
 Example: game playing.

REINFORCEMENT ADAPTATION

 Most closely related to biological systems

 Has roots in dynamic programming (Optimal control theory)

 Often waits until the time series of inputs is complete to judge the fitness

 The system “critic” only looks at outcomes, not individual error measures

 Example: Particle swarm optimization

16
 2/25/2021

UNSUPERVISED ADAPTATION

 The system adapts to regularities in the data according

to rules implicit in its design. The 'design' is a substitute
teacher. Targets don't exist. (Reed and Marks 1999)
 No external teacher or critic is involved in system
adaptation. Instead, a dataset comprising example
vectors of the system’s variable parameters is provided.
 No indication of fitness exists whatsoever
 Offline evaluation occurs after the algorithm stops
running
 Examples: SOFM and LVQ networks (clustering)

ADAPTATION SPACE

 Input parameter (problem) space

 Defined by dynamic ranges of input variables

 System output (function) space

 Defined by dynamic ranges of output variables

 Fitness space
 Defines “goodness” of solutions; often scaled from 0 to 1

 Remember that, in general, system output and fitness values aren’t the same.

17
 2/25/2021

SELF ORGANIZATION

 apparently spontaneous (autonomous) order

 matter's incessant (continuous) attempts to organize itself into ever more complex structures, even in
the face of the incessant forces of dissolution described by the second law of thermodynamics
 overall system state is emergent property of the system interconnected system components become
organized in a productive or meaningful way based on local information
 The overall system state of a self-organizing system is an emergent property of the system.

 Complex systems can self-organize

 The self-organization process works near the "edge of chaos"

SELF ORGANIZATION

 Bonabeau’s definition of self-organization: “A set of dynamical mechanisms whereby structures appear

at the global level of a system from interactions among its lower-level components. The rules specifying
the interactions among the system’s constituent units are executed on the basis of purely local
information, without reference to the global pattern, which is an emergent property of the system rather
than a property imposed on the system by an external ordering influence.”
 Examples: Formation of ice crystals, salt crystals. Cellular automata. The human brain.

18
 2/25/2021

NEW VIEW OF EVOLUTION

 Complex systems can “appear” over relatively short time (compared with Darwinian evolution)

 It appears that natural selection and self-organization work “hand-in-hand,” i.e., that

evolution = natural selection + self organization

HISTORY OF CI

 First use of the term (in its current context) by James Bezdek in 1992

 First IEEE World Congress on Computational Intelligence in Orlando in 1994

 First CI text in 1996

 Second IEEE World Congress on CI in Anchorage in 1998

 Subsequent World Congresses in Hawaii (2002), Vancouver (2006)…next in Hong Kong (2008)

19
 2/25/2021

COMPONENTS OF INTELLIGENT SYSTEM

ATTRIBUTES OF INTELLIGENCE

 Some attributes of intelligence are not explicitly represented on the diagram:

 Complexity -generally increases from left to right on diagram
 Stochasticity/chaos -probably present in each element of diagram

20
 2/25/2021

CHAOS OR STOCHASTICITY

 CI paradigms are replete (full of) with “stochasticity” or “randomness.”

 NN weight initialization
 NN asynchronous updating
 NN and EC simulated annealing
 EC crossover (recombination)
 EC mutation
 EC selection (usually)

RANDOMNESS

 Randomness is only simulated in computers with deterministic programs

 We therefore are really dealing with pseudo randomness

 As for nature, “God does not play dice.” –A. Einstein

 What we observe as “random” or “stochastic” in nature are actually nonlinear dynamics systems

21
 2/25/2021

GENERALIZATION

 Assume a function y = f(x) maps each input to an output in the problem space, and that our dataset
represents only a small part of the problem space.
 We want to build a model f*(x) such that other values of x will be mapped into Y such that f *(x)  f(x)
for x* not in the dataset. This is generalization.
 We usually assume that f*(x) = f(x) for a perfect system.

 Note that we usually split our data set into training and test sets, and we thus usually measure the
generalization capability on the test set. Note also that the size of the dataset must be sufficiently large.

WHAT IS ABOUT AI?

 Where does AI fit in? At the shell of the Adaptation and Self-organization node, and in the World Model,
mainly.
 CI attributes that do not hold for AI and hard computing:
 The ability to generalize
 The ability to deal with partial truths and uncertainty
 Tolerance for errors and noise
 The ability to perform well in complex and changing environments

 Hard computing attributes that do not hold for CI systems:

 Precision
 Certainty

22
 2/25/2021

SOFT COMPUTING

 Soft computing is not a single methodology. Rather, it is a consortium of computing methodologies

which collectively provide a foundation for the conception, design and deployment of intelligent systems.
At this juncture, the principal members of soft computing are fuzzy logic, neurocomputing, genetic
computing, and probabilistic computing, with the last subsuming evidential reasoning, belief networks,
chaotic systems, and parts of machine learning theory. In contrast to traditional hard computing, soft
computing is tolerant of imprecision, uncertainty and partial truth. The guiding principle of soft computing
is: exploit the tolerance for imprecision, uncertainty and partial truth to achieve tractability, robustness,
low solution cost and better rapport with reality.
 L. Zadeh

ARTIFICIAL INTELLIGENCE

 In the 1992 Dictionary of Science and Technology published by Academic Press (Christopher Morris, Ed.,
San Diego, CA: Academic Press, page 160), Gordon S. Novak (then at the University of Texas) defines
artificial intelligence as:
 “the study of the computation required for intelligent behavior and the attempt to duplicate such
computation using computers. Intelligent behavior connects perception of the environment to action
appropriate for the goals of the actor. Intelligence, biologically costly in energy, pays for itself by
enhancing survival. It isn’t necessary to understand perfectly, but only to understand well enough to act
appropriately in real time.”
 Might substitute the word “processing” for “computation,” and say “using computers and other
systems,” but generally the definition is reasonable.

23
 2/25/2021

CONCLUSION

 Computational Intelligence provides success stories that are often hard to justify with formal
mathematical models (which are but a subset of all computational models, some of which are based on
mathematics, and some of which are not).
 Jim Bezdek 1992

RECOMMENDED READINGS

 Dellermann, Dominik, et al. "Hybrid intelligence." Business & Information Systems Engineering 61.5 (2019):
637-643.
 Kacprzyk, Janusz, and Witold Pedrycz, eds. Springer handbook of computational intelligence. Springer, 2015.

 Duch, Włodzisław. "Towards comprehensive foundations of computational intelligence." Challenges for

Computational Intelligence. Springer, Berlin, Heidelberg, 2007. 261-316.
 Duch, Włodzisław. "What is Computational Intelligence and where is it going?." Challenges for computational
intelligence. Springer, Berlin, Heidelberg, 2007. 1-13.

24
 2/25/2021

ASSIGNMENTS

1. What is the difference between fuzziness and probability? Provide an example to illustrate the difference.

2. What is the definition of artificial intelligence? List some differences between computational intelligence and
artificial intelligence.
3. Find an article or a chapter in another book on emergent computing. Compare the concept of emergent
computing as presented there with the concept of self-organization presented in this chapter.
4. Find another source of information on cellular automata. Discuss the relationship between cellular automata and
self-organization.

25