MECE 694 M.

Nazarahari

MECE 694: Applied Computational Intelligence for Engineers

Assignment 3
Due Date: See course syllabus
Instructor: Milad Nazarahari ([email protected])

1. Guidelines:
• Submit the finalized codes (.mat file) as well as your report (.pdf file), including results and
engineering analysis, as a zip file on eClass.
• Include the assignment rubric on the first page of your report.
• You don’t need to include the finalized codes in your report. Only include the qualitative
answers or parameter investigation results.
• Your report must be produced with “Overleaf” using this template.
• Remark: Failing to comply with the above requirements will result in an assignment mark
reduction.

Assignment 3 has three sections:

1. Implementing a Back-Propagation Algorithm to train a Multi-layer Perceptron.
2. Implementing a Kernel Support Vector Machine for Classification with a Nonlinear
Decision Boundary.
3. Implement Automatic Clustering Using a single-objective Genetic Algorithm.
For each section, sample codes are provided for you on eClass, and you need to complete/modify
the codes to obtain the required results. See the posted videos for help with your assignment.

2. Multi-layer Perceptron Training with Backpropagation Algorithm

At this point, you should be able to use libraries and GUIs included in MATLAB to solve classification
or function approximation problems using Artificial Neural Networks; otherwise, please see the
posted video.

You are going to implement the backpropagation algorithm to train an MLP with a simple architecture
(input layer, one hidden layer, and output layer) to solve a two-class classification problem and then
compare your trained network with the one that can be obtained using MATLAB libraries.

The provided function named “Classification_ANN_Incomplete” contains all the codes and
functions (which you need to complete) to do this section of the assignment.
• Section 1: Creates a random dataset for a two-class classification problem. Variable “x” will
contain the data, and variable “t_org” will contain the true targets (true classes).
• Section 2: Contains the codes to create an MLP using MATLAB libraries, tests the trained
network using all input data, and plots the confusion matrix.

Page 1 of 6
MECE 694 M. Nazarahari

o “t_org contains” 1s for Class 1 and -1s for Class 2. Yet, to plot a confusion matrix
using the “plotconfusion” function in MATLAB, true targets and network outputs “y”
must have the following format:

Question 1 (Section 3 of “Classification_ANN_Incomplete”)

• Section 3: contains all the codes for performing backpropagation training for our simple
network. The architecture of the network is: input-layer, one hidden-layer, and output-layer.
You can set the number of neurons in the hidden-layer using the parameter
“hiddenLayerSize”.
• Follow the backpropagation procedure in your lecture notes and comments in the code to
better understand what you need to do in this section.
• (code) In STEP 1 of the code, you must complete the function
“RandomInitialWeights_Incomplete” which generates random initial weights for MLP. This
function receives the number of incoming “In” and outgoing “Out” connections in a layer
and creates a random weight matrix with a size of Out-by-(In+1).
o Note that this function is called twice in Section 3 to create initial weights for
connections between (1) input-and-hidden layers and (2) hidden-and-output layers.
• In STEP 2, the code will unroll network weights (from two matrices to a vector) to make it
suitable for optimization.
• (code) In STEP 3, you must complete the function “MyANNObjecFunction_Incomplete”
which:
o PHASE 0: Rolls the network weights from a vector to two matrices (already done for
you).
o PHASE 1: uses the network weights to perform forward propagation and compute
the network targets using provided inputs and estimated weights. Then, uses the
obtained targets by MLP and true targets to calculate the regularized objective function
value. The objective value is the first output of “MyANNObjecFunction_Incomplete”
(complete the codes according to the provided guides and lecture notes).
o PHASE 3: performs backpropagation to calculate the gradients, which can be used by
an optimization algorithm such as “fminunc” to correct the weights of each layer. The
gradient values unrolled together are the second output of function
“MyANNObjecFunction_Incomplete” (complete the codes according to the
provided guides and lecture notes).
o For your implementation, consider the following case for activation functions: Hidden
layer: Sigmoid; output layer: Sigmoid.

Page 2 of 6
 MECE 694 M. Nazarahari

• STEP 4: Use “fminunc” to find the set of network weights which minimizes objective function
value. Use “MyANNObjecFunction_Incomplete” to create the function handle for
“fminunc” (already done for you).
• STEP 5 and 6: roll the optimized network weights and predict the network targets using the
function “PredictMLPOutputs”. Finally, plot the confusion matrix (already done for you).

Notes:
• Your code only works for a two-class classification problem and a network with an input-
layer, one hidden-layer, and output-layer architecture.
• We used all the data to perform training (no validation). Thus, the percentages of validation
and test data in MATLAB MLP were set to 0.
• We used all the data to test our networks after training (Resubstitution Method).

Question 2
• (report) Compare the performance (accuracy from confusion plot) of your trained MLP with
MATLAB implementation of MLP by filling the following table:

Number of artificial neurons in the hidden layer 2 5 10 15 20

Your MLP with “Sigmoid” activation
MATLAB MLP (with trainlm)
MATLAB MLP (with trainbr)
MATLAB MLP (with trainscg)

• To make sure that your results are reliable, for each scenario (e.g., your MLP with 5 neurons
in hidden layer), execute the “Classification_ANN_Incomplete” 10 times and report the
average accuracy ± standard deviation (each execution generates a random dataset and
random initial weights for MLP).
• Briefly describe (qualitatively) the differences between “trainlm”, “trainbr”, and “trainscg” and
why one might achieve superior outcome compared to others (max 3 sentences for each).
• Discuss any performance difference observed between the best results of your MLP and
MATLAB MLP.
• Qualitatively describe (briefly - max 3 sentences) what changes are needed in the code if you
want to implement MLP with any desired number of hidden layers.
• Qualitatively describe (briefly - max 3 sentences) what changes are needed in the code if you
want to solve a multi-class classification problem.

Page 3 of 6
MECE 694 M. Nazarahari

3. Kernel Support Vector Machine

You are going to solve a two-class classification problem using KSVM. You should use the provided
code called “Classification_KSVM_Incomplete” for this purpose.
• Section 1 of “Classification_KSVM_Incomplete” creates the two-class classification problem
including matrix “x” which contains the data and vector “t_org” which contains the class
labels (1 for Class 1 and -1 for Class 2).
• Section 2 of “Classification_KSVM_Incomplete” uses MATLAB implementation of KSVM
using the “fitcsvm” function to solve the classification problem. Also, the confusion matrix
will be plotted to show the accuracy of the MATLAB KSVM (after changing the KSVM target
and true target to the proper format).
• Section 3 of “Classification_KSVM_Incomplete” contains an incomplete implementation of
KSVM. Your task is to complete this section to solve the same two-class classification
problem, plot the non-linear decision boundary, and then compare results with MATLAB
KSVM. To complete the KSVM implementation, follow the procedure in the lecture notes
and guides provided for you in Section 3.

Question 3 (Section 3 of “Classification_KSVM_Incomplete”)

• (code) Complete function “KernelFunction_Incomplete”. This function receives a structure
named “Kernel” including kernel name (Kernel.Type) and kernel parameter
(Kernel.Parameter) and must return the function handle which can be later used to calculate
the kernel value for two observations (xi and xj). You must complete the code for Gaussian,
Logistic, and Hyperbolic Tangent kernels using their definitions in your lecture notes (Linear
kernel has already been implemented as an example for you).
• (code) Complete the required codes in Section 3 of the “Classification_KSVM_Incomplete”
based on the procedure in your lecture notes to perform operations of KSVM. You can use
the provided “MySumFunc” wherever needed (see the operation of “MySumFunc” inside the
function).
• At this point, you will be able to run Section 3 and solve the two-class classification problem
with your KSVM implementation. Also, you will see the nonlinear decision boundary and the
confusion matrix.

Question 4
• (report) Compare the performance (accuracy obtained from confusion matrix) of your
implemented KSVM (Section 3) with MATLAB implementation of KSVM (Section 2) for
different kernel functions and for three random problems in the table below:
o To create a random problem, run Section 1 of “Classification_KSVM_Incomplete”.
Then, to get the results for each column (one problem), run Sections 2 and 3 with
proper settings and record accuracy from the confusion matrix. You are also
encouraged to add a few (!) figures of data and decision boundaries for your KSVM.

Page 4 of 6
MECE 694 M. Nazarahari

o To improve your KSVM performance, try various parameter values for each kernel
for a number of random problems before filling the above table, and then use the
selected parameter values to perform a comparison (report the selected parameters).

Random Problem 1 2 3
Your KSVM (Gaussian, Parameter = ?)
Your KSVM (Logistic, Parameter = ?)
Your KSVM (Hyperbolic Tangent, Parameter = ?)
MATLAB KSVM (Gaussian)
MATLAB KSVM (polynomial)

• Discuss the reasons for any performance difference between different Kernel functions for
your implementation of KSVM.
• For Gaussian Kernel, show the effect of Kernel parameters by varying the parameter in a
range (your choice) and reporting KSVM performance. Discuss your observations.

4. Evolutionary Clustering Using Genetic Algorithm (Optional – Will not be Marked)

You are going to implement automatic clustering using GA to identify the number of clusters and
then perform clustering. You can use the provided code called “GeneticAlgorithm_Incomplete” for
this purpose.
• Section 1 of “GeneticAlgorithm_Incomplete” loads one of the given data sets (Model_1: 3
clusters/easy, Model_2: 3 clusters/difficult, Model_3: 5 clusters/easy, Model_4: 5
clusters/difficult) and then plots the data (each colour shows one cluster).
• Section 2 of “GeneticAlgorithm_Incomplete” first creates the objective function for clustering
using the function named “ClusteringObjValue”. You can use (1) within class distance (WCD)
or (2) DB Index (DB_Index) as the clustering objective (already implemented for you).

Question 5 (Sections 2, 3, 4, and 5 of the function named “GeneticAlgorithm_Incomplete”)

• (code) Similar to the provided simple evolutionary clustering algorithm (when number of
clusters is known), complete Sections 2, 3, 4, and 5 to perform automatic clustering using GA
(we start with a large number for number of clusters and GA determines how many clusters
to use).
o The only differences between simple and automatic clustering are (1) coding of the
solution and (2) calculation of objective value. The objective function is already
implemented for you (“ClusteringObjValue”). Thus, the only modification in
automatic clustering compared to simple GA is the solution coding. In other words,
instead of using a k-by-2 matrix (k is the number of clusters) as a solution (each row
contains the coordinates x and y of a cluster center), you must use a k-by-3 matrix,

Page 5 of 6
MECE 694 M. Nazarahari

where the first two columns of each row contain the coordinates x and y of a cluster
center, and the third column shows the activation-value for that cluster center. When
the activation value is higher than 0.5, that cluster center is activated and data will be
assigned to that. On the other hand, when the activation value is less than 0.5, the
automatic clustering with GA will ignore that cluster center.
o All other steps of the GA in automatic and simple clustering approaches are the same.
Thus, you can use the same procedure for sorting solutions and crossover/mutation
operations.
• (report) Compare the performance of (1) within a class distance (WCD) and (2) DB Index for
four datasets and qualitatively/quantitively discuss the reason(s) for observed differences. As
the objective values are not comparable directly, you can present the number of correctly
clustered data points by comparing the GA result with true cluster labels in variable “t” (for
each case, you might execute the GA a few times and report only one which seems better to
you).

Page 6 of 6