Neural Networks

Jorge Andreu Montañes#1

#
Jaume I University​
Castellón de la Plana (España)
1
[email protected]

Abstract— This document presents information related to the on historical data. We are going to mainly talk
laboratory practices on Neural Networks carried out in the
course EI1028-IR2128-MT1028 – Intelligent Systems (2024-2025) about classifying messages, images, signals, etc.,
at Jaume I University which is what we mostly deal with in the lab
practices.
I.​ INTRODUCTION II.​ PERCEPTRONS
An information technology system based on the A perceptron is an algorithm designed for
human brain´s structure that provides computers to supervised learning of binary classifiers (functions
be able to use artificial intelligence is known as a that determine whether an input (a vector of
neural network. numbers) belongs to a specific class). It serves as a
linear classifier, a classification algorithm that
Neural networks consist of at least two-layer makes predictions based on a linear predictor
models (input layer and output layer) and may also function that combines the input feature vector with
include additional intermediate layers known as the set of weights.
hidden layers. The perceptron algorithm was developed in the
late 1950s, and its first custom hardware became
The information reaches the neurons in the input one of the first artificial neural networks to be
layer as a signal and is processed there, each neuron created.
is assigned a numerical weight that represents its The perceptron was the first neural network
level of importance. Using the activation function capable of learning and only composed of input and
and the threshold value, the neuron´s output value is output neurons.
calculated and weighted. Based on the result, other In a perceptron, input neurons have two states:
neurons are linked. ON and OFF, and the output neurons use a simple
Through these connections and weights, the threshold activation function to produce a decision.
algorithm is designed to generate results for each In its simplest form, this type of algorithm can only
input, training is used to optimize the weights to solve linear problems.
improve the algorithm and achieve increasingly
accurate results.

One of the main reasons that neural networks are

used is their facility to learn, they figure out by
themselves how to perform their function with
sample inputs, another reason is their ability to
generalize, they can produce reasonable outputs for
inputs that they have not been taught to deal with.

Neural Networks are mainly used nowadays to

classify (i.e. recognize patterns, extract features,
compare pictures, etc.), to reduce noise recognizing Fig. 1 Perceptron

patterns in the inputs and produce noise-free

outputs, and to predict, making extrapolation based
 About perceptrons, in the labs, we have done
implementation exercises of the AND function, the
OR function and linearly separable classes, which
are basically about having two sets that are linearly
separable by a straight line. Now, we are going to
see the exercise of the linearly separable classes in
more detail.
During training, a perceptron adjusts its weights
(values that modify the inputs it receives) so that it
can correctly classify data points. Each neuron adds
up its weighted inputs and applies a threshold
function to determine an output. If the weighted Fig. 3 Incorrect classification
sum of the inputs exceeds a certain threshold, the
output will be 1; otherwise, it will remain 0. In the Initially, the decision boundary may not separate
linearly separable classes, a matrix is formed, the two sets correctly, as we can see in Fig. 3, there
containing the properties and coordinates of points are some points crossed by the line, so we must
labeled as either 0 (red) or 1 (white). The goal is to repeat the process several times. This process it’s
train the perceptron to create a decision boundary called “training”.
that correctly separates the two groups. To avoid having to repeat the process many times
over, in class we implemented a loop of 100
iterations that conclude when the sets are correctly
separated, i.e. Target and Prediction match.

Fig. 2 Sets are linearly separable

Using the perceptron learning rule, the weights of

the input connections are adjusted based on the
error between the actual output and expected output Fig. 4 Correct classification
to train the perceptrons. This loop of the process
continues until all of the training examples are Although perceptrons have limitations,
correctly classified by the perceptron. particularly in handling complex datasets, they are
still used as building blocks for more complex
neural network architectures, such as multi-layer
perceptrons and convolutional neural networks.
 III.​MULTILAYER PERCEPTRONS
Multilayer perceptrons are the most common
neural network, often considered an extension of
the perceptron, because unlike a simple perceptron,
it has multiple layers. Besides, the activation
function is not only a threshold, but also a sigmoid
function. Fig. 6 Test sets and training sets
The advantage of these neural networks is that it To check that the program is running correctly,
is not limited to linear problems, as the output of we need to verify it by providing a test of the
one layer acts as the input of the next layer. quality of the network.
A common mistake is to test the neural network
using the same samples that were used to train it.
Verifying the network in this way will obviously
make the network work well for me, due to it has
been optimized with the same samples. One way to
make sure we are testing well is to iterate the
training and testing procedures a few times.
A.​ XOR
XOR problem is a well-known non-linear
problem that a single perceptron cannot solve,
however Multilayer perceptrons can. The goal of
Fig. 5 Multilayer Perceptrons this code is to separate points of different colors
into different regions, i.e. the region of one color
The training of the neurons consists mostly of
corresponds to the features classified as 0 (black
minimizing the error function.
points), while the region of another color
A crucial aspect when designing a multilayer
corresponds to the features classified as 1 (red
network is determining the number of hidden layers
points).
and neurons. In most cases, a single hidden layer is
sufficient, while two layers may be necessary for
discontinuous functions. The number of neurons
must be carefully chosen; too few neurons prevent
the network from learning critical details, while too
many may cause it to overfit by memorizing
irrelevant patterns.
We also need to know that the set of all samples
is divided into two sets: the training set and the test
set. The first set contains the samples used to train
the neural network and the second set contains the
samples used to test the performance of the neural Fig. 7 Example of an image with acceptable resolution

network. For this, in the code we have used the 'sgd' solver,
one hidden layer with 5 neurons each, and 4000
iterations maximum. We have achieved this graph
through 2578 iterations, it is below the maximum
allowed and its score is 100% as you can see on
Fig. 7 that it divides the points correctly.
B.​ Classification
With 2500 samples we get a score of 92 going up
to 142 iterations, there is an improvement in the
The goal of this practice is to apply the multilayer score but also in the number of iterations.
perceptron, which performs the linear perceptron, to
a classification task. Specifically, we will load the
data, which will be classified; divide them into two
sets, the training set, which will help me to get the
neurons to learn how they work by themselves, and
the test set, which will determine whether I have
succeeded in getting the neurons to do their job.
In addition, we will have to select a predictive
model, which will be the multilayer perceptron.
Finally, after executing the code and see that it
works, we will be able to evaluate it.
Fig. 9 2500 data plot
For the generation of samples in this practice, we In the case of 25000 samples, we get a score of
have the possibility to choose between two datasets, 91,70 similar to 2500 samples but with less
we will choose the moons dataset, in order to not iterations, only 86.
make a very common mistake of testing the
program on the same data, we will separate them
randomly.
When creating the multilayer perceptron model to
divide the blue points from the red points by two
planes, we have used a hidden layer with 5 neurons
and a maximum of 4000 iterations like the previous
exercise. The fitting function will iterate
automatically until the convergence is achieved or
until the maximum number of iterations is reached.
In our case we did reach convergence with 276
iterations, obtaining a score of 89. This score is the Fig. 10 25000 data plot
percentage of correct classification of the test data. With these checks we can conclude that the larger
the amount of test data, the fewer iterations we need
to obtain a similar percentage of correct
classification.

C.​ Digits Recognition

This practice involves recognizing handwritten
digits using a dataset of 1,797 images. To apply a
data classifier, we first need to convert the images
into a sample matrix.
A crucial step in this process is determining
Fig. 8 250 data plot whether standardization is necessary. Without
standardizing the dataset, the neural network may
All the data has been obtained with 250 samples not function optimally, as the features may not
as test size, and I will test it for a data set of 2500 resemble a standard normal distribution (Gaussian
and 25000 to see the improvement. distribution with zero mean and unit variance. To
do this we use StandardScaler, which calculates the
mean and standard deviation from a training set in layer of 784 (28x28 pixels) while the output layer is
order to apply the same transformation again on the 10.
test set. We tested with CNN (Convolutional neural
In our case, in the code used in class we use the network) and “nadam” optimizer, and the results are
'sgd' solver, a hidden layer of 5 neurons each and in TABLE 1.
4000 as the maximum number of iterations.
The model successfully recognized handwritten TABLE I​
digits with an accuracy of 96%, which is a highly IMAGE CLASSIFIER KERAS TENSORFLOW CNN

satisfactory result, as we can see in Figure 11, and Optimizer Epochs Accuracy Acc Loss Acc
we can also observe the main classification metrics. val val test
nadam 20 0.976 0.928 0.426 0.926
“The precision is intuitively the ability of the
classifier not to label as positive a sample that is
negative.
The recall is intuitively the ability of the classifier
to find all the positive samples.
f1-score: a weighted average of the precision and
recall.
The support is the number of occurrences of each
class.” [1].
Fig. 12 Image Classifications

The Flatten layer is in charge of converting each

input image into a 2D matrix, and the Dense layer
initializes the connection weights randomly which
is crucial to break the symmetry.
We must compile the model first and then train it,
in this case we will call the model.compile() and
model.fit() functions, while in the models discussed
in the previous sections in most cases, we call the
net.fit() function.
As we observe in the results, the validation and
Fig. 11 Accuracy of the classification
test performance are not the same, and the latter is
lower, this is because the hyperparameters are
When running multiple tests, we observed adjusted only in the validation set, not in the test
variation in classification results. This occurs set.
because the dataset is split differently in each test.
To obtain more reliable results, we implemented a In the next notebook we enable GPUs, because
loop in which the network was tested 100 times Colab has by default configuration a CPU-only
with different test sets. runtime.
After this we run the program and we can see that
D.​ Image Sorter it is highly faster, also the precision and loss
In this section, we utilize Keras alongside decrease minimally, considering the difference in
TensorFlow, an open-source deep learning library time it takes to run, it is highly recommended to use
that simplifies the process of building, training, it to reduce the waiting time in this type of practice
evaluating, and executing various neural networks. when you use a considerable number of Epochs to
For this practice we classify images of clothes using assure the accuracy.
a dataset with 60000 grayscale images, and an input
E.​ Traffic Sign Classification .
In this practice we also have used Keras and
F.​ References
TensorFlow. What we do is based on multi-class
single-image classification to build a classifier that ●​ https://github.com/RobInLabUJI/Multilayer
is able to recognize the image of one of the many Perceptron/blob/main/Digits.ipynb [1]
traffic signals we have upload from GTSRB - The
German Traffic Sign Recognition Benchmark that IV.​CONCLUSIONS
contains more than 40 classes and more than 50,000 Neural networks are widely used today due to the
images of traffic signs, which are prepared for valuable assistance they provide. Although we may
multi-class, single-image classification challenge. not always notice them, they are an integral part of
our daily lives, for instance, in fingerprint or facial
recognition when unlocking a smartphone.
As discussed earlier, there are different types of
neural networks. While the perceptron is a simpler
and faster option for solving problems with linearly
separable functions, it becomes ineffective when
these conditions are not met. To face more complex
problems, we use multilayer perceptrons, as they
are capable of handling these problems.
ACKNOWLEDGMENT
The heading of the Acknowledgment section and
the References section must not be numbered.
Causal Productions wishes to acknowledge
Michael Shell and other contributors for developing
and maintaining the IEEE LaTeX style files which
have been used in the preparation of this template.
To see the list of contributors, please refer to the top
of file IEEETran.cls in the IEEE LaTeX
Fig. 13 Model Predictions distribution.
We have executed the test with 20 and 30 epochs. REFERENCES
The accuracy of the program with 20 epochs is [1]​ Equipo editorial de IONOS, “Neural networks: ¿de qué son capaces
las redes neuronales artificiales?”, 2020.
about 0.695 with a 1.291 of loss, and with 30 [2]​ https://aulavirtual.uji.es/pluginfile.php/7132030/mod_resource/content/
epochs the accuracy is about 0.689 with a 1.473 of 6/Various%20NN%202025.pdf
[3]​ https://github.com/RobInLabUJI/Perceptron/blob/main/AND.ipynb
loss. [4]​ https://github.com/RobInLabUJI/Perceptron/blob/main/OR.ipynb
We have also changed the number of neurons per [5]​ https://github.com/RobInLabUJI/Perceptron/blob/main/Classes.ipynb
layer and the results are quite similar. [6]​ https://github.com/RobInLabUJI/MultilayerPerceptron/blob/main/XOR
.ipynb
As we can see from the results, the greater [7]​ https://github.com/RobInLabUJI/MultilayerPerceptron/blob/main/Class
number of epochs used does not necessarily mean ification.ipynb
[8]​ https://github.com/RobInLabUJI/MultilayerPerceptron/blob/main/Digit
that the accuracy of the program will be better, the s.ipynb
accuracies are quite similar and there is not a great [9]​ https://github.com/RobInLabUJI/MultilayerPerceptron/blob/main/Imag
e_Classifier_Keras_TF.ipynb
difference in the results. [10]​ https://github.com/RobInLabUJI/MultilayerPerceptron/blob/main/Imag
e_Classifier_Keras_TF_CNN.ipynb
[11]​ https://github.com/RobInLabUJI/MultilayerPerceptron/blob/main/Imag
As for the loss of accuracy we come to the same e_Classifier_Keras_TF_CNN_GPU.ipynb
conclusion, increasing the number of epochs will [12]​ https://github.com/RobInLabUJI/MultilayerPerceptron/blob/main/Traff
ic_Sign_Classification_Training.ipynb
not give us a less loss of accuracy. [13]​ https://github.com/RobInLabUJI/MultilayerPerceptron/blob/main/Traff
ic_Sign_Classification_Evaluation.ipynb