Faculty of : ……… Program:……….

Class/Section: ……………… Date: …………

Name of Faculty: ……………………. Name of Course: …………… Code: ……………

Neural Network
A neural network is a machine learning program, or model, that makes decisions in a manner
similar to the human brain, by using processes that mimic the way biological neurons work
together to identify phenomena, weigh options and arrive at conclusions.
Every neural network consists of layers of nodes, or artificial neurons—an input layer, one or
more hidden layers, and an output layer. Each node connects to others, and has its own
associated weight and threshold. If the output of any individual node is above the specified
threshold value, that node is activated, sending data to the next layer of the network. Otherwise,
no data is passed along to the next layer of the network.

Neural networks rely on training data to learn and improve their accuracy over time. Once they
are fine-tuned for accuracy, they are powerful tools in computer science and artificial
intelligence, allowing us to classify and cluster data at a high velocity. Tasks in speech
recognition or image recognition can take minutes versus hours when compared to the manual
identification by human experts. One of the best-known examples of a neural network is
Google’s search algorithm.
Neural networks are sometimes called artificial neural networks (ANNs) or simulated neural
networks (SNNs). They are a subset of machine learning, and at the heart of deep
learning models.

How do neural networks work?

Think of each individual node as its own linear regression model, composed of input data,
weights, a bias (or threshold), and an output. The formula would look something like this:
∑wixi + bias = w1x1 + w2x2 + w3x3 + bias
output = f(x) = 1 if ∑w1x1 + b>= 0; 0 if ∑w1x1 + b < 0

Once an input layer is determined, weights are assigned. These weights help determine the
importance of any given variable, with larger ones contributing more significantly to the output
compared to other inputs. All inputs are then multiplied by their respective weights and then
summed. Afterward, the output is passed through an activation function, which determines the
output. If that output exceeds a given threshold, it “fires” (or activates) the node, passing data to
the next layer in the network. This results in the output of one node becoming in the input of the
next node. This process of passing data from one layer to the next layer defines this neural
network as a feedforward network.
Let’s break down what one single node might look like using binary values. We can apply this
concept to a more tangible example, like whether you should go surfing (Yes: 1, No: 0). The
Session 2024-25
decision to go or not to go is our predicted outcome, or y-hat. Let’s assume that there are three
factors influencing your decision-making:

1. Are the waves good? (Yes: 1, No: 0)

2. Is the line-up empty? (Yes: 1, No: 0)
3. Has there been a recent shark attack? (Yes: 0, No: 1)

Then, let’s assume the following, giving us the following inputs:

 X1 = 1, since the waves are pumping
 X2 = 0, since the crowds are out
 X3 = 1, since there hasn’t been a recent shark attack

Now, we need to assign some weights to determine importance. Larger weights signify that
particular variables are of greater importance to the decision or outcome.
 W1 = 5, since large swells don’t come around often
 W2 = 2, since you’re used to the crowds
 W3 = 4, since you have a fear of sharks

Finally, we’ll also assume a threshold value of 3, which would translate to a bias value of –3.
With all the various inputs, we can start to plug in values into the formula to get the desired
output.

Y-hat = (1*5) + (0*2) + (1*4) – 3 = 6

If we use the activation function from the beginning of this section, we can determine that the
output of this node would be 1, since 6 is greater than 0. In this instance, you would go surfing;
but if we adjust the weights or the threshold, we can achieve different outcomes from the
model. When we observe one decision, like in the above example, we can see how a neural
network could make increasingly complex decisions depending on the output of previous
decisions or layers.

In the example above, we used perceptrons to illustrate some of the mathematics at play here,
but neural networks leverage sigmoid neurons, which are distinguished by having values
between 0 and 1. Since neural networks behave similarly to decision trees, cascading data from
one node to another, having x values between 0 and 1 will reduce the impact of any given
change of a single variable on the output of any given node, and subsequently, the output of the
neural network.
As we start to think about more practical use cases for neural networks, like image recognition
or classification, we’ll leverage supervised learning, or labeled datasets, to train the algorithm.
As we train the model, we’ll want to evaluate its accuracy using a cost (or loss) function. This is
also commonly referred to as the mean squared error (MSE). In the equation below,
 i represents the index of the sample,
 y-hat is the predicted outcome,
 y is the actual value, and
 m is the number of samples.

𝐶𝑜𝑠𝑡 𝐹𝑢𝑛𝑐𝑡𝑖𝑜𝑛= 𝑀𝑆𝐸=1/2𝑚 ∑129_(𝑖=1)^𝑚▒(𝑦 ̂^((𝑖) )−𝑦^((𝑖) ) )^2

Ultimately, the goal is to minimize our cost function to ensure correctness of fit for any given
observation. As the model adjusts its weights and bias, it uses the cost function and
reinforcement learning to reach the point of convergence, or the local minimum. The process in
which the algorithm adjusts its weights is through gradient descent, allowing the model to
Session 2024-25
determine the direction to take to reduce errors (or minimize the cost function). With each
training example, the parameters of the model adjust to gradually converge at the minimum.

Most deep neural networks are feedforward, meaning they flow in one direction only, from
input to output. However, you can also train your model through backpropagation; that is, move
in the opposite direction from output to input. Backpropagation allows us to calculate and
attribute the error associated with each neuron, allowing us to adjust and fit the parameters of
the model(s) appropriately.

Types of neural networks

Neural networks can be classified into different types, which are used for different purposes.
While this isn’t a comprehensive list of types, the below would be representative of the most
common types of neural networks that you’ll come across for its common use cases:

The perceptron is the oldest neural network, created by Frank Rosenblatt in 1958.

Feedforward neural networks, or multi-layer perceptrons (MLPs), are what we’ve primarily
been focusing on within this article. They are comprised of an input layer, a hidden layer or
layers, and an output layer. While these neural networks are also commonly referred to as
MLPs, it’s important to note that they are actually comprised of sigmoid neurons, not
perceptrons, as most real-world problems are nonlinear. Data usually is fed into these models to
train them, and they are the foundation for computer vision, natural language processing, and
other neural networks.
Convolutional neural networks (CNNs) are similar to feedforward networks, but they’re usually
utilized for image recognition, pattern recognition, and/or computer vision. These networks
harness principles from linear algebra, particularly matrix multiplication, to identify patterns
within an image.
Recurrent neural networks (RNNs) are identified by their feedback loops. These learning
algorithms are primarily leveraged when using time-series data to make predictions about future
outcomes, such as stock market predictions or sales forecasting.

Session 2024-25
Neural networks vs. deep learning
Deep Learning and neural networks tend to be used interchangeably in conversation, which can
be confusing. As a result, it’s worth noting that the “deep” in deep learning is just referring to
the depth of layers in a neural network. A neural network that consists of more than three layers
—which would be inclusive of the inputs and the output—can be considered a deep learning
algorithm. A neural network that only has two or three layers is just a basic neural network.

History of neural networks

The history of neural networks is longer than most people think. While the idea of “a machine
that thinks” can be traced to the Ancient Greeks, we’ll focus on the key events that led to the
evolution of thinking around neural networks, which has ebbed and flowed in popularity over
the years:

Session 2024-25
1943: Warren S. McCulloch and Walter Pitts published “A logical calculus of the ideas
immanent in nervous activity (link resides outside ibm.com)” This research sought to
understand how the human brain could produce complex patterns through connected brain cells,
or neurons. One of the main ideas that came out of this work was the comparison of neurons
with a binary threshold to Boolean logic (i.e., 0/1 or true/false statements).

1958: Frank Rosenblatt is credited with the development of the perceptron, documented in his
research, “The Perceptron: A Probabilistic Model for Information Storage and Organization in
the Brain” (link resides outside ibm.com). He takes McCulloch and Pitt’s work a step further by
introducing weights to the equation. Leveraging an IBM 704, Rosenblatt was able to get a
computer to learn how to distinguish cards marked on the left vs. cards marked on the right.

1974: While numerous researchers contributed to the idea of backpropagation, Paul Werbos
was the first person in the US to note its application within neural networks within his PhD
thesis (link resides outside ibm.com).

1989: Yann LeCun published a paper (link resides outside ibm.com) illustrating how the use of
constraints in backpropagation and its integration into the neural network architecture can be
used to train algorithms. This research successfully leveraged a neural network to recognize
hand-written zip code digits provided by the U.S. Postal Service.

Advantages of Neural Networks

Neural networks are widely used in many different applications because of their many benefits:
 Adaptability: Neural networks are useful for activities where the link between inputs
and outputs is complex or not well defined because they can adapt to new situations and
learn from data.
 Pattern Recognition: Their proficiency in pattern recognition renders them efficacious in
tasks like as audio and image identification, natural language processing, and other
intricate data patterns.
 Parallel Processing: Because neural networks are capable of parallel processing by
nature, they can process numerous jobs at once, which speeds up and improves the
efficiency of computations.
 Non-Linearity: Neural networks are able to model and comprehend complicated
relationships in data by virtue of the non-linear activation functions found in neurons,
which overcome the drawbacks of linear models.

Disadvantages of Neural Networks

Neural networks, while powerful, are not without drawbacks and difficulties:
 Computational Intensity: Large neural network training can be a laborious and
computationally demanding process that demands a lot of computing power.
 Black box Nature: As “black box” models, neural networks pose a problem in important
applications since it is difficult to understand how they make decisions.
 Overfitting: Overfitting is a phenomenon in which neural networks commit training
material to memory rather than identifying patterns in the data. Although regularization
approaches help to alleviate this, the problem still exists.
 Need for Large datasets: For efficient training, neural networks frequently need sizable,
labeled datasets; otherwise, their performance may suffer from incomplete or skewed
data.

Session 2024-25
Session 2024-25
Session 2024-25