Introduction to Data

Science (CCPS521)

Session 6:
Neural Networks
 Neural Networks
Neural Networks:
The term neural networks historically refers to networks of neurons in the mammalian brain. Neurons are its
fundamental units of computation, and they are connected together in networks to process data. This can be a very
complex task. The dynamics of such neural networks in response to external stimuli is therefore often quite
intricate. Inputs and outputs of each neuron vary as functions of time in the form of spike trains, but the network
itself also changes over time: we learn and improve our data-processing capabilities by establishing new
connections between neurons.
https://www-cambridge-org.ezproxy.lib.torontomu.ca/core/services/aop-cambridge-
core/content/view/333E2FDFA4595FCACCF1558F7C0B90F3/9781108494939c1_1-12.pdf/introduction.pdf

Source (Active login to my.torontonmu.ca is required): Mehlig,

B. (2022). Machine learning with
neural networks: An introduction for scientists and engineers. Cambridge University
Press.
https://www-cambridge-org.ezproxy.lib.torontomu.ca/core/books/machine-learning-with-neural-
networks/0028D1883AF842C81340508119AB6491#fndtn-contents
 Neural Networks
Neural Networks:
Neural-network algorithms for machine learning are inspired by the architecture and the dynamics of networks of neurons
in the brain. The algorithms use highly idealized neuron models. Nevertheless, the fundamental principle is the same:
artificial neural networks learn by changing the connections between their neurons. Such networks can perform a
multitude of information-processing tasks.
Neural networks can for instance learn to recognise structures in a data set and, to some extent, generalise what they have
learnt. A training set contains a list of input patterns together with a list of corresponding labels, or target values, that
encode the properties of the input patterns the network is supposed to learn. Artificial neural networks can be trained to
classify such data very accurately by adjusting the connection strengths between their neurons, and can learn to
generalise the result to other data sets – provided that the new data is not too different from the training data.
https://www-cambridge-org.ezproxy.lib.torontomu.ca/core/services/aop-cambridge-
core/content/view/333E2FDFA4595FCACCF1558F7C0B90F3/9781108494939c1_1-12.pdf/introduction.pdf

Source (Active login to my.torontonmu.ca is required): Mehlig,

B. (2022). Machine learning with neural
networks: An introduction for scientists and engineers. Cambridge University Press.
https://www-cambridge-org.ezproxy.lib.torontomu.ca/core/books/machine-learning-with-neural-
networks/0028D1883AF842C81340508119AB6491#fndtn-contents
 Neural Networks
Neural Networks:
A prime example for a problem of this type is object recognition in images, for instance in the sequence of camera images
taken by a self-driving car. Recent interest in machine learning with neural networks is driven in part by the success of
neural networks in visual object recognition.
Another task at which neural networks excel is machine translation with dynamical or recurrent networks. Such networks
take sentences as inputs. As one feeds word after word, the network outputs the words in the translated sentence.
Recurrent networks can be efficiently trained on large training sets of input sentences and their translations. Google
translate works in this way.
These are all examples of supervised learning, where the networks are trained to associate a certain target, or label, with
each input.

https://www-cambridge-org.ezproxy.lib.torontomu.ca/core/services/aop-cambridge-
core/content/view/333E2FDFA4595FCACCF1558F7C0B90F3/9781108494939c1_1-12.pdf/introduction.pdf
Source (Active login to my.torontonmu.ca is required): Mehlig,
B. (2022). Machine learning with neural
networks: An introduction for scientists and engineers. Cambridge University Press.
https://www-cambridge-org.ezproxy.lib.torontomu.ca/core/books/machine-learning-with-neural-
networks/0028D1883AF842C81340508119AB6491#fndtn-contents
 Neural Networks
What is a neural network?
Neural networks, also known as artificial neural networks (ANNs) or simulated
neural networks (SNNs), are a subset of machine learning and are at the heart
of deep learning algorithms. Their name and structure are inspired by the human
brain, mimicking the way that biological neurons signal to one another.
Artificial neural networks (ANNs) are comprised of a node layers, containing an
input layer, one or more hidden layers, and an output layer. Each node, or artificial
neuron, connects to another and has an associated weight and threshold. If the
output of any individual node is above the specified threshold value, that node is
activated, sending data to the next layer of the network. Otherwise, no data is
passed along to the next layer of the network.
Source: https://www.ibm.com/topics/neural-networks
 Neural Networks
What is a neural network?
Neural networks rely on training data to learn and improve their accuracy over
time. However, once these learning algorithms are fine-tuned for accuracy, they
are powerful tools in computer science and artificial intelligence, allowing us to
classify and cluster data at a high velocity. Tasks in speech recognition or image
recognition can take minutes versus hours when compared to the manual
identification by human experts. One of the most well-known neural networks is
Google’s search algorithm.

Source: https://www.ibm.com/topics/neural-networks
 Neural Networks
Why Neural Networks are important?
Neural networks can help computers make intelligent decisions with limited human
assistance. This is because they can learn and model the relationships between input and
output data that are nonlinear and complex. For instance, they can do the following
tasks.
• Make generalizations and inferences
• Neural networks can comprehend unstructured data and make general observations
without explicit training. For instance, they can recognize that two different input
sentences have a similar meaning:
Can you tell me how to make the payment?
How do I transfer money?
• A neural network would know that both sentences mean the same thing. Or it would be
able to broadly recognize that Baxter Road is a place, but Baxter Smith is a person’s
name.

Source: https://aws.amazon.com/what-is/neural-network/
 Neural Networks
What are Neural Networks used for?
Neural networks have several use cases across many industries, such as the
following:
• Medical diagnosis by medical image classification
• Targeted marketing by social network filtering and behavioral data analysis
• Financial predictions by processing historical data of financial instruments
• Electrical load and energy demand forecasting
• Process and quality control
• Chemical compound identification

Source: https://aws.amazon.com/what-is/neural-network/
 Neural Networks
What are Neural Networks used for?
We give four of the important applications of neural networks below.
Computer vision: Computer vision is the ability of computers to extract information and
insights from images and videos. With neural networks, computers can distinguish and
recognize images similar to humans. Computer vision has several applications, such as
the following:
• Visual recognition in self-driving cars so they can recognize road signs and other road
users
• Content moderation to automatically remove unsafe or inappropriate content from
image and video archives
• Facial recognition to identify faces and recognize attributes like open eyes, glasses, and
facial hair
• Image labeling to identify brand logos, clothing, safety gear, and other image details

Source: https://aws.amazon.com/what-is/neural-network/
 Neural Networks
What are Neural Networks used for?
Speech recognition: Neural networks can analyze human speech despite varying
speech patterns, pitch, tone, language, and accent. Virtual assistants like Amazon
Alexa and automatic transcription software use speech recognition to do tasks like
these:
• Assist call center agents and automatically classify calls
• Convert clinical conversations into documentation in real time
• Accurately subtitle videos and meeting recordings for wider content reach

Source: https://aws.amazon.com/what-is/neural-network/
 Neural Networks
What are Neural Networks used for?
Natural language processing: Natural language processing (NLP) is the ability to
process natural, human-created text. Neural networks help computers gather
insights and meaning from text data and documents. NLP has several use cases,
including in these functions:
• Automated virtual agents and chat-bots
• Automatic organization and classification of written data
• Business intelligence analysis of long-form documents like emails and forms
• Indexing of key phrases that indicate sentiment, like positive and negative
comments on social media
• Document summarization and article generation for a given topic

Source: https://aws.amazon.com/what-is/neural-network/
 Neural Networks
What are Neural Networks used for?
Recommendation engines
Neural networks can track user activity to develop personalized recommendations.
They can also analyze all user behavior and discover new products or services that
interest a specific user. For example, Curalate, a Philadelphia-based startup, helps
brands convert social media posts into sales. Brands use Curalate’s intelligent
product tagging (IPT) service to automate the collection and curation of user-
generated social content. IPT uses neural networks to automatically find and
recommend products relevant to the user’s social media activity. Consumers don't
have to hunt through online catalogs to find a specific product from a social media
image. Instead, they can use Curalate’s auto product tagging to purchase the
product with ease.

Source: https://aws.amazon.com/what-is/neural-network/
 Neural Networks
Simple neural network architecture: A basic neural network has interconnected artificial
neurons in three layers:
Input Layer: Information from the outside world enters the artificial neural network from
the input layer. Input nodes process the data, analyze or categorize it, and pass it on to
the next layer.
Hidden Layer: Hidden layers take their input from the input layer or other hidden layers.
Artificial neural networks can have a large number of hidden layers. Each hidden layer
analyzes the output from the previous layer, processes it further, and passes it on to the
next layer.
Output Layer: The output layer gives the final result of all the data processing by the
artificial neural network. It can have single or multiple nodes. For instance, if we have a
binary (yes/no) classification problem, the output layer will have one output node, which
will give the result as 1 or 0. However, if we have a multi-class classification problem, the
output layer might consist of more than one output node.

Source: https://aws.amazon.com/what-is/neural-network/
 Neural Networks
Deep neural network architecture
Deep neural networks, or deep learning networks, have several hidden layers with
millions of artificial neurons linked together. A number, called weight, represents
the connections between one node and another. The weight is a positive number if
one node excites another, or negative if one node suppresses the other. Nodes
with higher weight values have more influence on the other nodes.

Theoretically, deep neural networks can map any input type to any output type.
However, they also need much more training as compared to other machine
learning methods. They need millions of examples of training data rather than
perhaps the hundreds or thousands that a simpler network might need.

Source: https://aws.amazon.com/what-is/neural-network/
 Neural Networks
What are the types of neural networks?
Artificial neural networks can be categorized by how the data flows from the input node to the
output node. Below are some examples:
Feedforward neural networks: Feedforward neural networks process data in one direction, from
the input node to the output node. Every node in one layer is connected to every node in the
next layer. A feedforward network uses a feedback process to improve predictions over time.
Backpropagation algorithm: Artificial neural networks learn continuously by using corrective
feedback loops to improve their predictive analytics. In simple terms, you can think of the data
flowing from the input node to the output node through many different paths in the neural
network. Only one path is the correct one that maps the input node to the correct output node.
To find this path, the neural network uses a feedback loop, which works as follows:
• Each node makes a guess about the next node in the path.
• It checks if the guess was correct. Nodes assign higher weight values to paths that lead to more
correct guesses and lower weight values to node paths that lead to incorrect guesses.
• For the next data point, the nodes make a new prediction using the higher weight paths and
then repeat Step 1.

Source: https://aws.amazon.com/what-is/neural-network/
 Neural Networks
What are the types of neural networks?
Convolutional neural networks: The hidden layers in convolutional neural
networks perform specific mathematical functions, like summarizing or filtering,
called convolutions. They are very useful for image classification because they can
extract relevant features from images that are useful for image recognition and
classification. The new form is easier to process without losing features that are
critical for making a good prediction. Each hidden layer extracts and processes
different image features, like edges, color, and depth.

Source: https://aws.amazon.com/what-is/neural-network/
 Neural Networks
Types of neural networks:
Neural networks can be classified into different types, which are used for different
purposes. While this isn’t a comprehensive list of types, the below would be
representative of the most common types of neural networks that you’ll come across for
its common use cases:
The perceptron is the oldest neural network, created by Frank Rosenblatt in 1958.
Feedforward neural networks, or multi-layer perceptrons (MLPs), are what we’ve
primarily been focusing on within this article. They are comprised of an input layer, a
hidden layer or layers, and an output layer. While these neural networks are also
commonly referred to as MLPs, it’s important to note that they are actually comprised of
sigmoid neurons, not perceptrons, as most real-world problems are nonlinear. Data
usually is fed into these models to train them, and they are the foundation for computer
vision, natural language processing, and other neural networks.
Source: https://www.ibm.com/topics/neural-networks
 Neural Networks
Types of neural networks:
Convolutional neural networks (CNNs) are similar to feedforward networks, but
they’re usually utilized for image recognition, pattern recognition, and/or
computer vision. These networks harness principles from linear algebra,
particularly matrix multiplication, to identify patterns within an image.
Recurrent neural networks (RNNs) are identified by their feedback loops. These
learning algorithms are primarily leveraged when using time-series data to make
predictions about future outcomes, such as stock market predictions or sales
forecasting.

Source: https://www.ibm.com/topics/neural-networks
 Neural Networks
What is Deep Learning in the context of Neural Networks?
Artificial intelligence is the field of computer science that researches methods of
giving machines the ability to perform tasks that require human intelligence.
Machine learning is an artificial intelligence technique that gives computers access
to very large datasets and teaches them to learn from this data.
Machine learning software finds patterns in existing data and applies those
patterns to new data to make intelligent decisions.
Deep learning is a subset of machine learning that uses deep learning networks to
process data.

Source: https://aws.amazon.com/what-is/neural-network/
 Neural Networks
How do neural networks work?
Think of each individual node as its own linear regression model, composed of
input data, weights, a bias (or threshold), and an output. The formula would look
something like this:

Once an input layer is determined, weights are assigned. These weights help
determine the importance of any given variable, with larger ones contributing
more significantly to the output compared to other inputs. All inputs are then
multiplied by their respective weights and then summed.
Source: https://www.ibm.com/topics/neural-networks
 Neural Networks
How do neural networks work?
Afterward, the output is passed through an activation function, which determines
the output. If that output exceeds a given threshold, it “fires” (or activates) the
node, passing data to the next layer in the network. This results in the output of
one node becoming in the input of the next node. This process of passing data
from one layer to the next layer defines this neural network as a feedforward
network.
Let’s break down what one single node might look like using binary values. We can
apply this concept to a more tangible example, like whether you should go surfing
(Yes: 1, No: 0). The decision to go or not to go is our predicted outcome, or y-hat.
Let’s assume that there are three factors influencing your decision-making:
Source: https://www.ibm.com/topics/neural-networks
 Neural Networks
How do neural networks work?
Let’s break down what one single node might look like using binary values. We can apply this concept to a more
tangible example, like whether you should go surfing (Yes: 1, No: 0). The decision to go or not to go is our predicted
outcome, or y-hat. Let’s assume that there are three factors influencing your decision-making:
• Are the waves good? (Yes: 1, No: 0)
• Is the line-up empty? (Yes: 1, No: 0)
• Has there been a recent shark attack? (Yes: 0, No: 1)
Then, let’s assume the following, giving us the following inputs:
X1 = 1, since the waves are pumping
X2 = 0, since the crowds are out
X3 = 1, since there hasn’t been a recent shark attack
Now, we need to assign some weights to determine importance. Larger weights signify that particular variables are
of greater importance to the decision or outcome.
W1 = 5, since large swells don’t come around often
W2 = 2, since you’re used to the crowds
W3 = 4, since you have a fear of sharks

Source: https://www.ibm.com/topics/neural-networks
 Neural Networks
How do neural networks work?
Finally, we’ll also assume a threshold value of 3, which would translate to a bias value of –
3. With all the various inputs, we can start to plug in values into the formula to get the
desired output.
Y-hat = (1*5) + (0*2) + (1*4) – 3 = 6
If we use the activation function from the beginning of this section, we can determine
that the output of this node would be 1, since 6 is greater than 0. In this instance, you
would go surfing; but if we adjust the weights or the threshold, we can achieve different
outcomes from the model. When we observe one decision, like in the above example, we
can see how a neural network could make increasingly complex decisions depending on
the output of previous decisions or layers.

Source: https://www.ibm.com/topics/neural-networks
 Neural Networks
How do neural networks work?
In the example above, we used perceptrons to illustrate some of the mathematics
at play here, but neural networks leverage sigmoid neurons, which are
distinguished by having values between 0 and 1. Since neural networks behave
similarly to decision trees, cascading data from one node to another, having x
values between 0 and 1 will reduce the impact of any given change of a single
variable on the output of any given node, and subsequently, the output of the
neural network.

Source: https://www.ibm.com/topics/neural-networks
 Neural Networks
How do neural networks work?
As we start to think about more practical use cases for neural networks, like image
recognition or classification, we’ll leverage supervised learning, or labeled datasets, to
train the algorithm. As we train the model, we’ll want to evaluate its accuracy using a cost
(or loss) function. This is also commonly referred to as the mean squared error (MSE). In
the equation below,
(Source: https://www.ibm.com/topics/neural-networks)

Source: https://builtin.com/machine-learning/cost-function
 Neural Networks
How do neural networks work?
Ultimately, the goal is to minimize our cost function to ensure correctness of fit for
any given observation. As the model adjusts its weights and bias, it uses the cost
function and reinforcement learning to reach the point of convergence, or the local
minimum. The process in which the algorithm adjusts its weights is through
gradient descent, allowing the model to determine the direction to take to reduce
errors (or minimize the cost function). With each training example, the parameters
of the model adjust to gradually converge at the minimum.
Source: https://www.ibm.com/topics/neural-networks
 Neural Networks
How do neural networks work?
Most deep neural networks are feedforward, meaning they flow in one direction only,
from input to output. However, you can also train your model through backpropagation;
that is, move in the opposite direction from output to input. Backpropagation allows us
to calculate and attribute the error associated with each neuron, allowing us to adjust
and fit the parameters of the model(s) appropriately.

Source: https://www.ibm.com/topics/neural-networks
 Neural Networks
Here’s what a simple neural network might look like:

Source: https://towardsdatascience.com/machine-learning-for-beginners-an-introduction-to-neural-networks-d49f22d238f9
 Neural Networks
Advantages and Disadvantages of Neural Networks

Advantages of Neural Networks

• Neutral networks that can work continuously and are more efficient than humans or
simpler analytical models. Neural networks can also be programmed to learn from prior
outputs to determine future outcomes based on the similarity to prior inputs.
• Neural networks that leverage cloud of online services also have the benefit of risk
mitigation compared to systems that rely on local technology hardware. In addition,
neural networks can often perform multiple tasks simultaneously (or at least distribute
tasks to be performed by modular networks at the same time).
• Last, neural networks are continually being expanded into new applications. While
early, theoretical neural networks were very limited to its applicability into different
fields, neural networks today are leveraged in medicine, science, finance, agriculture, or
security.

Source: https://www.investopedia.com/terms/n/neuralnetwork.asp
 Neural Networks
Disadvantages of Neural Networks
• Though neutral networks may rely on online platforms, there is still a hardware
component that is required to create the neural network. This creates a physical risk of
the network that relies on complex systems, set-up requirements, and potential
physical maintenance.
• Though the complexity of neural networks is a strength, this may mean it takes months
(if not longer) to develop a specific algorithm for a specific task. In addition, it may be
difficult to spot any errors or deficiencies in the process, especially if the results are
estimates or theoretical ranges.
• Neural networks may also be difficult to audit. Some neural network processes may feel
"like a black box" where input is entered, networks perform complicated processes, and
output is reported. It may also be difficult for individuals to analyze weaknesses within
the calculation or learning process of the network if the network lacks
general transparency on how a model learns upon prior activity.

Source: https://www.investopedia.com/terms/n/neuralnetwork.asp