10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

This member-only story is on us. Upgrade to access all of Medium.

Member-only story

Fitting a Neural Network Using Randomized

Optimization in Python
How randomized optimization can be used to find the optimal weights for machine
learning models, such as neural networks and regression models

Genevieve Hayes, PhD · Follow

Published in Towards Data Science
8 min read · Jan 25, 2019

Listen Share More

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 1/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

Python’s mlrose package provides functionality for implementing some of the most
popular randomization and search algorithms, and applying them to a range of
different optimization problem domains.

In this tutorial, we will discuss how mlrose can be used to find the optimal weights
for machine learning models, such as neural networks and regression models. That
is, to solve the machine learning weight optimization problem.

This is the third in a series of three tutorials about using mlrose to solve randomized
optimization problems. Part 1 can be found here and Part 2 can be found here.

What is the Machine Learning Weight Optimization Problem?

For a number of different machine learning models, the process of fitting the model
parameters involves finding the parameter values that minimize a pre-specified loss
function for a given training set.

Examples of such models include neural networks, linear regression models and
logistic regression models, and the optimal model weights for such models are
typically found using methods such as gradient descent.

However, the problem of fitting the parameters (or weights) of a machine learning
model can also be viewed as a continuous-state optimization problem, where the
loss function takes the role of the fitness function, and the goal is to minimize this
function.

By framing the problem this way, we can use any of the randomized optimization
algorithms that are suited to continuous-state optimization problems to fit the
model parameters.

Solving Machine Learning Weight Optimization Problems with mlrose

mlrose contains built-in functionality for solving the weight optimization problem
for three types of machine learning models: (standard) neural networks, linear
regression models and logistic regression models. This is done using the
NeuralNetwork() , LinearRegression() and LogisticRegression() classes respectively.

Each of these classes includes a fit method, which implements the three steps for
solving an optimization problem defined in the previous tutorials, for a given
training set. That is,

1. Define a fitness function object.

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 2/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

2. Define an optimization problem object.

3. Select and run a randomized optimization algorithm.

However, when fitting a machine learning model, finding the optimal model
weights is merely a means to an end.

We want to find the optimal model weights so that we can use our fitted model to
predict the labels of future observations as accurately as possible, not because we
are actually interested in knowing the optimal weight values.

As a result, the above mentioned classes also include a predict method, which, if
called after the fit method, will predict the labels for a given test set using the
fitted model.

The steps involved in solving a machine learning weight optimization problem with
mlrose are then, typically:

1. Initialize a machine learning weight optimization problem object.

2. Find the optimal model weights for a given training set by calling the fit

method of the object initialized in Step 1.

3. Predict the labels for a test set by calling the predict method of the object
initialized in Step 1.

To fit the model weights, the user can choose between using either randomized hill
climbing, simulated annealing, the genetic algorithm or gradient descent.

[In mlrose, the gradient descent algorithm is only available for use in solving the
machine learning weight optimization problem and has been included primarily for
benchmarking purposes, since this is one of the most common algorithms used in
fitting neural networks and regression models.]

We will now work through an example to illustrate how mlrose can be used to fit a
neural network and a regression model to a given dataset.

Before starting with the example, you will need to import the mlrose and Numpy
Python packages.

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 3/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

import mlrose
import numpy as np

Example: the Iris Dataset

The Iris dataset is a famous multivariate classification dataset first presented in a
1936 research paper by statistician and biologist Ronald Fisher.

Irises (1889) by Vincent Van Gogh

It contains 150 observations of three classes (species) of iris flowers (50 observations
of each class), with each observation providing the sepal length, sepal width, petal
length and petal width (i.e. the feature values), as well as the class label (i.e. the
target value), of each flower under consideration.

The Iris dataset is included with Python’s sklearn package.

The feature values and label of the first observation in the dataset are shown below,
along with the maximum and minimum values of each of the features and the
unique label values:

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 4/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

1 from sklearn.datasets import load_iris

2
3 # Load the Iris dataset
4 data = load_iris()
5
6 # Get feature values
7 print('The feature values for Obs 0 are: ', data.data[0])
8
9 # Get feature names
10 print('The feature names are: ', data.feature_names)
11
12 # Get target value of first observation
13 print('The target value for Obs 0 is:', data.target[0])
14
15 # Get target name of first observation
16 print('The target name for Obs 0 is:', data.target_names[data.target[0]])
17
18 # Get minimum feature values
19 print('The minimum values of the four features are:', np.min(data.data, axis = 0))
20
21 # Get maximum feature values
22 print('The maximum values of the four features are:', np.max(data.data, axis = 0))
23
24 # Get unique target values
25 print('The unique target values are:', np.unique(data.target))

Iris_Exploration.py hosted with ❤ by GitHub view raw

The feature values for Obs 0 are: [5.1 3.5 1.4 0.2]

The feature names are: ['sepal length (cm)', 'sepal width (cm)',
'petal length (cm)', 'petal width (cm)']

The target value for Obs 0 is: 0

The target name for Obs 0 is: setosa

The minimum values of the four features are: [4.3 2. 1. 0.1]

The maximum values of the four features are: [7.9 4.4 6.9 2.5]

The unique target values are: [0 1 2]

From this we can see that all features in the Iris data set are numeric, albeit with
different ranges, and that the class labels have been represented by integers.

In the next few sections we will show how mlrose can be used to fit a neural network
and a logistic regression model to thisOpen
dataset,
in app to predict the species of an iris flower
https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 5/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

given its feature values.

Data Pre-Processing
Before we can fit any sort of machine learning model to a dataset, it is necessary to
manipulate our data into the form expected by mlrose.

Each of the three machine learning models supported by mlrose expect to receive
feature data in the form of a Numpy array, with one row per observation and
numeric features only (any categorical features must be one-hot encoded before
passing to the machine learning models).

The models also expect to receive the target values as either: a list of numeric values
(for regression data); a list of 0–1 indicator values (for binary classification data); or
as a Numpy array of one-hot encoded labels, with one row per observation (for
multi-class classification data).

In the case of the Iris dataset, all of our features are numeric, so no one-hot
encoding is required. However, it is necessary to one-hot encode the class labels.

In keeping with standard machine learning practice, it is also necessary to split the
data into training and test subsets, and since the range of the Iris data varies
considerably from feature to feature, to standardize the values of our feature
variables.

These pre-processing steps are implemented below:

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 6/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

1 from sklearn.model_selection import train_test_split

2 from sklearn.preprocessing import MinMaxScaler, OneHotEncoder
3
4 # Split data into training and test sets
5 X_train, X_test, y_train, y_test = train_test_split(data.data, data.target,
6 test_size = 0.2, random_state = 3)
7
8 # Normalize feature data
9 scaler = MinMaxScaler()
10
11 X_train_scaled = scaler.fit_transform(X_train)
12 X_test_scaled = scaler.transform(X_test)
13
14 # One hot encode target values
15 one_hot = OneHotEncoder()
16
17 y_train_hot = one_hot.fit_transform(y_train.reshape(-1, 1)).todense()
18 y_test_hot = one_hot.transform(y_test.reshape(-1, 1)).todense()

Neural Networks
Once the data has been pre-processed, fitting a neural network in mlrose simply
involves following the steps listed above.

Suppose we wish to fit a neural network classifier to the Iris dataset with one hidden
layer containing 2 nodes and a ReLU activation function (mlrose supports the ReLU,
identity, sigmoid and tanh activation functions).

For this example, we will use the Randomized Hill Climbing algorithm to find the
optimal weights, with a maximum of 1000 iterations of the algorithm and 100
attempts to find a better set of weights at each step.

We will also include a bias term; use a step size (learning rate) of 0.0001 (to find
neighbors of the current set of weights); and limit our weights to being in the range
-5 to 5 (to reduce the landscape over which the algorithm must search in order to
find the optimal weights).

This model is initialized and fitted to our pre-processed data below:

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 7/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

1 # Initialize neural network object and fit object

2 nn_model1 = mlrose.NeuralNetwork(hidden_nodes = [2], activation = 'relu',
3 algorithm = 'random_hill_climb', max_iters = 1000,
4 bias = True, is_classifier = True, learning_rate = 0.00
5 early_stopping = True, clip_max = 5, max_attempts = 100
6 random_state = 3)
7
8 nn_model1.fit(X_train_scaled, y_train_hot)
9

Iris NN Fit py hosted with ❤ by GitHub view raw

Once the model is fitted, we can use it to predict the labels for our training and test
sets, and use these predictions to assess the model’s training and test accuracy.

1 from sklearn.metrics import accuracy_score

2
3 # Predict labels for train set and assess accuracy
4 y_train_pred = nn_model1.predict(X_train_scaled)
5
6 y_train_accuracy = accuracy_score(y_train_hot, y_train_pred)
7
8 print('Training accuracy: ', y_train_accuracy)
9
10 # Predict labels for test set and assess accuracy
11 y_test_pred = nn_model1.predict(X_test_scaled)
12
13 y_test_accuracy = accuracy_score(y_test_hot, y_test_pred)
14
15 print('Test accuracy: ', y_test_accuracy)

Iris_NN_Predict.py hosted with ❤ by GitHub view raw

Training accuracy: 0.45

Test accuracy: 0.533333333333

In this case, our model achieves training accuracy of 45% and test accuracy of
53.3%. These accuracy levels are better than if the labels were selected at random,
but still leave room for improvement.

We can potentially improve on the accuracy of our model by tuning the parameters
we set when initializing the neural network object. Suppose we decide to change the
https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 8/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

optimization algorithm to gradient descent, but leave all other model parameters
unchanged.

1 # Initialize neural network object and fit object

2 nn_model2 = mlrose.NeuralNetwork(hidden_nodes = [2], activation = 'relu',
3 algorithm = 'gradient_descent', max_iters = 1000,
4 bias = True, is_classifier = True, learning_rate = 0.0
5 early_stopping = True, clip_max = 5, max_attempts = 10
6 random_state = 3)
7
8 nn_model2.fit(X_train_scaled, y_train_hot)
9
10 # Predict labels for train set and assess accuracy
11 y_train_pred = nn_model2.predict(X_train_scaled)
12
13 y_train_accuracy = accuracy_score(y_train_hot, y_train_pred)
14
15 print('Training accuracy: ', y_train_accuracy)
16
17 # Predict labels for test set and assess accuracy
18 y_test_pred = nn_model2.predict(X_test_scaled)
19
20 y_test_accuracy = accuracy_score(y_test_hot, y_test_pred)
21
22 print('Test accuracy: ', y_test_accuracy)

❤ i

Training accuracy: 0.625

Test accuracy: 0.566666666667

This results in a 39% increase in training accuracy to 62.5%, but a much smaller
increase in test accuracy to 56.7%.

Linear and Logistic Regression Models

Linear and logistic regression models are special cases of neural networks. A linear
regression is a regression neural network with no hidden layers and an identity
activation function, while a logistic regression is a classification neural network
with no hidden layers and a sigmoid activation function.

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 9/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

As a result, we could fit either of these models to our data using the NeuralNetwork()

class, with parameters set appropriately.

For example, suppose we wished to fit a logistic regression to the Iris data using the
randomized hill climbing algorithm and all other parameters set as for the example
in the previous section. We could do this by initializing a NeuralNetwork() object like
so:

1 lr_nn_model1 = mlrose.NeuralNetwork(hidden_nodes = [], activation = 'sigmoid',

2 algorithm = 'random_hill_climb', max_iters = 1000,
3 bias = True, is_classifier = True, learning_rate = 0
4 early_stopping = True, clip_max = 5, max_attempts =
5 random_state = 3)

Iris Logistic NN Init py hosted with ❤ by GitHub view raw

However, for convenience, mlrose provides the LinearRegression() and
LogisticRegression() wrapper classes, which simplify model initialization.

In the Iris dataset example, we can, thus, initialize and fit our logistic regression
model as follows:

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 10/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

1 # Initialize logistic regression object and fit object

2 lr_model1 = mlrose.LogisticRegression(algorithm = 'random_hill_climb', max_iters = 1000
3 bias = True, learning_rate = 0.0001,
4 early_stopping = True, clip_max = 5, max_attempts
5 random_state = 3)
6
7 lr_model1.fit(X_train_scaled, y_train_hot)
8
9 # Predict labels for train set and assess accuracy
10 y_train_pred = lr_model1.predict(X_train_scaled)
11
12 y_train_accuracy = accuracy_score(y_train_hot, y_train_pred)
13
14 print('Training accuracy: ', y_train_accuracy)
15
16 # Predict labels for test set and assess accuracy
17 y_test_pred = lr_model1.predict(X_test_scaled)
18
19 y_test_accuracy = accuracy_score(y_test_hot, y_test_pred)
20
21 print('Test accuracy: ', y_test_accuracy)

i i i i i ❤ i view raw

Training accuracy: 0.191666666667

Test accuracy: 0.0666666666667

This model achieves 19.2% training accuracy and 6.7% test accuracy, which is worse
than if we predicted the labels by selecting values at random.

Nevertheless, as in the previous section, we can potentially improve model accuracy

by tuning the parameters set at initialization.

Suppose we increase our learning rate to 0.01.

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 11/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

1 # Initialize logistic regression object and fit object

2 lr_model2 = mlrose.LogisticRegression(algorithm = 'random_hill_climb', max_iters = 1000
3 bias = True, learning_rate = 0.01,
4 early_stopping = True, clip_max = 5, max_attempts
5 random_state = 3)
6
7 lr_model2.fit(X_train_scaled, y_train_hot)
8
9 # Predict labels for train set and assess accuracy
10 y_train_pred = lr_model2.predict(X_train_scaled)
11
12 y_train_accuracy = accuracy_score(y_train_hot, y_train_pred)
13
14 print('Training accuracy: ', y_train_accuracy)
15
16 # Predict labels for test set and assess accuracy
17 y_test_pred = lr_model2.predict(X_test_scaled)
18
19 y_test_accuracy = accuracy_score(y_test_hot, y_test_pred)
20
21 print('Test accuracy: ', y_test_accuracy)

i i i i i ❤ i view raw

Training accuracy: 0.683333333333

Test accuracy: 0.7

This results in significant improvements to both training and test accuracy, with
training accuracy levels now reaching 68.3% and test accuracy levels reaching 70%.

Summary
In this tutorial we discussed how mlrose can be used to find the optimal weights of
three types of machine learning models: neural networks, linear regression models
and logistic regression models.

Applying randomized optimization algorithms to the machine learning weight

optimization problem is most certainly not the most common approach to solving
this problem. However, it serves to demonstrate the versatility of the mlrose package
and of randomized optimization algorithms in general.

To learn more about mlrose, visit the GitHub repository for this package, available here.

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 12/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

Dr Genevieve Hayes is a data scientist, educator and AI and analytics specialist with
Genevieve Hayes Consulting. You can follow her on LinkedIn or Twitter. She is also the host
of Value Driven Data Science, a twice-monthly podcast for businesses looking to maximise
the value of their data and data teams.

Want to unlock the value of your business’s data, but don’t know where to begin?
Download the FREE Data Science Project Discovery Guide.

Machine Learning Data Science Python Programming Towards Data Science

Follow

Written by Genevieve Hayes, PhD

1.95K Followers · Writer for Towards Data Science

Data scientist and educator with a PhD in Statistics — Helping data professionals maximize the value of data
without expensive tools — www.genevievehayes.com.

More from Genevieve Hayes, PhD and Towards Data Science

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 13/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

Genevieve Hayes, PhD in Towards Data Science

Team R or Team Python?

“It is a truth universally acknowledged, that a data scientist in possession of a good grounding
in computer science and statistics, must…

· 10 min read · Aug 18, 2019

113

Damian Gil in Towards Data Science

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 14/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

Mastering Customer Segmentation with LLM

Unlock advanced customer segmentation techniques using LLMs, and improve your clustering
models with advanced techniques

23 min read · Sep 27

3.1K 25

Khouloud El Alami in Towards Data Science

Don’t Start Your Data Science Journey Without These 5 Must-Do Steps
From a Spotify Data Scientist
A complete guide to everything I wish I’d done before starting my Data Science journey, here’s
to acing your first year with data

· 18 min read · Sep 25

2.3K 23

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 15/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

Genevieve Hayes, PhD in Towards Data Science

Getting Started with Reinforcement Learning and Open AI Gym

Solving the Mountain Car environment using Q-learning.

· 8 min read · Feb 22, 2019

535 9

See all from Genevieve Hayes, PhD

See all from Towards Data Science

Recommended from Medium

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 16/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

Gazala

Mixed Integer Programming Python (Pulp)

Mixed Integer Programming (MIP) is a powerful optimization technique used to solve complex
decision-making problems that involve a…

8 min read · Jun 9

11 1

Damian Gil in Towards Data Science

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 17/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

Mastering Customer Segmentation with LLM

Unlock advanced customer segmentation techniques using LLMs, and improve your clustering
models with advanced techniques

23 min read · Sep 27

3.1K 25

Lists

Predictive Modeling w/ Python

20 stories · 488 saves

Practical Guides to Machine Learning

10 stories · 557 saves

Coding & Development

11 stories · 216 saves

New_Reading_List
174 stories · 149 saves

Ms Aerin in IntuitionMath

Chi Square Test — Intuition, Examples, and Step-by-Step Calculation

The best way to see if two variables are related.

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 18/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

· 15 min read · Feb 13

408 3

Anta Osotsi in The Modern Scientist

An In-Depth Look at Genetic Algorithms: The Art of Python Optimization

Master Genetic Algorithms by Examining the Traveling Salesman Problem

· 10 min read · May 3

61 1

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 19/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

Virat Patel

I applied to 230 Data science jobs during last 2 months and this is what
I’ve found.
A little bit about myself: I have been working as a Data Analyst for a little over 2 years.
Additionally, for the past year, I have been…

· 3 min read · Aug 11

1.4K 35

Sandipan Dutta in Level Up Coding

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 20/21
10/15/23, 10:25 AM Fitting a Neural Network Using Randomized Optimization in Python | by Genevieve Hayes, PhD | Towards Data Science

Advanced Python: Performance Optimization

10 Common Scenarios and Strategies

7 min read · 6 days ago

51

See more recommendations

https://towardsdatascience.com/fitting-a-neural-network-using-randomized-optimization-in-python-71595de4ad2d 21/21