IPTC-22151-MS

Novel Stuck Pipe Troubles Prediction Model Using Reinforcement Learning

Downloaded from http://onepetro.org/IPTCONF/proceedings-pdf/22IPTC/2-22IPTC/D021S042R003/2619531/iptc-22151-ms.pdf by U. of Alberta Library user on 15 September 2023

Majed Alzahrani, Bader Alotaibi, and Beshir Aman, Saudi Aramco

Copyright 2022, International Petroleum Technology Conference DOI 10.2523/IPTC-22151-MS

This paper was prepared for presentation at the International Petroleum Technology Conference held in Riyadh, Saudi Arabia, 21-23 February 2022.

This paper was selected for presentation by an IPTC Programme Committee following review of information contained in an abstract submitted by the author(s).
Contents of the paper, as presented, have not been reviewed by the International Petroleum Technology Conference and are subject to correction by the author(s). The
material, as presented, does not necessarily reflect any position of the International Petroleum Technology Conference, its officers, or members. Papers presented at
IPTC are subject to publication review by Sponsor Society Committees of IPTC. Electronic reproduction, distribution, or storage of any part of this paper for commercial
purposes without the written consent of the International Petroleum Technology Conference is prohibited. Permission to reproduce in print is restricted to an abstract of
not more than 300 words; illustrations may not be copied. The abstract must contain conspicuous acknowledgment of where and by whom the paper was presented.
Write Librarian, IPTC, P.O. Box 833836, Richardson, TX 75083-3836, U.S.A., fax +1-972-952-9435.

Abstract
Predicting stuck pipe problems during oil and gas drilling operation is one of the most complex problems
in the drilling business. The complexity of the problem is driven not only by the complexity of the natural
factors, but it extends to the nature of the drilling operation itself. The drilling operation is continuously
influenced by a dynamic smart system. The dynamic part of the system is impacted by natural forces like
formation related characteristics, and also is impacted by human activities during the operation such as
drilling, tripping and hole cleaning. The smartness of this system is driven by the fact that the operation is
controlled by a number of experts, i.e. drilling engineers, trying to run the best sequence of operations using
best operation parameters to achieve operation objective. At the top of that, the engineers can change their
operation plan whenever they find it necessary to address any operational condition, including a potential
stuck pipe problem.
In this paper we prove the stuck pipe prediction problem is not a binary classification problem. Instead,
we define the stuck pipe prediction problem as a multi-class problem which takes into consideration the
dynamic nature of the drilling operation. A reinforcement learning based algorithm is proposed to solve
the redefined problem, and its performance and evaluation results is shared in details. The accuracy of the
developed algorithm in terms of detecting true stuck pipe events is shown. The results will compare the
performance of different machine learning algorithms, which is then used to justify the selection of the
best performing method. In addition, we show the accuracy performance improvement through time by
employing the feedback channel to retrain the model. The presented method is using a reinforcement logic,
in which the solution is connected to the operation reporting to label the solution prediction for false and true
predictions. This information is then used to return the neural networks to learn new operational patterns
to enhance accuracy.
Keywords: Stuck Pipe Prediction, Machine Learning, Neural Networks, Reinforcement Learning

Introduction
Stuck Pipe events is one of the major challenges drilling operators face on regular basis. As the industry
drill more complex wells across depleted and deeper reservoirs, the challenge of stuck pipe continues to
become costly. Due to the complexity of stuck pipe events, there are many factors that could lead to the
2 IPTC-22151-MS

drill pipe getting stuck. Such as wellbore collapse, poor hole cleaning, excessive overbalance or improper
mud weight. This raise the need for accurate methods to predict stuck pipe events before they happen to
give the rig crew enough time to act.
In this work, reinforcement learning is being applied to drilling real-time data to identify early indicators
of stuck pipe events. The developed approach starts with several cases with and without stuck incidents and
then continually learn by including more cases as soon as the learning accuracy start to decay. The result of
this work shows continues improvement of the model ability to detect stuck pipe events before they happen

Downloaded from http://onepetro.org/IPTCONF/proceedings-pdf/22IPTC/2-22IPTC/D021S042R003/2619531/iptc-22151-ms.pdf by U. of Alberta Library user on 15 September 2023

and with low false positive rate.
This work will set the stage for self-learning and continually improveing models to ensure that the driller
has the right support needed.

Literature Review
Reinforcement learning was studied in many applications in the oil and gas industry e.g., Waterflooding
Optimization [1], Reservoir Optimization [2], Trajectory Tracking for Unmanned Surface Vehicle [3], Gas
Transport Control [4] and Oil Production management in smart wells [5]. Time synchronization is a primary
prerequisite for integrating multiple sensor sources [1,2].
With more collection and utilization of drilling real-time data, it became necessary to process and analyze
such instantaneously for trouble avoidance and optimization opportunities. Alzahrani et al [6], developed
a real-time analysis engine to recognize the operation activity (rig state). The rig state is identified using
surface parameters sensors such as hookload (HKLD), Hook Hight (HKHT), Torque (TRQ), Stand-pipe
pressure (SPP), flow in (GPM), rate-of penetration (ROP), Depths (BITDEPTH and HOLEDEPTH) and
Weight-on Bit (WOB). Using such sensors, wide range of rig states can be identified such as Drilling,
tripping-in tripping out, connection, reaming and washing. More in depth rig states are identified such as
rotary drilling and slide drilling.
Several studies have been done to study the impact of utilizing machine learning methods such as
regression and decision tress to identify early warning signs that could lead to potential stuck pipe events.
Omogbolahan et al [7] showed that stuck pipe events can be detected with reasonable accuracy when several
machine learning models are monitoring different signals. Such signals are hookload abnormality, string
movement restrictions, stand-pipe pressure increase or torque spikes. The result of such models is then
aggregated to determine if immediate intervention is required to mitigate such risk. Their work showed
promising result with a precision of 0.67 and sensitivity of 0.8.

Stuck Pipe Problem and Rig State Importance

Modern rigs are connected to tens of sensors measuring the physical state of rig equipment. Table.1 is
showing the main surafce sensors commonly found in any rig real-time stream data. These sensors are
streaming their readings to the rig data acquisition system in real-time. Working with these data requires
full understanding of each sensor function, data range and relation to other sensors. For example, Torque
measuring sensor is correlated with RPM sensor, and hook-load sensor is influenced by the movement
direction of the BHA. The hook-load is the main sensor used to detect any restriction in the hole which
might lead to stuck pipe event. The movement direction influenced difference in value for the same depth
can be as significant as 30%, especially in the horizontal section, Figure.1.

Table 1—Listing of all used surface sensors parameters and their unit of measure.

Parameters Unit

Hookload Klbf

Hook Hight Ft
IPTC-22151-MS 3

Parameters Unit

Rate of Penetration (ROP) Ft/hr

Weight on Bit (WOB) Klbf

Torque Kft.lbf

Revolution Per Minute (RPM) Rpm

Flow-in Pumps GPM

Downloaded from http://onepetro.org/IPTCONF/proceedings-pdf/22IPTC/2-22IPTC/D021S042R003/2619531/iptc-22151-ms.pdf by U. of Alberta Library user on 15 September 2023

Stand Pipe Pressure PSI

Rig Time Time

Figure 1—Hook-load values are sensitive to different operation types. Blue, Red and Green lines are showing the hook-load
readings at different depths while the pipe is moving out of the hole, moving in the hole and while on slips, respectively.

Although Stuck Pipe event can be casted as a binary classification problem, i.e. Stuck and Non-Stuck, the
micro operation classification of the real-time data will help a lot in reducing the variance in sampled data
and will help in achieving higher accuracy. Therefore, the data can be labeled using six groups instead of two
as follows: POH-Stuck, POH-No-Stuck, RIH-Stuck, RIH-No-Stuck, OTHR-Stuck and OTHR-No-Stuck.
The objective of this labeling is to allow any supervised machine learning algorithm to handle each class of
the data differently and to reduce complexity of the problem by incorporating the rig state information.
One challenge might be observed using this methodology is that no enough sample data for each class can
be collected to train a stable model ahead of expected stuck event. Especially the POH class of operations,
usually it comes late at the operation. To over come this issue, we assumed a constant ratio to adjust the
hook-load data between different rig states to a unified value for a given string length. This method might
not be practical for a production solution, but have shown excellent results in our limited experiments in
this paper. A more comprehensive adjusting method might be more accurate for real operation application
and will need the use of more data including inclination and mud data and other factors.

Methodology
In this section, the overall methodology of the developed model will be highlighted and the underlying
components will be further explained in subsequent sections below.
4 IPTC-22151-MS

Data Input Control

In real-world applications, sensors transmitted data are subject to noise and miss calibration. Thus, a strong
data quality verification is a mandatory step in machine learning solution pipe line. In the proposed solution
we go through all used data points and we validate their ranges. For any out of range data point, we replace it
is value with the last known one. For consistency, the model will process only data when the bit is off bottom
and not on slips. In addition, BHA handling data are skipped as they are not of interest to our application.
For the training data, we label all data point with No-Stuck, except the last 6 hours prior to the stuck time

Downloaded from http://onepetro.org/IPTCONF/proceedings-pdf/22IPTC/2-22IPTC/D021S042R003/2619531/iptc-22151-ms.pdf by U. of Alberta Library user on 15 September 2023

which will be labeled as Stuck.
This labeling scheme is very naïve and might give less precise results as it can be easily proven that the
6 hours data prior to the stuck are not all having symptoms of stuck that can be detected by the classifier.
However, the No-Stuck part is mostly true and can be used to calculate a highly accurate specificity score
value. The only issue is in calculating the sensitivity score. So, we change the scoring system to give full
mark for any stuck prediction that take place in the Stuck data interval. And the overall sensitivity score
is calculated for all the testing wells.
The model Specificity score is calculated as follows:

(1)

Such that TNd is the total number of real-time records labeled as No-Stuck by the model and are part of
the No-Stuck data interval. And FPd is the total number of real-time records that are labeled as Stuck by
the model and are part of the No-Stuck data interval.
The model Sensitivity score is calculated as follows:
The model Sensitivity score is calculated as follows:

(2)

Such that TPw is the total number of wells that have at least one real-time record labeled as Stuck by
the model and is part of the Stuck data interval. And FNw is the total number of wells that have Stuck data
interval, and the model labeled no record as Stuck in the Stuck data interval.

Feature Engineering
To allow the model to work on the raw data, we calculate some features to handle the different operations
that influence the sensors readings. The calculated features are as follow: String Length: which is the total
length of the drill pipe string. It equals the bit measured depth combined with the block height. This feature
can better explain the variation on the hook load than the bit depth alone. In addition we calculate two flags
for the Bit-on-Bottom and on-Slips flag. The bit on bottom can be easily identified by comparing the bit
depth and the hole depth, if they are close enough, we set this flag to one, and zero otherwise. For the on-
slips flag, we basically check if the hook load is smaller than a threshold that can safely separate all on-slips
data from the off-slips ones. As we are mostly interested in operations with deep depths, we can assume
a number in the range 60-80 k.lbs.
The string length is calculated as follows:
(3)
The Bit-on-Bottom flag is calculated as follows:

(4)

The on-Slips flag is calculated as follows:

IPTC-22151-MS 5

(5)

Operation Anomaly Detection

The stuck pipe prediction can be casted as operation anomaly detection. In this sense, we can use an encoder
algorithm such that we try to predict the normal operation condition and then we evaluate the difference
between the two values. For that we use a regression model to learn and estimate the hook load for a given
string length. Then we compare the estimated value with the actual sensor value. The larger the variance the

Downloaded from http://onepetro.org/IPTCONF/proceedings-pdf/22IPTC/2-22IPTC/D021S042R003/2619531/iptc-22151-ms.pdf by U. of Alberta Library user on 15 September 2023

higher the chance the operation is going into up normal conditions, or a probability for stuck pipe event. As
the operation is going through different depths and the hook load can vary greatly between these depths and
among different rigs, we normalize the variance value by the actual hook load value, thus, we only evaluate
the variance ratio (vr) instead of evaluating the absolute variance value.
The estimated hook load is calculated as follows:
(6)
Such that HookLoadp is the estimated hook load and HookLoadf is a regression function that returns the
hook load for a given String Length.
The variance ratio (vr) is calculated as follows:

(7)

Such that vr is the variance ration, HookLoadp is the estimated hook load as calculated by the regression
model and HookLoada is the actual hook load reading.
Evaluating a single value might not be sufficient and will usually lead to unstable prediction. The stuck
pipe event hazard is usually persistent and increasing in value closer to the event time. Therefore, we
calculate few statistical factors using the operation anomaly variance (vr). For different sliding window
lengths of the variance data, we label the window with the following values: average, 50, 75, 90 and 95
percentiles. In our experiments, we set the sliding window length to be 1,5,10 and 20 minutes.
These features are then used to train a classifier to find the relation between different variance ratio in
different window lengths and the label of the operation, e.g. Stuck or No-Stuck.

The Reinforcement Step

In the training phase we try to emulate the reinforcement step in testing. Thus, we start the model training
by using few Negative and Positive datasets. These datasets are then used to train the classifier. Then we
randomly pick a new dataset. We run the data through the model and we use the pre-trained classifier to
predict the data labels. After that, we assess the classifier accuracy by measuring the True Positive Rate
and the True Negative Rate. If the classifier accuracy fall below 80% of any these measures we add this
dataset to the classifier training dataset and we retrain the classifier, otherwise we keep the classifier and
pick new dataset and iterate.

Experiments and Results

In the following section we explain the used datasets, the experiment steps and the obtained results.

Stuck Pipe Golden Dataset

We have collected a large dataset which contains real-time data from 125 wells located in nearby oil fields.
The data are focused on deep holes and they start at the beginning of hole section. The data for each well
vary in length. It is duration ranges between 7 days and 2 weeks. The dataset contains 65 positive well cases,
i.e., wells with true stuck event, and 60 negative well cases, i.e., the well experienced no stuck event.
6 IPTC-22151-MS

Training and Testing

In the start we randomly select one positive and one negative well cases to train the model. The developed
model will not evaluate any classifier accuracy at this point. There are two machine learning algorithms
used in our implementation. 1) the regression model, which we simply used a linear regression model that is
trained for each wells data alone. The model is cleared once the BHA is pulled out of hole and new training
is required. Also, the regression will retrain only if the string length is 150 feet longer than the maximum
string length from the initial data used to train the model. 2) the classifier model, which we simply used

Downloaded from http://onepetro.org/IPTCONF/proceedings-pdf/22IPTC/2-22IPTC/D021S042R003/2619531/iptc-22151-ms.pdf by U. of Alberta Library user on 15 September 2023

Decision Tree algorithm.
Initially, the classifier will be trained using the full dataset from the selected two wells. Then, we select
new well randomly. The new well will be processed using the model one data point at a time, and the
classifier will predict the label of that data point, e.g. Stuck or No-Stuck.
Once all record of that well is processed, the model will evaluate the Specificity and Sensitivity scores
and retrain the classifier algorithm using the recent well data if needed.

Discussion
While testing the model, we have observed a significant increase in the model accuracy. By accuracy we
mean the rolling average Sensitivity for the recent 15 wells. The model accuracy is highly correlated with
the increase in the processed dataset. As can be seen in Figure.2, the correlation was found to be p=0.92.
thus, the model is benefiting from seeing new data. Also, the selective retraining procedure allows the model
to focus more on the new information which the model was showing low scores and tries adapting to it
without overwhelming the model with repeated patterns of data which the model already excels with.

Figure 2—Showing the increase of the model accuracy as the model processes more data.
The blue line is the number of total count of processed wells that are used to train the
classification model. The red line shows the moving average sensitivity for the last 15 wells.

The regression model is a key component of the proposed solution. It tries to calculate the expected hook
load value for a given string length under normal conditions. In Figure.3 and Figure.4, we show samples
of the predicted hook load value by the regression model compared to the actual ones. In Figure.3, the data
was selected from interval of No-Stuck data, the model prediction is clearly stable and static as the input
string length is constant for the sampled data, while the actual hook load value is varying slightly.
IPTC-22151-MS 7

Downloaded from http://onepetro.org/IPTCONF/proceedings-pdf/22IPTC/2-22IPTC/D021S042R003/2619531/iptc-22151-ms.pdf by U. of Alberta Library user on 15 September 2023

Figure 3—Showing predicted hook load value vs. actual hook load values for No-Stuck data interval.
Blue line is the predicted hook load value. Red line is the actual hook load value from the real-time data.

Figure 4—Showing predicted hook load value vs. actual hook load values for Stuck data interval. Blue
line is the predicted hook load value. Red line is the actual hook load value from the real-time data.

In a clear contrast, in Figure.4 the actual hook load data is varying greatly compared to the predicted
values. These data are sampled from the Stuck data interval. There are two observations we can notice for
from these two figures: 1) in the No-Stuck data, the actual is always less than predicted values, while the
Stuck data, it has many data points greater than the predicted values. 2) The statistical features we calculate
as input to the classifier model will have very close values in the No-Stuck data interval, while the Stuck
data interval will show bigger differences between these features.
Stuck pipe is a rare event, therefore, a good stuck pipe prediction solution must have a high Specificity
score value to limit the false alerts to minimum. In our experiments, the model is averaging 98.5% for
specificity for the Positive well cases. This shows that the model is very conservative in labeling data as
stuck. Figure.5, is showing the overall specificity for every tested positive well. It is clear that the model
specificity is not degraded while improving the sensitivity score.
8 IPTC-22151-MS

Downloaded from http://onepetro.org/IPTCONF/proceedings-pdf/22IPTC/2-22IPTC/D021S042R003/2619531/iptc-22151-ms.pdf by U. of Alberta Library user on 15 September 2023

Figure 5—Showing the Specificity score for all tested positive wells.

Conclusion
Drilling operations are becoming more complex and costly which raised the need for operators to utilize
latest technologies and big data to try to foresee potential risk and devise appropriate risk management
plans. One of those risk is stuck pipe events. It became necessary to try to apply all available resources in
real-time to monitor how the risk of stuck pipe change as the operation progress.
In this paper we have presented a multi-steps algorithm to handle the Stuck Pipe prediction problem. The
presented model showed 80% and 98.5% for Sensitivity and Specificity scores, respectively. The model
uses reinforcement algorithm, to allow the solution to correct its classification model to adapt to the new
evaluated data. The reinforcement step is found very helpful in improving the Sensitivity score from 15%
to 80% while not losing Specificity.
To enhance this model, other data sources can be included such as Rheology data, bottom-hole assembly
and well trajectory. As well as understand how such work will be impacted when generalized to mixed areas
with different operational practices and data patterns.

References
1. Ma, Hongze, Yu, Gaoming, She, Yuehui, and Yongan Gu. "Waterflooding Optimization under
Geological Uncertainties by Using Deep Reinforcement Learning Algorithms." Paper presented
at the SPE Annual Technical Conference and Exhibition, Calgary, Alberta, Canada, September
2019. doi: https://doi.org/10.2118/196190-MS
2. Miftakhov, Ruslan, Al-Qasim, Abdulaziz, and Igor Efremov. "Deep Reinforcement Learning:
Reservoir Optimization from Pixels." Paper presented at the International Petroleum Technology
Conference, Dhahran, Kingdom of Saudi Arabia, January 2020. doi: https://doi.org/10.2523/
IPTC-20151-MS
3. Jin, Kefan, Wang, Hongdong, and Hong Yi. "End-to-End Trajectory Tracking Algorithm for
Unmanned Surface Vehicle Using Reinforcement Learning." Paper presented at the The 29th
International Ocean and Polar Engineering Conference, Honolulu, Hawaii, USA, June 2019.
4. Žácik, Tibor, Mracka, Igor, Hajossy, Rudolf, and Marek Hycko. "Reinforcement Learning in Gas
Transport Control." Paper presented at the PSIG Annual Meeting, Deer Valley, Utah, USA, May
2018.
5. Talavera, Alvaro Lopez, Túpac, Y. J., and Marley M. Vellasco. "Controlling Oil Production in
Smart Wells by MPC Strategy with Reinforcement Learning." Paper presented at the SPE Latin
IPTC-22151-MS 9

American and Caribbean Petroleum Engineering Conference, Lima, Peru, December 2010. doi:
https://doi.org/10.2118/139299-MS
6. Alzahrani MA, Otavora WC, Al-Nefai MS, inventors; Saudi Arabian Oil Co, assignee. Systems
and Methods to Characterize Well Drilling Activities. United States patent application US
16/588,483. 2021 Apr 1.
7. Ahmed, Omogbolahan S, Aman, Beshir M, Zahrani, Majed A., and Folorunsho I. Ajikobi. "Stuck
Pipe Early Warning System Utilizing Moving Window Machine Learning Approach." Paper

Downloaded from http://onepetro.org/IPTCONF/proceedings-pdf/22IPTC/2-22IPTC/D021S042R003/2619531/iptc-22151-ms.pdf by U. of Alberta Library user on 15 September 2023

presented at the Abu Dhabi International Petroleum Exhibition & Conference, Abu Dhabi, UAE,
November 2019. doi: https://doi.org/10.2118/197674-MS