Hindawi

Wireless Communications and Mobile Computing

Volume 2022, Article ID 4218696, 9 pages
https://doi.org/10.1155/2022/4218696

Research Article
Fragmented Task Scheduling for Load-Balanced Fog Computing
Based on Q-Learning

Mian Muaz Razaq , Shahnila Rahim, Byungchul Tak , and Limei Peng
School of Computer Science and Engineering, Kyungpook National University, Daegu 41566, Republic of Korea

Correspondence should be addressed to Limei Peng; [email protected]

Received 28 December 2021; Accepted 26 January 2022; Published 10 March 2022

Academic Editor: Xiaojie Wang

Copyright © 2022 Mian Muaz Razaq et al. This is an open access article distributed under the Creative Commons Attribution
License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is
properly cited.

5G and beyond (B5G) applications generate tremendous computing-intensive, latency-sensitive, and privacy-sensitive tasks,
which differ from the legacy cloud computing tasks, requiring more sophisticated scheduling strategies. We must satisfy the
stringent service requirements, particularly privacy preservation that has not been sufficiently considered in the past.
Meanwhile, we need to balance the tasks offloaded to different edge nodes to avoid overwhelming some fog nodes, which may
degrade the overall performance. To appropriately schedule the privacy-sensitive tasks while balancing the traffic load, we
define IoT tasks according to their security need, processing time, and real-time requirement and segment IoT tasks into
smaller pieces based on their privacy levels. The sliced tasks are scheduled to multiple fog nodes with satisfactory security
reputations to avoid a compromised fog node handling a whole task. Meanwhile, we consider the constraint of the response
time of all available fog nodes before scheduling IoT tasks to avoid chaos task scheduling that may overwhelm some fog nodes.
Regarding this, we propose a reinforcement learning (RL) model in which the agent tends to satisfy the required latency and
security requirements while avoiding overloading some fog nodes to minimize the average delay. The numerical results
demonstrate that the proposed approach performs well in a better-balanced load and less performance violation in latency and
security.

1. Introduction sures. For instance, authors in [4] sliced an IoT task into
three pieces with different sizes and schedule them to the
With the mature of 5G technologies that support pervasive cloud, fog, and local machine, respectively. Each of the sliced
Internet connections anywhere at any time, the number of pieces lacks a part of crucial information, thus increasing the
Internet users will reach 5.3 billion, almost 71 percent of privacy of overall data even if a portion of data is being com-
the population, in 2023 [1]. Most of them use wireless promised. To avoid data leakages from location-based ser-
devices, such as smartphones, tablets, smartwatches, home vices (LBS) and keep the actual data trajectory, dummy
meters, and wearable devices, proliferating numerous mobile trajectories were sent along with encrypted data to LBS for
IoT data that require real-time and secure services. These processing and getting results at fog nodes [5]. A decentra-
services are pretty different from the legacy cloud data ser- lized blockchain-based technique is proposed in [6] utilizing
vices, thus soliciting the fog nodes at the edge rich in com- the edge nodes to assist the burdens of cluster heads formed,
puting, storage, and bandwidth resources, to satisfy the aiming to minimize the system latency while maximizing the
emerging service demands of IoT data [2, 3]. data safety of the user.
Efficiently offloading IoT tasks to fog nodes while satisfy- Another crucial issue is to avoid overwhelming some fog
ing the QoS requirement with minimized fog resources is of nodes with numerous IoT tasks while leaving some other fog
primary concern. Motivated by the fact that attacking cloud nodes underutilized, which would degrade the overall per-
servers results in significant data leakage, some literature has formance of fog networks. Local fog managers and SDN
considered fragmenting IoT tasks to increase privacy mea- controllers have been utilized to balance the load both locally
 6302, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1155/2022/4218696 by Jordan Hinari NPL, Wiley Online Library on [03/01/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
2 Wireless Communications and Mobile Computing

and globally by splitting each fog node into several levels, the tasks to fog nodes by considering a generalized model
with each level indicating a load threshold at that fog node. with flexible topology [10]. Wang et al. have proposed an
Fog managers monitor the load levels of fog nodes belonging imitation learning-based approach to provide the most
to their respective clusters and distribute the tasks accord- updated information to the vehicular network by minimiz-
ingly [7]. Some works have considered offloading the task ing the Age-of-Critical-Information (AoCI) taking advan-
to the nearest available fog node for processing first and then tage of the edge nodes [11].
to the cloud if the fog node is overloaded [8]. Mobile fog computing using unmanned aerial vehicles
To satisfy the privacy requirement while balancing the (UAVs) as fog nodes has been widely discussed in the past
load of fog nodes, this paper fragments each IoT task accord- few years. In [12], Ning et al. considered offloading tasks
ing to their security level and offloads each sliced task piece to UAV-based edge servers that are deployed to minimize
to the fog node with the least response time and satisfactory the total cost. Authors in [13] considered grouping the IoT
security reputation. For this purpose, we categorize the fog users to different clusters and providing computing services
nodes and IoT tasks into three different security levels, i.e., through UAVs serving as mobile edge nodes. In [14], Wang
low, medium, and high. The processing capacity of fog et al. proposed an imitation learning-based approach to
nodes and processing requirement of IoT tasks are catego- deploy UAVs as mobile edge nodes and addressed the issue
rized in a similar way. A fog node with a higher processing of deploying UAVs belonging to different service providers
capacity can process IoT tasks in less time than the one with in a shared area such that the profits of the owners along
low processing capacity. with the utilities of users are maximized.
To achieve the above ideas, this paper proposes to deploy Some literature also addressed the energy efficiency issue
a reinforcement learning agent at one of the edge servers in while offloading the tasks to fog nodes. Zhang et al. proposed
the fog layer. Before scheduling an IoT task to any fog node an energy-minimizing algorithm that selects a fog node for
or cloud, the agent slices the task into several fragments task offloading, considering the energy consumption, history
based on the security requirement and size of the IoT task. about the average energy usage of fog nodes, and priorities of
Only one sliced piece can be scheduled to the same fog node. fog nodes [15]. Yang et al. maximized the energy efficiency
If the agent schedules an IoT task piece to a fog node with a of fog networks consisting of fog nodes with different com-
lower security level than the fragment requires, it is consid- puting resources by proposing the maximal energy-efficient
ered a security violation, and the agent will receive a sub- task scheduling algorithm [16]. Jalali et al. compared the
stantial negative penalty for such action. Moreover, the nanodata centers (nDCs) used in fog computing with the
agent also monitors the required response time to serve the centralized cloud data centers. They pointed out that energy
task piece by all the available fog nodes at that time step efficiency is impacted more by the application types, the type
and selects the node with a lesser response time, thus avoid- of access networks attached to the nanoservers, and the
ing overloading only the nearest fog nodes. Regarding this, active and idle time of nanoservers, rather than the number
we propose a Q-learning-based algorithm, which guides of hops [17].
the agent to schedule the incoming IoT task pieces to fog
nodes or cloud, considering the security, real-time, and
bandwidth requirements while avoiding overwhelming some 3. System Description and System Model
fog nodes.
The remainder of this paper is organized as follows. Sec- 3.1. System Description. A three-layer integrated fog and
tion 2 introduces the related works. Section 3 presents the cloud framework is considered as shown in Figure 1. The
preliminaries, including the system model and description, bottom IoT layer consists of IoT devices such as tablets
the reinforcement learning environment, and the proposed and smartphones, which generate IoT tasks with different
Q-learning-based algorithm. Section 4 defines the simula- requirements in terms of their nature, such as real time or
tion environment and discusses the simulation results. not, security and privacy preservation level, and band-
Finally, Section 5 concludes the work. width-intensity.
The middle fog layer comprises fog nodes with different
2. Related Works capabilities, such as different security credits, computing and
storage capability, and mobility. Specifically, their security
Minimizing the delay while scheduling IoT tasks to fog reputation and processing capability can be categorized into
nodes has been the research focus of many existing works. high, medium, and low. The processing ability of a fog node
Chiti et al. in [9] proposed a distributed algorithm to opti- affects the task execution time when a task goes from the
mally select fog nodes based on matching theory, aiming to task waiting queue to the processing stage. On the other
minimize the maximum total task completion time to hand, the reinforcement learning (RL) agent is deployed in
achieve efficient task offloading for real-time applications the edge node, which is responsible for scheduling and off-
with considerations on communications and computational loading IoT tasks to other fog nodes or the cloud for
costs. Tran-Dang et al. proposed the adaptive resource- processing.
aware task offloading scheme to minimize the average delay The top cloud layer is assumed to have sufficient com-
by selecting an optimal policy to determine the most appro- puting and storage resources to serve all the IoT tasks but
priate fog node with available resources for task offloading. cannot satisfy the real-time requirement due to its long dis-
Yousefpour et al. reduced the service delay when offloading tance from the IoT users. Serving an IoT task with the real-
 6302, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1155/2022/4218696 by Jordan Hinari NPL, Wiley Online Library on [03/01/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Wireless Communications and Mobile Computing 3

: Data center
: RL agent
: Control unit
: Link
: Wireless node
: IoT devices
: Wired node

Figure 1: Three layered integrated cloud fog framework.

time requirement by the distant cloud is deemed as a delay 3.2. System Model. Table 1 shows the notations of sets,
violation. parameters, variables, and their descriptions used for
Fog nodes can serve IoT tasks with real-time require- describing the system. Cloud with sufficient computing, stor-
ments and security levels less than the security reputation age, and network resources is represented by C and can process
of the fog nodes. We slice a single IoT task into smaller all but the real-time tasks. We use the sets of F = fF 1 , F 2 , ⋯,
pieces based on their security requirement to help in privacy F j g and T = fT 1 , T 2 , ⋯, T i g to denote the fog nodes and IoT
preservation. A fog node with an equal or higher security tasks, respectively, where j and i are integer indexes.
reputation than the security level of an IoT task piece can Each fog node F j in the set F is a four tuple, i.e., F j = fF
provide service. Moreover, a single fog node is not allowed
R j , FS j , FP j , RP j g, where FR j , FS j , FP j , and RP j are the avail-
to serve more than one piece of a sliced IoT task. IoT task
pieces can be offloaded to either the cloud or fog nodes flex- able computing/storage resources, security level, processing
ibly only if their requirements can be satisfied. The objective power, and response time of the fog node j. Specifically, the
of task offloading is to minimize the average end-to-end value of 0, 1, or 2 for FS j means fog node j has low, medium,
(E2E) delay, avoiding overwhelming some fog nodes to bal- or high security reputation; the value of 0, 1, or 2 of FP j indi-
ance the traffic load among fog nodes. We treat the average cates low, medium, or high processing capability of fog node j.
response time of the whole iCloudFog as the vital metric for Similarly, each IoT task element in the set T is a three tuple
evaluating the network load balance. The reason behind this characterized by quality-of-service (QoS) requirements, i.e.,
is that if there are some overloaded fog or cloud nodes, their T i = fSLi , RTi , RCi g, where SLi , RTi , and RCi denote the
response time may become very considerably large, leading requirements of security level, real-time service, and required
to a high average response time. computing/storage resources. Specifically, the value of 0, 1,
The response time is determined by the E2E delay, con- or 2 for SLi indicates the low-, medium-, and high security
sisting of the processing time, queuing delay, propagation requirement; RTi is one when task i has the real-time require-
time, and transmission time when serving an IoT task. ment and zero vice versa. The value of 0, 1, or 2 for RCi indi-
Therefore, we always consider the E2E delay before schedul- cates task i is low, medium, or high computing-intensive. DEi
ing any task and avoid offloading IoT task segments to nodes denotes the maximum delay requirement of taski.
requiring long response time, which in turn helps balance
the overall traffic load. Based on the above assumptions, this 4. RL Model for Load Balancing and Privacy-
paper tackles the problem of IoT task scheduling by propos- Aware Task Offloading
ing an intelligent solution based on Q-learning that meets
the requirements of security, real-time, etc., of IoT tasks The objective is to minimize the average delay of task off-
while minimizing the average E2E delay of iCloudFog when loading to balance the traffic load of fog nodes and mean-
offloading IoT tasks for load balancing. while satisfy various QoS requirements, especially the
 6302, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1155/2022/4218696 by Jordan Hinari NPL, Wiley Online Library on [03/01/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
4 Wireless Communications and Mobile Computing

Table 1: Sets, parameters, and variables.

C Cloud computing server

F Set of fog nodes
T Set of IoT tasks
SðkÞ State space of system at time slot k
AðkÞ Action space of system at time slot k
R Possible states of available resources
RTi Binary variable. One indicates IoT task i is real-time task; zero, vice versa; i ∈ T
dvij Binary variable. One indicates there is a delay violation while serving task i at fog node j; i ∈ T j ∈ F
svij Binary variable. One indicates there is a security violation while serving task i at fog node j; i ∈ T j ∈ F
TRi Required resources of IoT task i; i ∈ T
SLi Security requirement of IoT task i; i ∈ T
RCi Computing required of IoT task i; i ∈ T
TFi Number of fragments of IoT task i; i ∈ T
DEi Maximum delay threshold of IoT task i; i ∈ T
FR j Available resources of fog node j; j ∈ F
FS j Security level of fog node j; j ∈ F
FP j Processing power of fog node j; j ∈ F
RPij Response time of fog node j for IoT task i; i ∈ T j ∈ F
RPc Response time of cloud
RR j Reward based on response time of fog node j; j ∈ F
RS j Reward based on security adherence j; j ∈ F
Tdij Transmission delay; i ∈ T j ∈ F
Pdij Propagation delay; i ∈ T j ∈ F
Prij Processing delay; i ∈ T j ∈ F
Tsi Task size; i ∈ T
BWij Link bandwidth; i ∈ T j ∈ F
Dsij Distance between fog node j and source of IoT task i; i ∈ T j ∈ F
c Speed of light
λj Arrival rate of tasks at fog node j; j ∈ F
μj Service rate of tasks at fog node j; j ∈ F
RWDas Reward received by agent after taking an action a in state s; a ∈ AðkÞ, s ∈ SðkÞ
RWDkj Cumulative future reward; j ∈ F
F ∗j Optimal fog node j; j ∈ F
α Learning rate
γ Discount factor

privacy requirement. We assume that a fog node with a high states, a finite set of actions, a transition probability matrix
traffic load suffers from a long response time to handle new for every action in each state, and the reward associated with
tasks, and hence, the response time of a fog node determines every state [18], respectively.
the probability of it being selected to provision a new IoT
task. The problem is formulated as a Markov decision pro-
cess (MDP) model which can be solved by the reinforcement 4.1. State. The state of fog node j at time slot k is determined
learning- (RL-) based algorithm proposed in this paper. The based on its available resources FR j , security level FS j ,
proposed RL MDP model is represented by a quadruple < response time FP j , and the response time of cloud RPc .
S, A, P , R > , where S, A, P , and R define a finite set of Based on the assumption that the cloud always has sufficient
 6302, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1155/2022/4218696 by Jordan Hinari NPL, Wiley Online Library on [03/01/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Wireless Communications and Mobile Computing 5

resources to serve tasks, the cloud resources do not affect the sition to the next state [19]. The reward of our agent on each
system state and thus is not considered here. Nonetheless, time step k is determined by the response time of fog node j
the response time of the cloud is vital for task scheduling and its security level deciding how well it can serve an IoT
and contributes to determining the system state. The state task with security requirement, denoted by RWDas ðkÞ as fol-
of fog node j consisting of a couple of substates at time slot lows:
k is represented as follows:
À Á RWDas ðkÞ = RR j ðkÞ + RS j ðkÞ, ð7Þ
j
Δ ðkÞ = FR j , FS j , RP j : ð1Þ
RS j ðkÞ = c2 − c3 ∗ svij , ð8Þ
The system state space is comprised of the states of all
fog nodes and the response time of the cloud, which is rep- RR j ðkÞ = −RP j ðkÞ − c1 ∗ dvij , ð9Þ
resented by Equation (2), where n is the total number of fog
where RS j ðkÞ is the negative or positive reward received by
nodes.
the agent based on the security level of fog node j and secu-
 à rity requirement of IoT task i; c1 , c2 , and c3 are constant
SðkÞ = Δ1 ðkÞ, Δ2 ðkÞ, Δ3 ðkÞ ⋯ Δn ðkÞ, RPc : ð2Þ
values where c3 > c1 > c2 ; dvij and svij are binary parameters,
Assume the state of available resource of fog node j where dvij = 1 indicates a delay violation, meaning the delay
changes from r to r ′ in time slot k and SR is the set of possible requirement of IoT task (fragment) i cannot be satisfied by
states of the available resources; the transition probability of fog node j, and svij = 1 indicates a security violation, mean-
j
the available resource state, i.e., Pr,r′ ðkÞ, is defined as follows: ing the security credit of fog node j is lower than IoT task
(fragment) i requires and vice versa; RR j ðkÞ is the negative
h i or positive reward that the agent gets based on the response
j
Pr,r ′ ðkÞ = Pr FR j ðk + 1Þ = r ′ ∣ FR j ðkÞ = r ∀r, r ′ ∈ SR : time of fog node j; and RP j ðkÞ therein is the response time of
ð3Þ fog node j for provisioning IoT task i at time slot k.

4.2. Action. The system action space consists of all the actions RP j ðkÞ = Tdij + Pdij + Prij , ð10Þ
that the agent can take at a particular state. The action for IoT
task T i in time slot k can be chosen from a combination of where Tdij , Pdij , and Prij are the transmission delay, propa-
available fog nodes and cloud C while avoiding selecting the gation delay, and processing delay, respectively, when sched-
same fog node. This is based on the assumption that an IoT uling task i to fog node j, as defined in Equations (11), (12),
task can be sliced into pieces and each node cannot provision and (13). Transmission delay Tdij is calculated by dividing
more than one of the pieces. The number of the combination task size Tsij over the link bandwidth BWij ; propagation
based on the fog nodes and cloud is defined as follows: delay Pdij is calculated by dividing the distance between
! fog node j and the source device of task i, i.e., Dsij , over
n+1 n + 1!
N= = , ð4Þ the light speed c; processing delay Prij considers the task
TF TF i ! ðð n + 1Þ − TFi Þ! arrival rate λ j and service rate μ j at fog node j.
i

where n + 1 means the total n available fog nodes plus the Tsi
cloud. TFi is the number of fragments that IoT task i has been Tdij = , ð11Þ
BWij
sliced. Hence, the action space can be represented as follows:
Dsij
Ai ðkÞ = ½a1 ðkÞ, a2 ðkÞ, a3 ðkÞ, ⋯, aN ðkފ, ð5Þ Pdij = , ð12Þ
c
where N is the number of combination of fog nodes and cloud λ 
j

obtained from Equation (4). Prij = μj − λj : ð13Þ
2μ j
4.3. State Transition. When the agent takes an action a ∈ Ai Based on Equations (7), (8), and (9), the agent will
ðkÞ in state S of the system at time slot k, the system will receive a positive reward c2 if it schedules the task to a fog
change its state from s to s ′ , and the transition probability node that satisfies the task security requirement. Nonethe-
matrix of the system is defined as follows: less, if it schedules the task to a fog node with lower security
h i level than the task requirement, causing a security violation,
Pas,s′ = Pr sðk + 1Þ = s ′ ∣ sðkÞ = s, aðkÞ = a : ð6Þ i.e., svij = 1, the agent will be penalized with a large negative
reward c3 .
4.4. Reward. When the reinforcement learning (RL) agent In addition, the agent will receive a positive reward equal
explores the environment, it learns from the reward fed back to the response time of fog node j, i.e., RP j , if there is no
by the environment when it takes some action from the delay violation when scheduling IoT task i. In case of delay
action set in a particular state, resulting in the system’s tran- violation, i.e., dvij = 1, the agent will then select the fog node
 6302, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1155/2022/4218696 by Jordan Hinari NPL, Wiley Online Library on [03/01/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
6 Wireless Communications and Mobile Computing

Input: SðkÞ, AðkÞ, R, T, F, C

Output: Delay violations, security violations, optimal Q-table values and policy
1 while All IoT tasks i ∈ T are scheduled do
2 Obtain TFi by slicing the IoT task i based on security requirement TSi and task size Tsi ;
3 Use ε − greedy policy Equation (16) to select action ai ∈ AðkÞ;
4 Get response time RPij of selected fog node j using (10).
5 Get FR j , FS j , FP j of selected fog node j.;
6 Calculate reward RWDkj using (14)
7 Use (17) to update the Q-table;
8 Update the available resource state R, system states SðkÞ, time step k
9 end while

Algorithm 1: Q-Learning Based Load Balanced Task Scheduling.

n o
Table 2: Parameter configuration. F ∗j = arg maxn RWDkj :
F j ∈F i
ð15Þ
Parameter Value
Episodes 10000 4.5. Q-Learning Policy. We use the most used RL method,
ε 0.5 i.e., Q-learning, to obtain the policy. The goal is to enable
α
the agent to learn from the environment and select the best
0.4
action under a particular state to maximize its rewards. The
γ 0.6 policy πðsÞ dictates the agent which action to select at some
DEi 50-500 ms particular state. To keep a balance between exploration and
BWij 1 Gbps exploitation, we use ε − greedy policy [19] in our model as
follows:
Dsij 5-1000 m
(
FR j 75-85 units a ∈ argMaxQi ðs, ai Þ, probability of 1 − ε,
πðsÞ =
TRi 5-9 units a random action ai , otherwise,
ð16Þ

with the least response time at that particular time step k, where ε is the probability of taking a random action in an
avoiding scheduling the task to a busy fog node with heavy ε − greedy policy. Exploration allows the agent to explore
load and thus balancing the system load. The considerable the environment by trying multiple actions and learning
negative penalty for delay violation forces the agent not to the best results for those actions. After learning next time,
schedule task i to a fog or cloud node whose response time i.e., k + 1, it exploits those values to schedule optimally. In
exceeds the delay requirement of IoT task i. Q-learning algorithm, the Bellman equation [19] is used to
To avoid greedy decisions when achieving the load- update the state-action paired values of the Q-table:
balanced environment, the long-term reward is considered
rather than short-term rewards by considering the expected Qπ ðsðkÞ, aðkÞÞ = ð1 − αÞQðsðkÞ, aðkÞÞ
cumulative future discounted reward for fog node j at time  ð17Þ
step k, i.e., RWDkj , computed as follows: + α RWDkj + γ max Qðsðk + 1Þ, aðk + 1ÞÞ,

" # where α is the learning rate that determines how much our
T agent cares about the previously learnt information. If α =
RWDkj = Eλ 〠 βt RWDas ðk + t + 1Þ , ð14Þ 1, the agent will override the most recent Q value rather than
t=0 learning from past values. A too-small α will result in slow
learning; α = 0 will restrict the agent from updating the old
values. γ is the discount factor, defining the nature of the
where T is the future time slot when the reward is deter- agent regarding long-term or short-term rewards while mak-
mined and β is the discounting factor. Therefore, the agents ing decisions.
try to select those actions that can maximize the sum of
rewards it will receive [19]. Specifically, based on Equation 4.6. Q-Learning-Based Algorithm for Scheduling. This section
(14), the agent determines the discounted cumulative future introduces the proposed Q-learning-based algorithm as
reward for all the fog nodes in its vicinity and activates the shown in Algorithm 1 in the pseudocode. It takes as input
fog node with the highest reward. The optimal fog node the system state space SðkÞ, action space AðkÞ, states of avail-
can be defined as follows: able resources R, set of IoT tasks T, set of fog nodes F, and
 6302, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1155/2022/4218696 by Jordan Hinari NPL, Wiley Online Library on [03/01/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Wireless Communications and Mobile Computing 7

350 35
325
301 299 32%
298

Ratio of delay violations (%)

300 30
268 260
Average delay (ms)

250 240 25
221 220
201 203 201 21%
200 20
15% 17%
156 158 17%
150 15 14%
110 110 13%
12%
100 85 90 10 10%
75 78
5% 6% 6% 5%
50 5 3% 3% 4% 4%
1% 2% 2%
0 0
100 200 300 400 500 100 200 300 400 500
Total tasks Total tasks

Non-sliced (Ratio 1:1:1) Non-sliced (Ratio 1:1:3) Load balancing (Ratio 1:1:1)
Sliced (Ratio 1:1:1) Sliced (Ratio 1:1:3) W/o load balancing (Ratio 1:1:1)
Load balancing (Ratio 1:1:3)
Figure 2: Average delays of sliced vs. nonsliced tasks with different W/o load balancing (Ratio 1:1:3))
ratios of security requirements.
Figure 5: Delay violations with and without load balancing for
different ratios of security requirement.
400
361
350
323
300 action by selecting a combination of the fog nodes or cloud
Average delay (ms)

265 260
250 235 to serve all the fragments of IoT task from the available fog
220
200 201
158
171 185 nodes and cloud server. The agent receives the response
150 132
110 118 110 time, available resources, security level, and processing
100 81 85 90
75 73 78 power of each selected fog node and then calculates the
50
reward based on the response time and security adherence
0
100 200 300 400 500 the selected fog node can provide to the IoT task. Then,
Total delay the agent updates the Q-table values, available resource state,
Non-sliced with load balancing
and system state.
Non-sliced w/o load balancing
Sliced with load balancing 5. Performance Evaluation
Sliced w/o load balancing
5.1. Simulation Environment. We used a Jupyter Notebook
Figure 3: Average delays of sliced vs. nonsliced tasks with and on an Intel Core i7 system with 16-GB RAM and an Nvidia
without load balancing approach.
GTX-1650 dedicated graphics card to evaluate the Q-
learning algorithm. Table 2 presents the parameters and
70
65%
their values used for simulation. Assume nine fog nodes
and one cloud server in the environment. The number of
Ratio of security violations (%)

60

50 48%
tasks varies from 100 to 500. The maximum delay threshold
40 39%
43% DEi for IoT tasks is set in a range of 50 ms to 500 ms. Link
bandwidth BWij of the link from IoT task i source to fog
36%
30 31%
25%
20
24%
17%
nodes j is assumed to be 1 Gbps, while the link between
10
11%
8%
11% fog nodes and cloud is assumed to be 4 Gbps [20]. The sizes
5%
0
2% 3% of IoT task Tsi are distributed as 10% ranging from 500 Mb
100 200 300 400 500 to 1 Gb, 20% ranging from 500 Kb to 1 Mb, and 70% ranging
Total tasks from 1 to 50 Mb. The distance Dsij between IoT task source
W/o security (Ratio 1:1:1) devices and fog nodes is assumed to be within 1000 m. The
W/o security (Ratio 1:1:3) resource units of fog nodes FR j range from 75 to 85, while
With security (Ratio 1:1:3)
the resource units required by the IoT tasks range from 5
Figure 4: Security violations with and without security constraints to 9.
for different ratios of security requirements.
6. Results and Analysis
cloud C. The objective of the algorithm is to select an opti- Figure 2 evaluates the system load balancing in terms of the
mal action for the agent so that the best fog node or cloud average delay under different numbers of IoT tasks, i.e., 100
for offloading the IoT task is picked. Regarding this, the out- to 500, for schemes with and without IoT task slicing based
put of our algorithm is the optimal policy and Q-table on their security requirements. We consider the ratio of IoT
values. tasks with low, medium, and high security levels as 1 : 1 : 1
For IoT task i, the agent first converts the task into TFi and 1 : 1 : 3, respectively. It can be noticed that the average
number of fragments based on the security requirements delay of the proposed approach is larger as compared to
and task size as shown in line 2. The agent then takes an the approach that does not consider the IoT task slicing.
 6302, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1155/2022/4218696 by Jordan Hinari NPL, Wiley Online Library on [03/01/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
8 Wireless Communications and Mobile Computing

The reason is that the task, when sliced, should be scheduled Data Availability
to different fog nodes, suffering from more complexity due
to satisfying the delay and security requirement of each The data of our paper is generated in our own lab based on
IoT task piece and thus leading to longer overall average the requirements of our simulation scenarios and we have
delay. Nonetheless, the delay is still within the required not used any public data. Nonetheless, we declare that once
range of general real-time tasks. Moreover, the delay for our paper is accepted, we will share our data by uploading it
the tasks with a higher ratio (i.e., schemes of 1 : 1 : 3) of high to the submission portal as a supportive file or anywhere
security requiring tasks is larger than the one with equal dis- required by the WCMC journal.
tribution of low, medium, and high security requiring tasks
(i.e., schemes of 1 : 1 : 1). This is due to security constraints
in place, since the agent must schedule the high security Conflicts of Interest
requiring tasks to the fog nodes with high security levels The authors declare that they have no conflicts of interest.
only, which are lesser in number. Hence, the tasks must be
queued first at those fog nodes, which add to the average
delay. Acknowledgments
Figure 3 compares the average end-to-end delay for
schemes with and without IoT task slicing and load balan- This work was supported by the National Research Founda-
cing considerations. It is evident that the task offloading tion of Korea (NRF) grant funded by the Korean Govern-
schemes taking load balancing into consideration, i.e., non- ment (Ministry of Education) (Grant No.
sliced with load balancing and sliced with load balancing, 2020R1I1A3072688).
have a lesser average delay than those without a load balan-
cing approach, i.e., nonsliced w/o load balancing and sliced
w/o load balancing. The difference is not that much for a References
small number of tasks, but as the number of tasks increases
from 300 to 500, the difference becomes more considerable. [1] T. Cisco and A. Internet, “Cisco: 2020 CISO benchmark
Figure 4 shows the performance of security violations for report,” Computer Fraud Security, vol. 2020, no. 3, p. 4, 2020.
schemes with and without security considerations. The [2] G. Peralta, M. Iglesias-Urkia, M. Barcelo, R. Gomez, A. Moran,
result figure shows that if not consider the security require- and J. Bilbao, “Fog computing based efficient IoT scheme for
ment, the agent will end up scheduling higher or medium the Industry 4.0,” in Proceedings of the 2017 IEEE International
Workshop of Electronics, Control, Measurement, Signals and
security requiring tasks to fog nodes with lesser security
their Application to Mechatronics, ECMSM 2017, pp. 1–6,
credit, reflected by high ratio of security violations with Donostia, Spain, 2017.
increasing IoT task numbers, which threatens the task pri- [3] F. Bonomi, R. Milito, J. Zhu, and S. Addepalli, “Fog computing
vacy. Moreover, the security violations increase when the and its role in the internet of things,” in MCC’12- Proceedings
ratio of high security requiring tasks increases from 1 : 1 : 1 of the 1st ACM Mobile Cloud Computing Workshop, pp. 13–15,
(33%) to 1 : 1 : 3 (60%). On the other hand, our proposed Helsinki, Finland, 2012.
model performs much better with even 60% high security [4] T. Wang, J. Zhou, X. Chen, G. Wang, A. Liu, and Y. Liu, “A
requiring tasks. three-layer privacy preserving cloud storage scheme based on
Figure 5 shows the ratio of delay violations for schemes computational intelligence in fog computing,” IEEE Transac-
with and without load balancing considerations for IoT tasks tions on Emerging Topics in Computational Intelligence,
with low to medium to high security ratios of 1 : 1 : 1 and vol. 2, no. 1, pp. 3–12, 2018.
1 : 1 : 3, respectively. For both ratios, it is evident that our [5] J. Z. TianWang, M. Z. A. Bhuiyan, H. Tian, Y. Cai, Y. Chen,
proposed model with security constraints and load balancing and B. Zhong, “Trajectory privacy preservation based on a
approach performs better with fewer delay violations than fog structure for cloud location services,” IEEE Access, vol. 5,
the approach that does not consider load balancing. pp. 7692–7701, 2017.
[6] Z. Ning, S. Sun, X. Wang et al., “Blockchain-enabled intelligent
transportation systems: a distributed crowdsensing frame-
7. Conclusion work,” IEEE Transactions on Mobile Computing, vol. 1233,
pp. 1–17, 2021.
[7] A. J. Kadhim and J. I. Naser, “Proactive load balancing mech-
This paper proposed an IoT task offloading algorithm based
anism for fog computing supported by parked vehicles in IoV-
on Q-learning that forces the agent to schedule the incoming
SDN,” China Communications, vol. 18, no. 2, pp. 271–289,
IoT tasks to fog nodes with lightweight traffic loads and sat- 2021.
isfactory security credit; while in task scheduling, the agent [8] M. Verma, N. Bhardwaj, and A. K. Yadav, “Real time efficient
fragmentedanIoT task first based on its security requirement scheduling algorithm for load balancing in fog computing
and size and then scheduled the task fragments to fog nodes environment,” International Journal of Information Technol-
with equal or higher security credits than the task fragments’ ogy and Computer Science, vol. 8, no. 4, pp. 1–10, 2016.
security level. During this process, the agent tried to balance [9] F. Chiti, R. Fantacci, and B. Picano, “A matching theory frame-
the load of the network by considering the response time of work for tasks offloading in fog computing for IoT systems,”
fog nodes for each particular IoT task, which reduced the IEEE Internet of Things Journal, vol. 5, no. 6, pp. 5089–5096,
overall average delay while ensuring security constraints. 2018.
 6302, 2022, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1155/2022/4218696 by Jordan Hinari NPL, Wiley Online Library on [03/01/2025]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License
Wireless Communications and Mobile Computing 9

[10] A. Yousefpour, G. Ishigaki, R. Gour, and J. P. Jue, “On reduc-

ing IoT service delay via fog offloading,” IEEE Internet of
Things Journal, vol. 5, no. 2, pp. 998–1010, 2018.
[11] X. Wang, Z. Ning, S. Guo, M. Wen, and V. Poor, “Minimizing
the age-of-critical-information: an imitation learning-based
scheduling approach under partial observations,” IEEE Trans-
actions on Mobile Computing, vol. 1233, p. 1, 2021.
[12] Z. Ning, Y. Yang, X. Wang et al., “Dynamic computation off-
loading and server deployment for UAV-enabled multi-
access edge computing,” IEEE Transactions on Mobile Com-
puting, vol. 1233, pp. 1–1, 2021.
[13] Z. Ning, P. Dong, M. Wen et al., “5G-enabled UAV-to-
community offloading: joint trajectory design and task sched-
uling,” IEEE Journal on Selected Areas in Communications,
vol. 39, no. 11, pp. 3306–3320, 2021.
[14] X. Wang, Z. Ning, S. Guo, M. Wen, L. Guo, and V. Poor,
“Dynamic UAV deployment for differentiated services: a
multi-agent imitation learning based approach,” IEEE Trans-
actions on Mobile Computing, vol. 1233, pp. 1–16, 2021.
[15] G. Zhang, F. Shen, Z. Liu, Y. Yang, K. Wang, and M. T. Zhou,
“FEMTO: fair and energy-minimized task offloading for fog-
enabled IoT networks,” IEEE Internet of Things Journal,
vol. 6, no. 3, pp. 4388–4400, 2019.
[16] Y. Yang, K. Wang, G. Zhang, C. Xu, X. Luo, and M. T. Zhou,
“Maximal energy efficient task scheduling for homogeneous
fog networks,” in IEEE INFOCOM 2018 - IEEE Conference
on Computer Communications Workshops (INFOCOM
WKSHPS)no. 5, pp. 274–279, HI, USA, 2018.
[17] F. Jalali, K. Hinton, R. Ayre, T. Alpcan, and R. S. Tucker, “Fog
computing may help to save energy in cloud computing,” IEEE
Journal on Selected Areas in Communications, vol. 34, no. 5,
pp. 1728–1739, 2016.
[18] C. Tofallis, S. I. Gass, and C. M. Harris, “Encyclopedia of oper-
ations research and management science,” Journal of the Oper-
ational Research Society, vol. 48, no. 7, pp. 759-760, 1997.
[19] R. S. Sutton and A. G. Barto, Reinforcement Learning: An
Introduction, The MIT Press, 2nd edition, 2018.
[20] J. Hecht, “Faster fiber links for data centers,” 2019, December
2021, https://spectrum.ieee.org/tech-talk/telecom/internet/
fasterfiber-links-for-data-centers.