CMPT 305: Computer Simulation/Modelling

Spring 2022

Introduction to Queuing Models

Alaa Alameldeen

© Copyright 2022 Alaa Alameldeen and Joseph Peters

 Servers/Service Facilities
• A service facility consists of:
➢One or more servers
➢A Queuing discipline (e.g. FCFS)
➢System capacity (throughput)
• An entity(resource) is a component of the system that requires
explicit representation in the model (e.g. server, customers)
• An attribute is a property of an entity (e.g. capacity of a server)
• A state is a set of variables that describe the system at a given
point in time (e.g. number of customers in system)

2
 Queuing Notation
• A service facility can be modelled by a Queuing system consisting
of: S
A
1. Arrival process A
2. Service time distribution S
m=1
3. Number of servers m
4. System capacity (number of buffers) B
5. Population size K
6. Service discipline SD
• This is usually expressed using the Kendall notation:
A/S/m/B/K/SD
3
 Arrival Process
• Arrival times of customers are t1, t2, …., tN
• Interarrival times: i = ti – ti-1, i = 2, 3, …, N
𝒊=𝟏 𝒊
σ𝑵
• Mean interarrival time  =
𝑵
• Mean arrival rate : Number of arrivals per unit of time (e.g., second)
𝑵 𝟏
 = σ𝑵 =
𝒊=𝟏 𝒊 𝒎𝒆𝒂𝒏 𝒊𝒏𝒕𝒆𝒓𝒂𝒓𝒓𝒊𝒗𝒂𝒍 𝒕𝒊𝒎𝒆

• Common assumption: Arrival times are a sequence of independent and identically-

distributed (IID) random variables
𝟏
=
𝑬()
• Poisson arrival process: Interarrival times are IID and exponentially distributed
• Infinite population model: Pattern of customer arrivals based on users’ combined
behavior
• Arrival rate can depend on the state of the system
4
 Service Time Distribution
• Service times of customers: s1, s2, …, sN
• Service requirement is specified in terms of units of work
➢ Instructions for CPUs, Packets for a network channel
• Server capacity (throughput): Units of work per unit of time (e.g., second)
𝒔𝒆𝒓𝒗𝒊𝒄𝒆_𝒓𝒆𝒒𝒖𝒊𝒓𝒆𝒎𝒆𝒏𝒕𝒊
• Service time sl =
𝑺𝒆𝒓𝒗𝒆𝒓 𝒄𝒂𝒑𝒂𝒄𝒊𝒕𝒚

• Service rate: Customers served per second (assuming no idle time)

σ𝑵
𝒊=𝟏 𝒔𝒊
• Mean service time 𝒔ത =
𝑵
𝑵 𝟏
• Mean service rate µ = σ𝑵 = 𝒎𝒆𝒂𝒏 𝒔𝒆𝒓𝒗𝒊𝒄𝒆 𝒕𝒊𝒎𝒆
𝒊=𝟏 𝒔𝒊

• Common assumption: Service times are a sequence of independent and identically-

distributed (IID) random variables
𝟏
µ = 𝑬(s)

• Service rate for m identical servers (with one queue) is mµ

5
 Service Disciplines
• First-Come First-Serve (FCFS)
• Round-robin (RR) with fixed-size quantum.
➢Processor sharing (PS) if quantum small because each of n jobs gets 1/n
of CPU
• Infinite Server (IS) or Delay center has fixed delay (e.g. satellite
communication link)
• Service discipline can depend on service times
➢Shortest Processing Time First (SPT)
• Priority class order (for priority queues)

6
Distributions for Interarrival and Service Times
• M Exponential
➢ Memoryless distribution: if interarrival times are exponentially distributed with mean µ=1/ then time to
next arrival is µ= 1/  independent of time since last arrival
−𝑥/µ
PDF: f(x) =  e -x = 𝑒 ൗµ f(x)
➢ Mean: µ=1/ ; variance: 1/2
• Ek Erlang with parameter k
➢ A distribution of a sum of k independent exponential variables, x
each with a mean of 1/ .
f(x) = k xk-1 e -x / (k-1)!
➢ Mean: : k/ ; variance: k/2
➢ When k=1, it’s the same as the exponential distribution
• D Deterministic (constant times, variance = 0)
• G General (distribution not specified)
• For a system with infinite capacity B, infinite population K, and FCFS service discipline:
M/M/m///FCFS or M/M/m in short
Arrival Distribution (Exponential) / Service Distribution (Exponential) / # of servers / Capacity / Population / Service Discipline

7
 Common Random Variables
• : Interarrival time
• : Mean arrival rate
• 1/: Mean interarrival time
• s: Service time per job (customer)
• µ: Mean service rate per server
• 1/µ: Mean service time per server
• nq: Number of jobs (customers) waiting for service (waiting in queue)
• ns: Number of jobs (customers) receiving service
• n = nq + ns : Number of jobs (customers) in system
• w: Waiting time (queuing delay)
• r: Response (Residence) time = time in system = w + s
• U: Utilization (time in use / total time)
8
 Impact of Utilization on Open Queues
• Example: Single-server open queue

exponential distribution graph

➢ As utilization increases with

arrival rate, response time and
queue length both increase
exponentially
Figures from LZGS Chapter 1
9
Fundamental Laws

10
 Definitions
Completions
• In this abstract system: Arrivals
System
➢T: Length of time we observe system
➢A: Observed number of request arrivals
➢C: Observed number of request completions
➢B: Length of time a resource is busy (for systems with a single resource/server)
𝑨
• Mean Arrival rate  =
𝑻
𝑪
• Throughput X =
𝑻
𝑩
• Utilization U =
𝑻
𝑩
• Average Service Time per request S =
𝑪

11
 The Utilization Law
𝑩 Arrivals Completions
System
Since U = ;
𝑻
𝑪 𝑩
X= ; S=
𝑻 𝑪

Utilization Law: U = 𝑿𝑺

• That is, the utilization of a server (resource) is the product of

the server throughput and the average service time

12
Utilization Law: Example 1
• An online warehouse has a throughput of 12,000 transactions
per second. Each transaction requires 10 microseconds of
service on average. What is the fraction of time that the
warehouse is utilized?
X = 12,000 transactions / second
S = 10 microseconds = 10 x 10-6 seconds
U = X S = 12,000 x 10 x 10-6 = 0.12 = 12%

So the warehouse is utilized 12% of the time on average.

13
Utilization Law: Example 2
• A gas station with a single gas pump requires an average of 3
minutes to serve a single customer. The pump is occupied
45% of the time. How many customers per hour are served in
the station?
U = 45% = 0.45
S = 3 minutes
X = U / S = 0.45 / 3 = 0.15 customers/minute
= 0.15 x 60 = 9 customers/hour

So the gas station serves 9 customers per hour on average.

14
 Little’s Law 1 # of requests in the system = distance between the arrivals and completions
shaded area = accumulated time in the system

• At any point in time, the distance between the arrivals and completions
functions is the number of requests in the system
• Over any time interval, the area between arrival and completion functions
is the accumulated time (W) in the system during that interval
Example: Shaded area represents the accumulated time W=2 between T=2
and T=6
𝑾
• From W’s definition: Average number of requests in the system N =
𝑻

Example: In the observed 4 minutes, the average number of requests in the

system is 2/4 = 0.5
𝑾
• The average system response (residence) time per request: R =
(important formula) 𝑪

Example: in the graph, 8 requests complete in the observation interval

between T=2 and T=6, so average response time per request is 2/8 = 0.25
• From the above two definitions:
N=
𝑾
𝑻
=
𝑪𝑾
𝑻 𝑪
=XR Little’s Law: N = 𝑿𝑹
• That is, the number of requests in a system is the product of the server
throughput and the average response (residence) time

1 Named after MIT Professor John Little who formulated the law in 1954 15
 Little’s Law
• Can be applied to any system with very few assumptions
➢Can help check consistency of measurement data
• Can help figure out any of the three values if we know the
other two:
➢ N = XR; R = N/X; X = N/R
• Can be applied to different levels in a system
➢A single resource
➢A subsystem
➢The whole system

16
 Little’s Law: Example 1
• In a convenience store with a single checkout counter, 40
customers are served every hour on average, and each customer
stays in the store 6 minutes on average. What is the average
number of customers in the store at any point during that hour?
X = 40 customers / hour
R = 6 minutes = 0.1 hours
N = X R = 40 x 0.1 = 4 customers

So the store has an average of 4 customers at any point inside the

store.
17
Little’s Law: Example 2
• An average of 10 customers are inside a bank asking for teller
service during the day. Each customer spends 12 minutes (on
average) inside the bank. How many customers per hour are
served by all tellers?
N = 10 customers
R = 12 minutes = 12/60 = 0.2 hours
X = N / R = 10 / 0.2 = 50 customers / hour

So all bank tellers serve an average of 50 customers per hour

18
 Little’s Law: Example 3
• Between 10 AM and 2 PM, a web server has 100 requests on
average either being served or waiting for service. The server
completes 100,000 requests per second on average. What is
the average response time for each request?
N = 100 requests
X = 100,000 requests/second
R = N / X = 100 / 100,000 = 0.001 seconds = 1 ms

So each request completes in 1 millisecond on average

19
 Little’s Law: Example 4
• 1000 cars, on average, are at a busy intersection at any point in time
during morning rush hours between 7 AM and 9 AM. 800 cars per
minute exit that intersection on average during this time. Each car
crosses the intersection in 10 seconds when light is green. What is
the average time each car is waiting at a red light? Kelly, Colleen. nyc11.jpg. 2009 Pics4Learning.com

N = 1000 cars
X = 800 cars/minute
R = N / X = 1000 / 800 = 1.25 minutes = 75 seconds
Wait time = Response time – Service time ( since R = w + s )
(important formula)
= 75 – 10 = 65 seconds

So each car spends 65 seconds on average waiting at a red light

20
 Little’s Law: Example 5
• A computer system with a single CPU runs sequential programs with variable service time
requirements. For example, programs can have total runtimes of 5, 0.2, 30, 5000 milliseconds.
Prove that the “Shortest Job First” scheduling algorithm leads to the minimum average response
time per program.
Assume we have P programs to run, and the total runtime to run them all is T. Assume the programs run in
the order t1, t2, … , tp
T = σ𝑷
𝒊=𝟏 𝒕𝒊 Where ti is the runtime for program i
System throughput X = total programs / total time = P/T
Response time per program i = 𝑹𝒊 = σ𝒊𝒋=𝟏 𝒕𝒋
𝒊
σ𝑷
𝒊=𝟏 𝑿 𝑹𝒊
𝑿 σ𝑷
𝒊=𝟏 σ𝒋=𝟏 𝒕𝒋
Average number of programs in system N = =
𝑷 𝑷
𝑿
= 𝑷 𝒕𝟏 + 𝒕𝟏 + 𝒕𝟐 + 𝒕𝟏 + 𝒕 𝟐 + 𝒕𝟑 + ⋯ + 𝒕𝟏 + 𝒕𝟐 + … + 𝒕𝑷
𝑿
= 𝑷𝒕𝟏 + 𝑷 − 𝟏 𝒕𝟐 + ⋯ + 𝒕𝑷
𝑷
𝑵 𝟏
Average response time per program R = 𝑿
= 𝑷 𝑷𝒕𝟏 + 𝑷 − 𝟏 𝒕𝟐 + ⋯ + 𝒕𝑷

21
 Little’s Law: Example 5 (Cont)
𝑵 𝟏
Average response time per program R = = 𝑷𝒕𝟏 + 𝑷 − 𝟏 𝒕𝟐 + ⋯ + 𝒕𝑷
𝑿 𝑷
Assume ti’s are ordered from smallest to largest leading to Response time RSJF.
That is, for every k > j : tk ≥ tj
For any other order O of ti’s (say switch tj and tk where k > j) Response time RO
𝟏 𝟏
= RSJF – 𝑷 − 𝒋 + 𝟏 𝒕𝒋 + 𝑷 − 𝒌 + 𝟏 𝒕𝒌 + 𝑷 − 𝒌 + 𝟏 𝒕𝒋 + 𝑷 − 𝒋 + 𝟏 𝒕𝒌
𝑷 𝑷
𝟏
= RSJF + 𝒌 − 𝒋 𝒕𝒌 − 𝒌 − 𝒋 𝒕𝒋
𝑷
𝟏
= RSJF + 𝒌 − 𝒋 (𝒕𝒌 −𝒕𝒋
𝑷
Since k > j and tk ≥ tj and P ≥ 1
Therefore for any other order O except SJF: RSJF ≤ RO
22
Little’s Law with Think Time servers

.
(Think) Centers
.
.

• Average think time per customer: Z

➢A customer spends, on average, time R waiting in queue and being served,
and time Z outside of queues and service

Little’s Law with think time : N = 𝑿(𝑹 + 𝒁)

23
Recap Fundamental Laws
Utilization Law: U = 𝑿𝑺

Little’s Law: N = 𝑿𝑹
servers

.
(Think).Centers
.

Little’s Law with think time : N = 𝑿(𝑹 + 𝒁)

24
Response Time Law
(Derived from Little's law with think time extension)
servers

.
(Think) Centers
.
.

• Response time (queuing + service) in a system with think

centers needs to discount think time Z
𝑵
Response Time Law: 𝑹= −𝒁
𝑿

25
 Response Time Law: Example 1
• A web server serves a total of 1 million subscribers to a streaming
service. The server completes 20,000 requests per hour on
average. Each subscriber, on average, waits two days after
finishing one request before requesting service again. What is the
average response time for each request?
N = 1,000,000 customers .
.
X = 20,000 requests/hour .
Z = 48 hours
R = N / X - Z = 1,000,000 / 20,000 - 48 = 2 hours

So each request completes in 2 hours on average.

26
 Response Time Law: Example 2
• A hotel has an open buffet restaurant that serves breakfast between 6 AM and 9
AM daily. On average, 80% of all hotel guests eat breakfast at the restaurant.
Each guest, on average, spends 30 minutes per day at the restaurant during
breakfast hours. The restaurant serves 200 guests per hour on average during
breakfast hours. What is the average number of guests staying at the hotel?
X = 200 guests/hour
R = 0.5 hours .
Z = 2.5 hours (time not at restaurant) .
.
N = X (R + Z) = 200 x (0.5 + 2.5) = 600 guests
Since N represents 80% of hotel guests, total guests = 600 x 100/80 = 750

So an average of 750 guests are staying at the hotel.

27
 Response Time Law: Example 3
• A cloud service provider has an average CPU utilization of 60%
across all CPUs in its data center. Each job has an average service
requirement of 30 microseconds. The response time for each job is
0.2 seconds. The system has 1,000,000 users on average. What is
the average think time per user?
.
N = 1,000,000 users, R = 0.2 seconds .
S = 30 x 10-6 seconds, U = 0.6 .

X = U/S = 0.6/(30 x 10-6 ) = 20,000 jobs/second (Utilization Law)

Z = N / X - R = 1,000,000 / 20,000 – 0.2 = 49.8 seconds (RT Law)

So each user waits 49.8 seconds between requests.

28
Fundamental Laws Recap
𝑩 𝑩
Mean Arrival rate  = 𝑨/𝑻 Throughput X = 𝑪/𝑻 Utilization U = Average Service Time per request S =
𝑻 𝑪

Utilization Law: U = 𝑿𝑺
Little’s Law: N = 𝑿𝑹
servers

.
(Think) Centers
.
.

Little’s Law with think time : N = 𝑿(𝑹 + 𝒁)

𝑵
Response Time Law: 𝑹= −𝒁
𝑿 29
 Forced Flow Law
Arrivals Departures

• In a system where customers may receive service from different resources, potentially
more than once, before leaving system
If Vk is the visit count of resource k
𝑪𝒌
𝑽𝒌 = (Completions at resource k as a ratio of all system completions)
𝑪
which can be rewritten as: 𝑪𝒌 = 𝑽𝒌 𝑪
Since throughput X is the number of completions over an observed time interval:

The Forced Flow Law: 𝑿𝒌 = 𝑽𝒌 𝑿

30
 Service Demand
• The Forced Flow Law means that the throughput of a resource is the
product of overall system throughput and the average number of visits to
that resource.
• Let’s define Dk as the service demand at resource k
➢ Total service time for all visits by a customer or job to resource k
➢ Product of number of visits to resource and service time per visit
𝑫𝒌 = 𝑽𝒌 𝑺𝒌
From the Utilization Law: 𝑼𝒌 = 𝑿𝒌 𝑺𝒌
Combining with the Forced Flow Law: 𝑿𝒌 = 𝑽𝒌 𝑿
𝑼𝒌 = 𝑽𝒌 𝑺𝒌 𝑿 = 𝑫𝒌 𝑿

31
 Forced Flow Law: Example 1
• A database system is limited by disk accesses. The system has one disk that is
busy 40% of the time on average. Each transaction requires 50 disk accesses
on average, with an average service time of 5 milliseconds per access. How
many transactions per minute are executed on this database system?
Udisk = 40% = 0.4
Sdisk = 5 ms; Vdisk = 50
Ddisk = Vdisk Sdisk = 50 x 5 ms = 250 ms = 0.25 seconds
Since Udisk = Ddisk X
X = Udisk / Ddisk = 0.4 / 0.25 = 1.6 transactions/second
= 1.6 x 60 = 96 transactions/minute

So the database system has a throughput of 96 transactions per minute on

average.
32
 Forced Flow Law: Example 2
• In a database system limited by disk accesses, a system has 20 disks that
transactions can execute on. Each transaction requires 50 sequential disk
accesses and 5 milliseconds per access. The load is evenly split on all 20 disks,
and each disk is utilized at 40%. How many transactions are executed per
minute on this system?
Udisk = 40% = 0.4
Sdisk = 5 ms; Vdisk = 50/20 = 2.5
…
Ddisk = Vdisk Sdisk = 2.5 x 5 ms = 12.5 ms = 0.0125 seconds
X = Udisk / Ddisk = 0.4 / 0.0125 = 32 transactions/second
= 32 x 60 = 1920 transactions/minute

So the database system has a throughput of 1920 transactions per minute on

average.
33
 Forced Flow Law: Example 3
• A cloud service provider (CSP) provides a subscription service with 24 thousand
subscribers. When a subscriber requests service, they visit the CSP’s server 300 times
on average, with an average service time of 10 microseconds per access. The CSP’s
server is utilized 30% of the time. A subscriber waits 3 minutes between service
requests. What is the average response time per request?
UCSP = 30% = 0.3, Z = 3 minutes
SCSP = 10 us; VCSP = 300
DCSP = VCSP SCSP = 300 x 10 us = 3000 us = 0.003 seconds
Since UCSP = DCSP X
X = UCSP / DCSP = 0.3 / 0.003 = 100 requests/second
= 100 x 60 = 6,000 requests/minute
R = N/X – Z = 24,000/6,000 – 3 = 1 minute

So the average response time per request is one minute.

34
 Flow Balance Assumption
Arrivals Departures

• In a balanced system, we can assume that the number of arrivals is the same as the
number of completions over time

Flow Balance Assumption: 𝑨 = 𝑪 therefore  = 𝑿

• This allows us to use the arrival rate  instead of throughput X to model system
parameters
• In an unbalanced system, the number of customers in the system will increase infinitely
over time
35
 Stability Condition
• In an open system with m servers each with service rate µ and
an arrival rate , the system is stable if this condition is
satisfied:
 <mµ
( < µ for a single server)
• If  ≥ m µ, then the number of customers in the system N will
grow infinitely, and the response time will also grow infinitely:
Utilization Law: U = X S = ()(1/ m µ)
So if  ≥ m µ, U ≥ 1 which is an unstable condition
• A system with finite population is always stable since N doesn’t
grow beyond population size
• A system with finite capacity (e.g., #buffers) is always stable
since N doesn’t grow above the system capacity

36
 Demo of Unstable System
• An open system with a single server has a fixed arrival rate of 1 customer/second, and a
service rate of one customer every 2 seconds
Time Event N (#Customers in R (Response Time for departing
System) customer)
1 Cust 1 arrives 1
2 Cust 2 arrives 2
3 Cust 3 arrives, Cust 1 departs 2 2
4 Cust 4 arrives 3
5 Cust 5 arrives, Cust 2 departs 3 3
6 Cust 6 arrives 4
7 Cust 7 arrives, Cust 3 departs 4 4
8 Cust 8 arrives 5
9 Cust 9 arrives, Cust 4 departs 5 5

So N and R both grow infinitely over time

37
 Example 1: Is this system stable?
• An airline reservation system using one server receives 3000 reservation
requests per minute, and each request requires 2 seconds to process.
 = 3000 requests/minute = 3000/60 = 50 requests/second
µ = 1/2 = 0.5 requests/second
Since  ≥ µ, the system is unstable
• How many servers does the reservation system need to become stable?
For stability,  < m µ
m>/µ
m > 50/0.5 = 100
So we need more than 100 servers (i.e., at least 101) for the system to
become stable.
38
 Example 2: Flow Balance Assumption
• An airline reservation system using 120 servers receives 3000 /120
reservation requests per minute, and each request requires 2
seconds to process. What is the average utilization per server?  X
 = 3000 requests/minute
= 3000/60 = 50 requests/second
S = 2 seconds  µ = 1/2 = 0.5 requests/second
m = 120  m µ = 120 x 0.5 = 60
Since  < m µ, the system is stable  X =  = 50 request/second
Userver = Xserver Sserver = (50/120) x 2 = 0.833
• If the system serves 10,000 requests, on average, at any point in time,
what is the average response time?
N = 10,000, X = 3,000 requests/minute = 50 requests/sec
R = N/X = 10,000/50 = 200 seconds = 3.333 minutes
39
 Class Announcements
• HW 2 out. Due Jan 31.
➢Implement a FCFS queue, compute utilization, throughput and average response
time (also some other simple queue functions)
• Tuesday:
➢Bounds on performance. Only on Zoom (not in classroom)
• Friday
➢Exam 1 during class time 10:30-11:15.
➢Online on canvas. You need to be on the lecture zoom meeting with your camera
turned on (lecture will not be recorded)
➢Open book, notes, calculator
• We’ll start in-person lectures on Tuesday Feb 1 in C9002
➢Check your email or Piazza for any changes
➢I’ll still try to livestream on zoom and record lecture
40