‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Algorithms Design

Linear time sorting

 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

More about sorting

• How many sorting algorithms do you
know?
• What are their time complexity?
• What’s common about them?
• Can we do better than Θ(n log n)?
• Yes and no
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Outline
• More about sorting
– Theoretical lower-bound
– Linear-time sorting algorithms
– Stability of sorting
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Theoretical lower-bound
• Comparison sort: determines the relative
order of two elements only by comparison
– What else can you do …

– The theoretical lower-bound for any

comparison-based sorting algorithm is
Θ(nlogn)
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Lower-bound of comparison-based sort

• Assume an array A with 3 distinct

elements, a1, a2, and a3
• Use insertion sort 1 < 2?
• # comparisons? yes no

• A = [9 6 5] 2 < 3? 1<3?
no no
• A = [5 6 9] yes yes
(1, 2, 3) 1 < 3? (2, 1, 3) 2 < 3?
• A=… yes yes no no

(1, 3, 2) (3, 1, 2) (2, 3, 1) (3, 2, 1)

 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Lower-bound of comparison-based sort

• Assume all elements are distinct,

each comparison has two possible
outcomes: ai < aj or ai > aj
• Based on the outcome, change the A
relative order of some elements ai < aj ai > aj
• Output is a permutation of the input
• A correct sorting algorithm can A’ A’’
handle any arbitrary input log(n!)

• n! possible permutations
• Therefore, at least log(n!) =
Θ(nlogn) comparisons in the worst
case

Θ(n!)
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Sorting in linear time

• Is there a problem with the theory?
• No. We are going to sort without doing
comparison
• How is that possible?
• Key: knowledge about the data
– The more you know about your data, the
more likely you can have a better algorithm
– Example: Almost sorted? All distinct? Many
identical ones? Uniformly distributed?
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Counting sort
• Knowledge: the numbers fall in a small range
• Example 1: sort the final exam score of a large
class
– 1000 students
– Maximum score: 100
– Minimum score: 0
– Scores are integers
• Example 2: sort students according to the first
letter of their last name
– Number of students: many
– Number of letters: 26
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Counting sort

• Input: A[1 . . n], where A[ j]{1, 2, …, k} .

• Output: B[1 . . n], sorted.
• Auxiliary storage: C[1 . . k] .

• Not an in-place sorting algorithm

• Requires  (n+k) additional storage besides the
original array
 ‫ﺩﺭﺱ‪ :‬ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‪ ،‬ﺍﺳﺘﺎﺩ‪ :‬ﺩﻭﻟﺘﺸﺎﻫﻰ‬

‫‪Intuition‬‬
‫•‬ ‫‪S1: 100‬‬ ‫‪0‬‬ ‫‪85‬‬ ‫‪90‬‬ ‫‪100‬‬
‫•‬ ‫‪S2: 90‬‬
‫•‬ ‫‪S3: 85‬‬ ‫‪S3‬‬ ‫‪S2‬‬ ‫‪S1‬‬

‫•‬ ‫‪S4: 100‬‬ ‫‪S5‬‬ ‫‪S4‬‬

‫•‬ ‫‪S5: 90‬‬
‫•‬ ‫…‬
‫‪… S3 … S2, S5, …, S1, S4‬‬
‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Intuition
75 85 90 100
1 1 2 2

50 students with score ≤ 75

What is the rank (lowest to highest) for a student with score = 75?
50

200 students with score ≤ 90

What is the rank for a student with score = 90? 200 or 199
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Counting sort
1. for i  1 to k Initialize
do C[i]  0
2. for j  1 to n Count

do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|

3.for i  2 to k Compute running sum

do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key  i}|

4.for j  n downto 1 Re-arrange

do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 ‫ﺩﺭﺱ‪ :‬ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‪ ،‬ﺍﺳﺘﺎﺩ‪ :‬ﺩﻭﻟﺘﺸﺎﻫﻰ‬

‫‪Counting-sort example‬‬
‫‪1‬‬ ‫‪2‬‬ ‫‪3‬‬ ‫‪4‬‬ ‫‪5‬‬ ‫‪1‬‬ ‫‪2‬‬ ‫‪3‬‬ ‫‪4‬‬

‫‪A:‬‬ ‫‪4‬‬ ‫‪1‬‬ ‫‪3‬‬ ‫‪4‬‬ ‫‪3‬‬ ‫‪C:‬‬

‫‪B:‬‬
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 1: initialization
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 0 0 0 0

B:

1. for i  1 to k
do C[i]  0
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 2: count
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 0 0 0 1

B:

2. for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 2: count
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 0 1

B:

2. for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 2: count
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 1 1

B:

2. for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 2: count
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 1 2

B:

2. for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 2: count
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 2 2

B:

2. for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 3: compute running sum

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 2 2

B: C': 1 1 2 2

3. for i  2 to k
do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key  i}|
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 3: compute running sum

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 2 2

B: C': 1 1 3 2

3. for i  2 to k
do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key  i}|
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 3: compute running sum

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 2 2

B: C': 1 1 3 5

3. for i  2 to k
do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key  i}|
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 4: re-arrange
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 1 3 5

B: 3 C': 1 1 3 5

4. for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 4: re-arrange
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 1 3 5

B: 3 C': 1 1 2 5

4. for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 4: re-arrange
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 1 2 5

B: 3 4 C': 1 1 2 5

4. for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 4: re-arrange
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 1 2 5

B: 3 4 C': 1 1 2 4

4. for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 4: re-arrange
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 1 2 4

B: 3 3 4 C': 1 1 2 4

4. for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 4: re-arrange
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 1 2 4

B: 3 3 4 C': 1 1 1 4

4. for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 4: re-arrange
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 1 1 4

B: 1 3 3 4 C': 1 1 1 4

4. for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 4: re-arrange
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 1 1 4

B: 1 3 3 4 C': 0 1 1 4

4. for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 4: re-arrange
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 0 1 1 4

B: 1 3 3 4 4 C': 0 1 1 4

4. for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 ‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Loop 4: re-arrange
1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 0 1 1 4

B: 1 3 3 4 4 C': 0 1 1 3

4. for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Analysis
1. for i  1 to k
(k) do C[i]  0
2.for j  1 to n
(n) do C[A[ j]]  C[A[ j]] + 1
3.for i  2 to k
(k) do C[i]  C[i] + C[i–1]
4.for j  n downto 1
(n) do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
(n + k)
‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Running time
If k = O(n), then counting sort takes (n) time.
• But, theoretical lower-bound sorting takes
W(n log n) time!
• Problem with the theory?
Answer:
• Comparison sorting takes W(n log n) time.
• Counting sort is not a comparison sort.
• In fact, not a single comparison between
elements occurs!
‫ ﺩﻭﻟﺘﺸﺎﻫﻰ‬:‫ ﺍﺳﺘﺎﺩ‬،‫ ﻃﺮﺍﺣﻰ ﺍﻟﮕﻮﺭﻳﺘﻢ‬:‫ﺩﺭﺱ‬

Stable sorting
Counting sort is a stable sort: it preserves
the input order among equal elements.

A: 4 1 3 4 3

B: 1 3 3 4 4

Why this is important?

What other algorithms have this property?