0% found this document useful (0 votes)

124 views504 pages

Algodat 2 Merged

Algodat exam

Uploaded by

Danny AIrish

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

124 views504 pages

Algodat 2 Merged

Algodat exam

Uploaded by

Danny AIrish

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 504

Introduction

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria

Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at

Contact Details

Univ.-Prof. Dr. Alois Ferscha

Exercise DI (FH) Stefan Grünberger
DI Martin Schobesberger
Dr. Dari Trendafilov
Dr. Markus Weninger

Address Johannes Kepler University Linz

Institute of Pervasive Computing
Altenberger Straße 69, 4040 Linz
Phone: +43 (732) 2468 4760
Science Park 3, 6th floor

Contact [email protected]
Please always mention your student id (8 digits)
and the course id (6 digits) in the subject line!

Algorithms and Data Structures 2 // 2021W // 2 A. Ferscha

Course Details

Course VL Algorithms and Data Structures 2, 340300

Workload 3 ECTS, 2 hpw
Original study plan Bachelor's programme Computer Science (UK 033/521)
Education level B2 – Bachelor‘s programme, 2. year

Language English; Programming language is Java (34031x) or Python (34032x)

Study material Slides and additional materials will be available in the course section in
Moodle.
Evaluation Examination at end of term

Time & Place Mon, 13.45-15.15, HS 1

https://studienhandbuch.jku.at/125317

Algorithms and Data Structures 2 // 2021W // 3 A. Ferscha

Course Details :: 2021W - Lecture

How is the content provided?

• Presence mode in Linz
• Recordings in Moodle (available for full semester)
• Slides online in Moodle https://moodle.jku.at/jku/course/view.php?id=15436

Required days of physical presence: 1 (exam)

What are the evaluation criteria?

• final exam in Linz/Vienna/Bregenz [alt.: moodle exam with zoom supervision]

Introduction Mon, October 11, 2021

Exam Mon, January 31, 2022

Algorithms and Data Structures 2 // 2021W // 4 A. Ferscha

Lecture Topics

Introduction Monte Carlo Tree Search

Randomized Algorithms Graphs

• Structure
Search Trees • Flows
• Height Balanced Trees
• Weight Balanced Trees Social Graphs

Hashing Community Detection

Double Hashing Community Analysis

PRAM Algorithms

Algorithms and Data Structures 2 // 2021W // 5 A. Ferscha

Lecture Schedule

Date Topic
Mo. 11.10.2021 Introduction + Randomized Algorithms
Mo. 18.10.2021 Height Balanced Trees
Mo. 25.10.2021 Weight Balanced Trees
Mo. 08.11.2021 Hashing
Mo. 15.11.2021 Double Hashing
Mo. 22.11.2021 Monte Carlo Tree Search
Mo. 29.11.2021 Graphs: Structure
Mo. 06.12.2021 Graphs: Flows
Mo. 13.12.2021 Social Graphs
Mo. 10.01.2022 Community Detection
Mo. 17.01.2022 Community Analysis
Mo. 24.01.2022 PRAM Algorithms
Mo. 31.01.2022 Exam

Algorithms and Data Structures 2 // 2021W // 6 A. Ferscha

Primary Literature
Ottmann, Thomas; Widmayer, Peter: Algorithmen und Datenstrukturen.
Berlin/Heidelberg: Springer, 6. Auflage, 2017. ISBN: 9783662556504
• Behandelt sehr fachkundig und leicht verständlich den „klassischen“ Themenkern einer
Standardvorlesung „Algorithmen und Datenstrukturen" (vom Suchen und Sortieren über
Adressberechnungsmethoden und Listenstrukturen (Bäume aller Art) bis zu
Geometrischen Algorithmen und Graphenalgorithmen.
• Auch in elektronischer Form verfügbar:
https://lisss.jku.at/permalink/f/n2r1to/ULI_alma5155435650003340

Goodrich, Michael T.; Tamassia, Roberto; Goldwasser Michael H.:

Data Structures and Algorithms in Python, Hoboken, N.J.: Wiley, 2013.
ISBN: 9781118290279

Goodrich, Michael T.; Tamassia, Roberto; Goldwasser Michael H.:

Data Structures and Algorithms in Java, Hoboken, N.J. [u.a.] : Wiley, 6.
Auflage, 2014. ISBN 978-1-118-80836-8

Weiss, Mark Allen: Data Structures and Algorithm Analysis in Java,

Harlow: Pearson, 3. Auflage, 2012. ISBN: 978-0-273-75211-0

Algorithms and Data Structures 2 // 2021W // 7 A. Ferscha

Secondary Literature
R. Sedgewick: Algorithms in Java, Addison Wesley Professional, 2007.

T. Cormen, C. Leiserson, R. Rivest, C: Stein: Introduction to Algorithms,

MIT Press, 3. Auflage, 2009.

N. Wirth: Algorithmen und Datenstrukturen, Stuttgart, Teubner, 3. Auflage,

1991.

A.V. Aho, J.E. Hopcroft, J.D. Ullman: Data Structures and Algorithms,
Addison Wesley, 1987.

Donald E. Knuth:
The Art of Computer Programming. Volume 1 / Fundamental Algorithms.
The Art of Computer Programming. Volume 2 / Seminumerical Algorithms.
Reading, Mass.: Addison Wesley Longman, 1997.
▪ Frühe Standardwerke in der Informatik Grundausbildung.

Algorithms and Data Structures 2 // 2021W // 8 A. Ferscha

Complexity Science

Stefan Thurner, Rudolf Hanel, Peter Klimek: Introduction to the Theory of

Complex Systems, OXFORD University Press, 2018.

Stefan Thurner, Peter Klimek, Rudolf Hanel: A network-based explanation of

why most COVID-19 infection curves are linear. Proceedings of the National
Academy of Sciences Sep 2020, 117 (37) 22684-22689; DOI:
10.1073/pnas.2010398117

Algorithms and Data Structures 2 // 2021W // 9 A. Ferscha

Recommended Literature

Michael T. Goodrich, Roberto Tamassia, Michael H. Goldwasser:

Data Structures and Algorithms in Java, Wiley & Sons Ltd, current
edition. ISBN 978-1-118-80836-8

Michael T. Goodrich, Roberto Tamassia, Michael H. Goldwasser:

Data Structures and Algorithms in Python, Wiley & Sons Ltd, current
edition. ISBN 978-1-118-54958-2

Robert Sedgewick, Kevin Wayne: Algorithmen und Datenstrukturen,

Pearson Deutschland, 2014.
JKU LISSS

Algorithms and Data Structures 2 // 2021W // 10 A. Ferscha

Exam Registration in KUSSS

Consider the deadline

for exam registration!

Algorithms and Data Structures 2 // 2021W // 11 A. Ferscha

Moodle Access

FAQ for Students: https://help.jku.at/im/en/it-systeme/moodle-lernplattform/faqs-fuer-studierende

Access the Moodle course from KUSSS

◦ In KUSSS, select "Study Room“ (1) and "My Courses“ (2). You will see a Moodle icon (3) in the "Extras" column.
Click on the icon to log in to the course's JKU Moodle class. If there is no icon, there is no Moodle class in the JKU
Moodle system.

◦ If you see the moodle icon but cannot access the course, please contact [email protected].

Algorithms and Data Structures 2 // 2021W // 12 A. Ferscha

Moodle Course Preview

Algorithms and Data Structures 2 // 2021W // 13 A. Ferscha

Exercise Schedule (preliminary)

Date Topic
Oct 05/06 Introduction
Oct 19/20 Exercise 1: Randomized Algorithms
Nov 09/10 Exercise 2: Search Trees
Nov 23/24 Exercise 3: Hashing
Nov 30/Dec 01 Exercise 4: Graphs (Structure)
Dec 14/15 Exercise 5: Graphs (Shortest paths)
Jan 11/12 Exercise 6: Graphs (Network flows)
Jan 25/26 (reserved)

34031x (Java): https://moodle.jku.at/jku/course/view.php?id=15722

34032x (Python): https://moodle.jku.at/jku/course/view.php?id=15721

Algorithms and Data Structures 2 // 2021W // 14 A. Ferscha

Introduction

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria

Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at

RANDOM NUMBERS &
RANDOMIZED ALGORITHMS

Univ.-Prof. Dr. Alois Ferscha

Institute of Pervasive Computing
Johannes Kepler University Linz
[email protected]

Algorithms and Data Structures 2 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
jku.at
ALGORITHMICALLY SOLVABLE PROBLEMS

On computable numbers,
with an application to the Entscheidungsproblem
A. M. Turing
Proceedings of the London mathematical society 2.1 (1937): 230-265.

Algorithms and Data Structures 2 // 2021W // 2 A. Ferscha

RANDOM NUMBERS :: APPLICATIONS
In the common parlance, randomness is the apparent
lack of pattern or predictability in events.
The Oxford English Dictionary
Simulation: Real-world processes
exhibit inherent randomness:

Example: intersection traffic control

◦ random arrival times
◦ random destinations
◦ random lengths
◦ random accelerations
Goal: optimal traffic-light switching
◦ avoid long waiting times
◦ capacity of the turning lanes
◦ risk of collisions

Algorithms and Data Structures 2 // 2021W // 3 A. Ferscha

RANDOM NUMBERS :: APPLICATIONS

Testing algorithms with arbitrary input data

◦ sorting
◦ searching

Gambling
◦ card games
◦ dice games

Security, Cryptography

Simulation

Machine Learning

… Algorithms and Data Structures 2 // 2021W // 4 A. Ferscha

PSEUDO/TRUE RANDOM NUMBER GENERATORS
Two main approaches to generating random numbers using a computer:

Pseudo-Random Number Generators (PRNGs)

• PRNGs are algorithms that use mathematical formulae or simply precalculated tables to
produce sequences of numbers that appear random.
True Random Number Generators (TRNGs)
• TRNGs extract randomness from physical phenomena and introduce it into a computer.
(Environmental Noise, Photoelectric Effect, Atmospheric Turbulence, Radioactive Decay,
quantum phenomenon or a phenomenon with chaotic behaviour
key question: is the universe deterministic or not?)

Characteristic Pseudo-Random NGs True Random NGs

Efficiency Excellent Poor

Determinism Deterministic Nondeterministic
Periodicity Periodic Aperiodic

Algorithms and Data Structures 2 // 2021W // 5 A. Ferscha

PRNGs - Linear Congruential Method

Algorithms and Data Structures 2 // 2021W // 6 A. Ferscha

RANDOM NUMBERS :: MODULO/SCALING

Mapping from 0..m-1 to 0..n-1 with n<m

Direct methods
◦ modulo calculation
§ z = randint () % n
◦ scaling
§ z = randint () * n / m

Problems
◦ overflow of the value range with scaling
◦ randomness by modulo

Algorithms and Data Structures 2 // 2021W // 8 A. Ferscha

RANDOM NUMBERS :: RANDOMNESS BY MODULO

z z%8 last 7 bits m = 65536, a = 3421, c = 1, 𝑋! = 2345

odd <- even
even <- odd

Algorithms and Data Structures 2 // 2021W // 10 A. Ferscha

RANDOM NUMBERS ::
LINEAR CONGRUENTIAL METHOD
Generate random integers X in the range [0,m-1] as follows X i +1 = (aX i + c) mod m, i = 0,1,2,...
§ m modulus, as large as possible
§ a multiplier, 2 ≤ a < m
§ c increment, 0 ≤ c < m
§ a good choice of a, c, m, and X0 (seed) is very important
Mixed congruential method (c¹ 0)
Multiplicative congruential method (c=0)
Additive congruential method
X i +1 = ( X i + X i −k ) mod m , k ≥ 1, i = 0,1,2,...
LCM has full period if and only if:
§ the only positive integer that divides exactly both m and c is 1
§ if q is a prime number that divides m, then q divides a - 1
§ if 4 divides m, then 4 divides a - 1

Algorithms and Data Structures 2 // 2021W // 11 A. Ferscha

RANDOM NUMBERS :: SCALING

Given X0 = 27, a = 17, c = 43, and m = 100

Xi
Convert the integers to random numbers R in (0,1) using Ri = , i = 1,2,...
m
The Xi and Ri values are
X1 = (17*27+43) mod 100 = 502 mod 100 = 2, R1 = 0.02
X2 = (17*2+43) mod 100 = 77 mod 100 =77, R2 = 0.77
X3 = (17*77+43) mod 100 = 1352 mod 100 = 52 R3 = 0.52
…
Numbers take values only from the set I = {0,1/m,2/m,….., (m-1)/m}
since each Xi is an integer in the set {0,1,2,….,m-1}
⇒ each Ri is discrete on I, instead of continuous on [0,1]

Algorithms and Data Structures 2 // 2021W // 12 A. Ferscha

RANDOM NUMBERS :: PROPERTIES

Maximum Density
◦ Ri values to avoid large gaps in [0,1]
◦ Problem: Ri is discrete (not continuous)
◦ Solution: a very large integer for modulus m (e.g., 231-1)
Maximum Period
◦ achieve maximum density and avoid cycling
◦ needs good choice of a, c, m, and X0
Efficiency
◦ binary representation of numbers
◦ modulus m a power of 2 (2b or close)
◦ Xi+1 is obtained from aXi+c by dropping the leftmost and using the b rightmost binary digits

Algorithms and Data Structures 2 // 2021W // 13 A. Ferscha

RANDOM NUMBERS ::
LINEAR CONGRUENTIAL METHOD
Linear Congruential Method randomness depends on bit position
◦ some methods use the top 32 bits of the 48-bit random seed

lower bits “less random” than upper bits

Algorithms and Data Structures 2 // 2021W // 14 A. Ferscha

RANDOM NUMBERS ::
UNIFORM DISTRIBUTIONS (IF N IS A DIVISOR OF M)
Modulo (m=1000, n=800): Scaling:
§ 0 % 800 = 0 0*800/1000 = 0
§ 1 % 800 = 1 1*800/1000 = 0
§ … 2*800/1000 = 1
§ 799 % 800 = 799 3*800/1000 = 2
§ 800 % 800 = 0 4*800/1000 = 3
§ 801 % 800 = 1 5*800/1000 = 4
§ … 6*800/1000 = 4
§ 999 % 800 = 199 7*800/1000 = 5

Algorithms and Data Structures 2 // 2021W // 15 A. Ferscha

RANDOM NUMBERS ::
DIVISION METHOD PRINCIPLE
Consider only the largest multiple k*n ≤ m and drop larger random numbers

def smallRand(n):
k = m/n # how often does [0..n) fit in [0..m)
max = k*n # greatest multiple ≤ m
while(r >= max): # restriction to values ∈ [0..max-1]
r = randint()
return r/k # scaling from [0..max-1] to [0..n-1]

Algorithms and Data Structures 2 // 2021W // 16 A. Ferscha

RANDOM NUMBERS ::
DIVISION METHOD SOLUTION
def smallRand(n):
k = m/n
max = k*n
while (r >= max):
r = randint()
return r/k

Worst case: n = m / 2 + 1 only once ⇒ k = 1

Almost every second random number is deleted
⇒ ca. 2 iterations per call

Algorithms and Data Structures 2 // 2021W // 17 A. Ferscha

RANDOM NUMBERS ::
FLOATING POINT RANDOM NUMBERS
Goal
◦ random number z in the interval [0,1)
◦ can easily be transformed into any other interval
◦ e.g., mapping to [x, y) by z'= z*(y-x) + x
Method
def randf():
return float(randint()) / float(m)
Gaps when m is small (random numbers are at a distance of 1/m)

m must be ≥ 2k if the mantissa of the float representation consists of k bits

Algorithms and Data Structures 2 // 2021W // 18 A. Ferscha

Random Number Generator Properties

Algorithms and Data Structures 2 // 2021W // 19 A. Ferscha

RANDOM NUMBERS :: PROPERTIES

Uniformity
◦ uniformly/equally distributed
◦ divide the interval [0,1] into n classes of equal length
◦ given N random numbers (observations):
expected number of observations in each interval is N/n

Independence
◦ probability of observing a value in a particular interval is independent of the previous value
◦ no correlation between consecutive random numbers

Algorithms and Data Structures 2 // 2021W // 20 A. Ferscha

RANDOM NUMBERS :: GENERATOR PROPERTIES

Determinism
◦ algorithm implies uniqueness
◦ generating numbers using a known method removes the potential for true randomness
◦ if the method is known, the set of random numbers can be replicated
◦ however, random numbers should not be predictable
⇒ Generate random numbers in a way completely opaque for the user of the generator
Pseudo-random
◦ random number sequence is reproducible and the same for every program run
◦ e.g., 17th call always returns 5761
⇒ Important for recreating exactly the same scenarios in simulations
⇒ Not desirable in games, to avoid e.g., sequences of dice to always start with 5,3,4

Algorithms and Data Structures 2 // 2021W // 21 A. Ferscha

RANDOM NUMBERS ::
GENERATION OF PSEUDO-RANDOM NUMBERS
Goal
◦ to produce a sequence of numbers in [0,1] that simulates or imitates the ideal properties of
random numbers - uniform distribution and independence

Challenges in pseudo-random number generation

◦ numbers might not be uniformly distributed
◦ numbers might be discrete-valued instead of continuous-valued
◦ mean of the numbers might be too low / too high
◦ variance of the numbers might be too low / too high
◦ interdependence (i.e., correlation)

Verification of randomness
◦ tests for uniformity and independence for a particular generation scheme

Algorithms and Data Structures 2 // 2021W // 22 A. Ferscha

RANDOM NUMBERS :: PRNG PROPERTIES

Considerations

◦ fast computation - a simulation may require many millions of random numbers

◦ portable - programming language independent -> produce same results
◦ sufficiently long cycle/period - length of random number sequence before previous numbers
begin to repeat in an earlier order
◦ replicable - given the same seed, it should generate the same set of random numbers,
independent of the system
◦ (approximately) uniform and independent
◦ memory efficient - least storage
◦ cryptographically secure

Algorithms and Data Structures 2 // 2021W // 23 A. Ferscha

RANDOM NUMBERS :: PROPERTIES OF PRNGS

Most random-number generators are of the form

start with 𝒛𝟎 (seed)

for n = 1, 2, … generate 𝒛𝒏 = 𝒇(𝒛𝒏$𝟏 )
and 𝒖𝒏 = 𝒈(𝒛𝒏 )

f is the pseudo-random generator

g is the output function

𝑢! , 𝑢" , … is the sequence of uniform random numbers in the interval (0,1)

Algorithms and Data Structures 2 // 2021W // 24 A. Ferscha

RANDOM NUMBERS :: EXAMPLE

Midsquare method (von Neumann, 1949)

◦ start with a 4-digit 𝑧! seed
◦ square it to obtain 8-digits (if necessary, append zeros to the left)
◦ take the middle 4 digits to obtain the next 4-digit number 𝑧" ; square 𝑧" , etc.
◦ derive next random number by placing decimal point left of each 𝑧# (i.e., divide by 10000)
Examples
◦ 𝑧! = 1234 -> 0.1234, 0.5227, 0.3215, 0.3362, 0.3030, 0.1809, 0.2724, 0.4201, 0.6484,
0.0422, 0.1780, 0.1684, 0.8361, 0.8561, 0.2907, ...
◦ 𝑧! = 2345 -> 0.2345, 0.4990, 0.9001, 0.0180, 0.0324, 0.1049, 0.1004, 0.0080, 0.0064,
§ Note: the two successive zeros behind the decimal will never disappear
◦ 𝑧! = 2100 -> 0.2100, 0.4100, 0.8100, 0.6100, 0.2100, 0.4100,
§ Note: after four iterations the sequence starts to repeat itself

Algorithms and Data Structures 2 // 2021W // 25 A. Ferscha

Generating Random Numbers of Certain PDF Types

Algorithms and Data Structures 2 // 2021W // 27 A. Ferscha

DISCRETE UNIVARIATE PDFs

T. Song, Relationships among some univariate distributions, IIE Transactions (2005) 37, 651–656

Algorithms and Data Structures 2 // 2021W // 28 A. Ferscha

CONTINUOUS UNIVARIATE PDFs

T. Song, Relationships among some univariate distributions, IIE Transactions (2005) 37, 651–656

Algorithms and Data Structures 2 // 2021W // 29 A. Ferscha

CONTINUOUS UNIVARIATE PDFs

T. Song, Relationships among some univariate distributions, IIE Transactions (2005) 37, 651–656

Algorithms and Data Structures 2 // 2021W // 30 A. Ferscha

UNIVARIATE RELATIONSHIPS

T. Song, Relationships among some univariate distributions, IIE Transactions (2005) 37, 651–656
Algorithms and Data Structures 2 // 2021W // 31 A. Ferscha
RANDOM NUMBERS :: PROBABILITY DENSITY
RNGs randint(), randf() generate
Uniform Distribution
but may want RNs with other PDFs:

Normal Distribution Exponential Distribution

Algorithms and Data Structures 2 // 2021W // 32 A. Ferscha

RANDOM NUMBERS :: NORMAL DISTRIBUTION

Normal Probability Density Function

◦ addition of equally distributed random numbers
Examples
◦ biological features, e.g. body size
◦ deviations in manufacturing processes

Method
def gauss():
sum = 0
for i in range(n):
sum = sum + (randf()*2.0-1.0)
return sum

Algorithms and Data Structures 2 // 2021W // 33 A. Ferscha

RANDOM NUMBERS :: NORMAL DISTRIBUTION

Box-Muller Transform

The Box–Muller transform, by George Edward Pelham Box and Mervin

Edgar Muller, is a random number sampling method for generating pairs
of independent, standard, normally distributed (zero expectation,
unit variance) random numbers, given a source of uniformly
distributed random numbers.
(wikipedia)

Alternative: Marsaglia polar method

(avoid notoriously slow trigonometric functions)

Algorithms and Data Structures 2 // 2021W // 34 A. Ferscha

RANDOM NUMBERS :: EXPONENTIAL DISTRIBUTION

Random events on a timeline (Poisson Process)

Number of events in time pan follow poisson distribution

Time intervals follow an exponential distribution

Algorithms and Data Structures 2 // 2021W // 35 A. Ferscha

RANDOM NUMBERS :: EXPONENTIAL DISTRIBUTION

Calculation using the inversion method Inverse Transform Method

def expRand(mean):
r = randf()
while (r == 0.0):
r = randf() }
return -ln(r) * mean

Used in discrete event-oriented simulations

After event occurrence (e.g. vehicle arrival at the intersection), predict the next event occurrence
Example
◦ ca. 5 cars per minute
◦ nextEventTime = currentTime + expRand (12.0) # average interval 12 seconds

Algorithms and Data Structures 2 // 2021W // 36 A. Ferscha

RANDOM NUMBERS :: RANDOM PERMUTATIONS

Given: array a[0:n-1] (elements of any type)

Goal: arbitrary sequence of the elements
Applications
◦ cards shuffling
◦ generate disordered test data

Method
def shuffle (a):
for (i=n, n-1, ... 3, 2):
k = smallRand(i) # random element from a [0: i-1]
a[i-1] ↔ a[k] # comes at the end

Algorithms and Data Structures 2 // 2021W // 37 A. Ferscha

Periodicity

Algorithms and Data Structures 2 // 2021W // 38 A. Ferscha

RANDOM NUMBERS :: EXAMPLE
X i +1 = (aX i + c) mod m, i = 0,1,2,...
m = 8; a = 5; c = 3; X0 = 0
1 2 3 4 5 6 7 8 9 10 11 12
3 2 5 4 7 6 1 0 3 2 5 ... ⇒ period = 8 (≤ m)

m = 8; a = 3; c = 5; X0 = 0
1 2 3 4 5 6 7 8 9 10 11 12
5 4 1 0 5 4 1 0 5 4 1 ... ⇒ period = 4

m = 8; a = 3; c = 2; X0 = 0
1 2 3 4 5 6 7 8 9 10 11 12
2 0 2 0 2 0 2 0 2 0 2 ... ⇒ period = 2

Algorithms and Data Structures 2 // 2021W // 39 A. Ferscha

RANDOM NUMBERS ::
LINEAR CONGRUENTIAL METHOD
Maximal period (cycle length)

◦ m a power of 2 (m=2b) and c¹0

§ max. period P=m= 2b
§ if c is relative prime to m (greatest common divisor of c and m is 1) and a=1+4k (k integer)

◦ m a power of 2 (m=2b) and c=0

§ max. period P=m/4= 2b-2
§ if X0 is odd and a=3+8k or a=5+8k for k=0,1,…

◦ m is prime and c=0

§ max. period P= m-1
§ if the smallest integer k such that ak-1 is divisible by m is k=m-1

Algorithms and Data Structures 2 // 2021W // 40 A. Ferscha

RANDOM NUMBERS :: EXAMPLE

Find the period of the multiplicative congruential generator for a=13, m=26=64 and X0=1,2,3 and 4

◦ m=64, c=0; Maximal period P=m/4 = 16 is achieved by using odd seeds X0=1 and X0=3 (a=13
is of the form 5+8k with k=1)
◦ X0=1 -> the generated sequence {1,5,9,13,…,53,57,61} has large gaps
◦ density insufficient
◦ period too short
◦ not a viable generator

Algorithms and Data Structures 2 // 2021W // 41 A. Ferscha

RANDOM NUMBERS :: MAXIMAL PERIOD

• only if c and m are relative prime (i.e., gcd (c, m) = 1)

• a-1 is a multiple of p for each prime factor p contained in m
• a-1 is a multiple of 4 if m is a multiple of 4

Example
m = 18
18 = 2·3$ ⇒ prime factors are {2, 3}
⇒ c ∈ {1, 5, 7, 11, 13, 17}
⇒ a = 2·3·k + 1 ⇒ a ∈ {7, 13}

Algorithms and Data Structures 2 // 2021W // 42 A. Ferscha

RANDOM NUMBERS :: MAXIMAL PERIOD

If m is a prime number
⇒ m itself is the only prime factor
⇒ the smallest possible a would be m + 1
⇒ because of 2≤a<m the maximum period length is not reachable

If m is a power of two (2% )

⇒ 2 is the only prime factor
⇒ a is of the form 4z + 1
⇒ c any odd number

Algorithms and Data Structures 2 // 2021W // 43 A. Ferscha

RANDOM NUMBERS :: EXAMPLE

c=0; a=75=16807; m=231-1=2,147,483,647 (prime)

Period P=m-1 (>2 billion)
Let X0=123,457

X1 = 75(123457)mod(231-1) =2,074,941,799
R1 = X1/231 =0.9662
X2 = 75(2,074,941,799) mod(231-1) =559,872,160
R2 = X2/231 =0.2607
X3 = 75(559,872,160) mod(231-1) =1,645,535,613
R3 = X3/231 =0.7662
…

Algorithms and Data Structures 2 // 2021W // 44 A. Ferscha

RANDOM NUMBERS :: MAXIMAL PERIOD

Rules for practical use (Knuth)

◦ m the greatest possible power of 2, e.g. 2% for k-bit numbers

◦ simple calculation of x%m by "masking" the k least significant bits (overflow)
◦ a ≈ m / 10 (about one decimal place less)
◦ a ends on ... g21 with an even number g
◦ binary representation of a has an irregular bit pattern (e.g. not ... 001100110011 ...)
◦ c=1

Good values for m = 2"& = 65536: a = 3421, c = 1

Algorithms and Data Structures 2 // 2021W // 45 A. Ferscha

RANDOM NUMBERS :: EXTENDING THE PERIOD

Periods should not be fully exhausted in practice

In a long sequence of random numbers no repetitions of patterns (sub-sequences) should occur
Example
§ if a simulation requires 1 million random numbers per second
§ given a random number generator with m = 231
§ after 2147 seconds (36 minutes) the random sequence will repeat itself

Period length is determined by the size of the "memory" (number of possible states)
§ 16 bit ⇒ max. period length 𝟐𝟏𝟔

Methods with extended period

◦ shift register method (Tausworthe)
◦ table method (MacLaren and Marsaglia)

Algorithms and Data Structures 2 // 2021W // 46 A. Ferscha

RANDOM NUMBERS ::
SHIFT REGISTER METHOD (TAUSWORTHE)

Random number

Algorithms and Data Structures 2 // 2021W // 47 A. Ferscha

RANDOM NUMBERS ::
SHIFT REGISTER METHOD (TAUSWORTHE)

Random number
Abbreviation ⊕
a) bitwise XOR: z = t[p]^t[r]
b) addition modulo m: z = (t[p]+t[r])%m

Algorithms and Data Structures 2 // 2021W // 48 A. Ferscha

RANDOM NUMBERS ::
TABLE METHOD (MACLAREN AND MARSAGLIA)

RG1

Random number RG2

Algorithms and Data Structures 2 // 2021W // 49 A. Ferscha

INTERNET SERVICES FOR RANDOMNESS
(FROM ATMOSPHERIC NOISE)

Algorithms and Data Structures 2 // 2021W // 50 A. Ferscha

HISTORY OF PRNGs https://en.wikipedia.org/wiki/List_of_random_number_generators

Algorithms and Data Structures 2 // 2021W // 51 A. Ferscha

Tests for Random Numbers

Algorithms and Data Structures 2 // 2021W // 60 A. Ferscha

RANDOM NUMBERS :: PRINCIPLES

Desirable properties of random numbers

◦ Uniformity
◦ Independence

Types of tests for these properties

◦ Frequency test
§ Kolmogorov-Smirnov or Chi-square test evaluating whether the numbers generated follow a
uniform distribution
◦ Autocorrelation test
§ tests the correlation between numbers and compares the sample correlation to the expected
zero correlation

Algorithms and Data Structures 2 // 2021W // 61 A. Ferscha

RANDOM NUMBERS :: CATEGORIES

Hypotheses for uniformity

H0: Ri ~ U[0,1]
H1: Ri !~ U[0,1]
§ failure to reject the null hypothesis, H0, means that evidence of non-uniformity has not been
detected

Hypotheses for independence

H0: Ri !~ independently distributed
H1: Ri ~ independently distributed
§ failure to reject the null hypothesis, H0, means that evidence of dependence has not been
detected

Algorithms and Data Structures 2 // 2021W // 62 A. Ferscha

RANDOM NUMBERS :: PROCESS

Level of significance

◦ a level must be specified for each test

◦ the probability of rejecting the null hypothesis H0 when the null hypothesis is true

a = P(reject H0|H0 is true)

Usually a is set to 0.01 or 0.05

Algorithms and Data Structures 2 // 2021W // 63 A. Ferscha

RANDOM NUMBERS :: APPLICATION

When to test
◦ if a well-known simulation language or random number generator is used, it is probably
unnecessary to test
◦ if the generator is not explicitly known or documented, e.g., spreadsheet programs,
symbolic/numerical calculators, tests should be applied to many sample numbers

Types of tests
◦ theoretical
§ evaluate the choices of m, a, and c without actually generating any numbers
◦ empirical
§ applied to actual sequences of numbers produced

Algorithms and Data Structures 2 // 2021W // 64 A. Ferscha

RANDOM NUMBERS :: TESTS OF UNIFORMITY

Frequency methods (goodness-of-fit)

◦ Kolmogorov-Smirnov test
◦ Chi-square test
◦ measure the degree of agreement between the distribution of a sample of generated random
numbers and the theoretical uniform distribution
◦ based on null hypothesis of no significant difference between the sample distribution and the
theoretical distribution

Given 𝑋" , …, 𝑋' observations

𝐻! hypothesis
◦ 𝑋# i.i.d. random variables with distribution function F

Algorithms and Data Structures 2 // 2021W // 65 A. Ferscha

RANDOM NUMBERS ::
UNIFORM DISTRIBUTION TEST
Split the random numbers into r equally large classes
Take n random samples
Target distribution: n/r per class

Linear congruential method

◦ for n = m the distribution is uniform
◦ should be approximately uniform also for n < m

Algorithms and Data Structures 2 // 2021W // 66 A. Ferscha

RANDOM NUMBERS :: SERIAL TEST

Consecutive dice rolls may produce the same number of eyes due to independence of rolls
Probability is 1/6
If the entire range of values is used the serial test may fail
When mapping to [0, n-1] with n < m, this should ideally occur in one of n cases

Algorithms and Data Structures 2 // 2021W // 67 A. Ferscha

RANDOM NUMBERS :: GAP TEST

Generalization of the serial test

◦ how long before the same number reappears

Definition of gap length

◦ 22 gap of length 1
◦ 232 gap of length 2
◦ 2312 gap of length 3

Linear congruential method with full value range [0, m-1]

◦ all gaps have length of m

Algorithms and Data Structures 2 // 2021W // 68 A. Ferscha

RANDOM NUMBERS :: GAP TEST

Example
◦ random numbers in [0, r-1]
◦ probability of gaps of length k
r = 10

Algorithms and Data Structures 2 // 2021W // 69 A. Ferscha

RANDOM NUMBERS :: RUNS-UP TEST

Runs-up is a sequence of ascending random numbers

◦ split sequence in blocks, where each block is a subsequence of increasing
numbers followed by a number that is smaller than its predecessor
◦ example: 1,3,8,6,2,0,7,9,5 can be divided in the blocks (1,3,8,6), (2,0), (7,9,5)

Procedure
◦ generate n runs-ups
◦ count the number of runs-ups of length 0, 1, …, k

Probability for a runs-up of length i:

◦ decreases exponentially with the run length

Algorithms and Data Structures 2 // 2021W // 70 A. Ferscha

RANDOM NUMBERS :: CHI-SQUARE TEST

Hypothesis
◦ actual distribution comes from a given n is the # of classes
target distribution (not necessarily uniform)
Ei is the expected
# in the ith class
Procedure (Oi - Ei ) 2
n

◦ uses the sample statistic c =å

2
0
i =1 Ei Oi is the observed
◦ split random numbers into classes 1…n # in the ith class
◦ determine frequencies 𝑂# for each class
◦ the following must apply: 𝑂# ≥10 for all 1≤i≤n
◦ calculate the sum of the quadratic deviations
◦ reject H0 if c02 > ca,N-12

1 n 1 n
deviations too large too regular
Algorithms and Data Structures 2 // 2021W // 71 A. Ferscha
RANDOM NUMBERS :: EXAMPLE: CHI-SQUARE TEST

Let a = 0.05 and n=10 intervals of equal length, namely [0,0.1),[0.1,0.2), …., [0.9,1.0)

Data Computations for Chi-square test

c02 = 3.4
Critical value from table c0.05,92 = 16.9
Hence, null hypothesis is not rejected
Algorithms and Data Structures 2 // 2021W // 72 A. Ferscha
RANDOM NUMBERS :: VISUAL TESTS

Bar test
◦ visualization of the frequency distribution
Sky test
◦ x/y pairs form a starry sky
Stripe test
◦ randomly coloured lines

Algorithms and Data Structures 2 // 2021W // 73 A. Ferscha

Randomized Algorithms

Algorithms and Data Structures 2 // 2021W // 77 A. Ferscha

RANDOMIZED ALGORITHMS

• In addition to the input, a randomized algorithm takes random numbers to make random
choices during execution
• Behavior (output) of the algorithm depends on a random experiment
◦ Output varies in each execution, also using same input
◦ Runtime and correctness can only be guaranteed with a certain probability
• Hence, Randomized Algorithms are nondeterministic

Deterministic vs. Randomized Algorithms

Algorithms and Data Structures 2 // 2021W // 78 A. Ferscha

RANDOMIZED ALGORITHMS

Advantages
◦ Efficiency
◦ Simplicity

Attention
◦ Randomized inputs (by itself) do not make randomized algorithms

Not a randomized algorithm

Algorithms and Data Structures 2 // 2021W // 79 A. Ferscha

RANDOMIZED ALGORITHMS :: MONTE CARLO ALGORITHM

Monte Carlo (MC) algorithms

◦ run for a fixed number of steps and produce an answer that is not necessarily correct
◦ mostly used in search- and decision problems

Form the basis for Monte Carlo Simulation

◦ random sampling used for example to determine pi or solve integrals

Algorithms and Data Structures 2 // 2021W // 80 A. Ferscha

RANDOMIZED ALGORITHMS :: MONTE CARLO ALGORITHM

find_char_monte_carlo(char[] A, k)
begin
i = 0
do
randomly select one element from A
i++
while (i < k and ‘a’ is not found)
end

The algorithm succeeds if ‘a’ is found, else it fails

After k iterations the probability of finding ‘a’ is 1 – (1/2)k

Algorithms and Data Structures 2 // 2021W // 81 A. Ferscha

RANDOMIZED ALGORITHMS ::
MONTE CARLO SIMULATION
Monte Carlo Simulations
◦ stochastic methods which use high numbers of random experiments (random sampling) as a
basis to solve problems based on probability

Algorithms and Data Structures 2 // 2021W // 82 A. Ferscha

RANDOMIZED ALGORITHMS ::
MONTE CARLO SIMULATION

Probabilistic determination of pi

Random points in range [-1,1] are sampled

𝐴𝑟𝑒𝑎 𝑜𝑓 𝑐𝑖𝑟𝑐𝑙𝑒 𝑟 !𝑝𝑖 𝑝𝑖 𝐻𝑖𝑡𝑠 𝑖𝑛 𝑐𝑖𝑟𝑐𝑙𝑒

= !
= = = 𝑃(𝑖𝑛 𝐶𝑖𝑟𝑐𝑙𝑒)
𝐴𝑟𝑒𝑎 𝑜𝑓 𝑠𝑞𝑢𝑎𝑟𝑒 2𝑟 4 𝑡𝑜𝑡𝑎𝑙 𝑝𝑜𝑖𝑛𝑡𝑠

Algorithms and Data Structures 2 // 2021W // 83 A. Ferscha

RANDOMIZED ALGORITHMS ::
MONTE CARLO SIMULATION

Solving Integrals m
A1 = n × m
• Random samples (x,y) from rectangle ((0,0), (n,m))
• Count the number of random pairs below (nin) and above n

(nout) the function f(x)

A2 = ò f ( x)dx
0 0

• Decide if pair is in f(x) : 0 n

◦ nin = nin+1 if: yrand <= f(xrand)
◦ nout = nout+1 if: yrand > f(xrand)

< 84 >
Algorithms and Data Structures 2 // 2021W // 84 A. Ferscha
RANDOMIZED ALGORITHMS :: LAS VEGAS ALGORITHM

Las Vegas algorithms

• produce an answer that is always correct but the runtime is not constant or
known beforehand

find_a_las_vegas(char[] A, k)
begin
do
randomly select one element from A
while (‘a’ is not found)
end

The algorithm succeeds with probability 1

The number of iterations varies and can be arbitrarily large

Algorithms and Data Structures 2 // 2021W // 85 A. Ferscha

RANDOMIZED ALGORITHMS :: MARKOV-CHAINS

Markov Chains
◦ discrete-time stochastic processes moving from one state to the next
◦ sequence of states Xi, for which the probability P(Xs) to reach a certain state Xs is dependent
only on the previous state of the system
Finite State Markov Chain M = (S, P)
◦ S is a finite set of states and P an S*S stochastic matrix holding the transition probabilities (i, j
∈ S such that 0 ⩽ Pij ⩽ 1 and ∑ Pij = 1)
◦ for all i, j ∈ S, Pij is the probability that M moves from state i to state j
◦ the probability only depends on i and not on previous steps
Let Xt be a random variable which determines the state of M at time t
P(Xt+1 = j | Xt = i) = Pij
◦ the initial state X0 is usually set stochastically based on some initial distribution

Algorithms and Data Structures 2 // 2021W // 86 A. Ferscha

RANDOMIZED ALGORITHMS :: MARKOV-CHAINS EXAMPLE

0 0,2 0,8 0 𝑖𝑓 𝑖 = 1
𝑀 = 0,5 0 0,5 𝑃 𝑋𝑡 + 1 = 𝑖 𝑋𝑡 = 1) = 𝑓 𝑥 = 60,2 𝑖𝑓 𝑖 = 2
0,1 0,9 0 0,8 𝑖𝑓 𝑖 = 3
stochastic matrix where the rows
represent the current state and the
columns the next state
0,5 𝑖𝑓 𝑖 = 1
𝑃 𝑋𝑡 + 1 = 𝑖 𝑋𝑡 = 2) = 𝑓 𝑥 = 6 0 𝑖𝑓 𝑖 = 2
0,5 𝑖𝑓 𝑖 = 3
state graph of the Markov chain

0,1 𝑖𝑓 𝑖 = 1
𝑃 𝑋𝑡 + 1 = 𝑖 𝑋𝑡 = 3) = 𝑓 𝑥 = 60,9 𝑖𝑓 𝑖 = 2
0 𝑖𝑓 𝑖 = 3
transition probabilities of the Markov
chain

Algorithms and Data Structures 2 // 2021W // 87 A. Ferscha

RANDOMIZED ALGORITHMS :: RANDOM WALKS

Given a G = (V, E) finite connected undirected graph with 𝑉 = 𝑛 vertices and 𝐸 = 𝑚 edges
◦ neighbors of vertex u, Γ(u) = {v | (u, v) ∈ E}
Consider a random walker who is located at any vertex of the graph at any time moving to any
vertex connected via an edge at equal probability
◦ muv is the expected number of steps needed to get from vertex u to vertex v
◦ Cuv = muv + mvu is the expected number of steps from vertex u to u visiting vertex v at least
once
Each random walk on a graph G is a finite Markov chain where Pij = 0 if there is no edge from i
"
to j or Pij = otherwise (with d(i) being the initial degree of vertex i)
?(#)

Algorithms and Data Structures 2 // 2021W // 88 A. Ferscha

RANDOMIZED ALGORITHMS ::
MARKOV CHAIN MONTE CARLO

Markov Chain Monte Carlo (MCMC) algorithms

◦ Monte Carlo Simulation for solving functions by random sampling
Limitations of Monte Carlo
◦ complex functions (i.e. not uniform or gaussian)
◦ higher dimensional space
◦ Monte Carlo sampling is problematic in these cases
Solution
◦ find a Markov Chain which has the desired distribution as a stationary distribution
◦ the state of such Markov Chain after many steps is used as sample for the desired
distribution

Algorithms and Data Structures 2 // 2021W // 91 A. Ferscha

RANDOMIZED ALGORITHMS :: METROPOLIS ALGORITHM

• Select an initial configuration x for which ρ(x) > 0

Metropolis Algorithm • Choose a maximum displacement value ∆xmax
• Calculate the initial ρ=ρ(x)
◦ MCMC method for random sampling from • Store the current state as x0=x
a probability distribution • Generate a random number vector u= (u1, u2, ..., uM)
◦ Direct sampling from this distribution would where each u is a uniform random number between -1
and 1
be too complicated • Generate a new trial state xʹ=x+u∆xmax
• Calculate the value of the function in the trial
Idea state ρʹ=ρ(xʹ)
◦ In a minimum of p(x) the trial state p’ is • Choose whether to move to the new state as follows:
• If ρʹ≥ρ accept the state
guaranteed to be larger than p • Ifρʹ< ρ: accept the state only if u <ρʹρ where u
◦ Next to a minimum of p(x) the trial state p’ is a random number∈[0,1]
could go to the minimum and the • If the state is accepted, go to the new state:
x=xʹ
probability of accepting that move is • If the state is rejected, return to the previous
smaller state: x=x0
•
◦ The system is more likely to go to, or •
Calculate the new ρ=ρ(x)
Do statistics of the current value of x or properties
remain in, states with larger p values dependent on it
• Return to step 1
Algorithms and Data Structures 2 // 2021W // 92 A. Ferscha
RANDOMIZED ALGORITHMS :: SIMULATED ANNEALING

• Select the initial configuration x

Annealing • Set the number of Monte Carlo steps nMCS = 0
◦ metal processing method, improving quality • Set the initial temperature to some high value
of steel by heating up and slowly cooling T0
• Choose a transition ∆x at random
down • Calculate the function value before the
◦ repeated in cycles to remove artifacts transition fb = f(x)
• Do the trial transition as x = x + ∆x
Idea • Calculate the function value after the
◦ introduce a variable “temperature” T (which •
transition fa = f(x)
Calculate ∆f = fa − fb, then :
may not have anything to do with ◦ If ∆f ≤ 0 accept the state
thermodynamics) ◦ If ∆f > 0 :
◦ use a generalized Metropolis scheme to §
§
Generate a random number u between 0 and 1
Accept the state only if u < e−∆f/T
simulate the system at temperature T, while T • If the state is rejected, return to the
previous state: x = x − ∆x
is gradually decreased from some high initial • Reduce the temperature by some small value:
temperature (”cooling”) T = T − εT
• Set nMCS = nMCS + 1
• If T > 0 return to step 1
Algorithms and Data Structures 2 // 2021W // 93 A. Ferscha
SIMULATED ANNEALING FOR
CONTINUOUS (GLOBAL) OPTIMIZATION
• given a continuous cost function F
• find the minimum/maximum value of F
across a defined domain of variables (=
solution space)

• gradient method problem:

get stuck in local optimum

Algorithms and Data Structures 2 // 2021W // 94 A. Ferscha

THE ANNEALING PROCESS

• avoid gradient method problem: controlled counter-stepping

• Use probabilistic control for counter-steps, i.e. adopt thermodynamics of the annealing
process modeled using the Boltzmann probability distribution:
Prob(E) ~ exp(-E/kT)
E ... possible energy state
Algorithms and Data Structures 2 // 2021W // 95
T ... system temperature A. Ferscha
k ... Boltzmann’s constant
THE BOLTZMANN DISTRIBUTION

Algorithms and Data Structures 2 // 2021W // 96 A. Ferscha

METROPOLIS ALGORITHM [METROPOLIS 53]

originally: computer simulation of thermodynamic processes

1. An initial configuration, Ci is chosen at random or by some other, usually heuristic

method. This configuration has an associated cost, Ei = F(Ci).
2. An initial temperature, T, is determined. Energy = objective Function
3. A random change in the configuration is generated.
This change produces a possible new configuration, Ci+1.
4. If Ei+1 £ Ei , then change is always accepted. Probabilistic Step
5. If Ei+1 > Ei , then change is accepted with probability

exp[-(Ei+1 – Ei)k T] cooling schedule

6. Update temperature T if required.

7. Repeat from 2. until T < Tstop stopping criterion

Algorithms and Data Structures 2 // 2021W // 97 A. Ferscha

SIMULATED ANNEALING

moderate
probability for
up-hill stepping at
low temperature

reasonable
probability for
up-hill stepping at
high temperature

Algorithms and Data Structures 2 // 2021W // 98 A. Ferscha

SIMULATED ANNEALING ALGORITHM

Anneal(I,S)
PRE: I = Instance of problem
S = initial feasible solution to I of size n > 0
h(S) = objective function, cost of solution
POST: S = feasible solution for I

Determine initial temperature T

repeat
tries = moves = goodmoves := 0
repeat
NewS := random neighbour of S
DS := h(NewS) - h(S)
tries := tries + 1
rnd := random number between 0 and 1
if DS < 0 or rnd < e-DS/T then
S := NewS
moves := moves + 1
if DS < 0 then goodmoves := goodmoves + 1
until goodmoves>n or moves>2*n or tries>20*n
A. Ferscha
T := 0.85*T Algorithms and Data Structures 2 // 2021W // 99
Until moves < 0.05 tries or T < e
SETTING AN INITIAL TEMPERATURE

Temperature(T)
POST: T provides about 95% chance
of changing solutions

select an initial solution S0

Repeat n times
generate a neighboring solution S
compute DS := h(S0) - h(S)
if DS > 0 then sum := sum + DS
set ave = sum/n
find a T such that 0.95 < e-ave/T

Algorithms and Data Structures 2 // 2021W // 100 A. Ferscha

TREES (HEIGHT BALANCED)

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at
DEFINITION & TERMINOLOGY

A tree is
• an acyclic, simple, coherent graph
• i.e. does not contain loops and cycles:
between each pair of nodes there is at maximum one edge

Generalization of lists:
• Element (node) may have multiple successors.
• Exactly 1 node has no predecessor: root
• Nodes without successors: leaves

Frequently used data structure: decision trees, syntax trees, derivation trees, search trees, ...
Frequently used representation form: quantity, bracket, recursive indentation, graph, ...

Here: Use of trees to store keys and realization of dictionary operations (search, insert, remove)
in e.g. binary trees

Algorithms and Data Structures 2 // 2021W // 2 A. Ferscha

DEFINITION & TERMINOLOGY

Tree B is ordered, if successors of each node are ordered (1., 2., 3. etc.; left, right).
In an ordered tree, the subtrees Bi of each node form an ordered set (e.g.: arithmetic expression)
Order of B: maximum number of successors of a node
Path of length k: Follow p0, ..., pk of nodes, such that p is successor of pi-1

• Height of a tree:
maximum distance of a leaf to the root.
• Depth of a node:
distance to the root (the number of edges on a path from this node to the root)
The nodes at level i are all nodes with depth i.

A tree of order n is called complete if all leaves have the same depth
and the maximum number of nodes is present at each level.

Algorithms and Data Structures 2 // 2021W // 3 A. Ferscha

DEFINITION & TERMINOLOGY

A root
B is parent of D and E
C is sibling of B
D and E are children of B
D, E, F, G, I are external nodes or leaves
A, B, C, H are internal nodes

The depth of E is 2.
The height of the tree is 3.
The order of B is 2, the order of C is 3.

We always have: number of edges = number of nodes - 1

Algorithms and Data Structures 2 // 2021W // 4 A. Ferscha

BINARY TREES

If T is a binary tree (order of all nodes <= 2) with n nodes and a height h, then T has the following properties

F M

D W C E

E R

• Number of external nodes is at least h+1 and at maximum 2h

• Number of internal nodes is at least h and at maximum 2h –1
• Number of nodes is at least 2h+1 and at maximum 2h+1 –1
• The height of T is at least log(n+1) –1 and at maximum (n-1)/2

Algorithms and Data Structures 2 // 2021W // 5 A. Ferscha

TRAVERSING BINARY TREES

• Preorder:
28, 16, 12, 8, 15, 19, 34, 31, 29, 49
• Postorder:
8, 15, 12, 19, 16, 29, 31, 49, 34, 28
• Inorder:
8, 12, 15, 16, 19, 28, 29, 31, 34, 49

Hints: If you traverse the tree starting from the root, you have
• Preorder by visiting all nodes on the left side of which you pass by.
• Postorder by visiting all nodes on the right side of which you pass by.
• Inorder by visiting all nodes on the bottom side of which you pass by.

Algorithms and Data Structures 2 // 2021W // 6 A. Ferscha

BINARY SEARCH TREES

A binary search tree is a binary tree T where:

• Each internal node stores a key-value pair of a dictionary

• Keys which are stored in nodes of the left subtree of a

node v, are less than or equal to the key stored in v

• Keys which are stored in nodes of the right subtree

of a node v, are greater than the key stored in v

• External nodes serve only as placeholders

and do not store elements

Algorithms and Data Structures 2 // 2021W // 7 A. Ferscha

SEARCHING IN BINARY SEARCH TREES

Assumption: For each node n with children cl, cr and key k we have (search tree condition):
• All keys stored in subtree of cl are smaller than k
• All keys stored in subtree of cr are larger than k
• All keys stored in subtree of cl are smaller than all keys stored in subtree of cr

Search for key x:

1. Compare key k of inspected node n with x.
2. If n == null, x is not in the binary tree.
3. If k < x, set n = cr(n) and jump to 1.
4. If k > x, set n = cl(n) and jump to 1.
5. If k == x, found.

Maximum number of inspected nodes: depth of tree

Search within the nodes, e.g. by linear or binary search. As l ≤ m, the complexity is constant.

Algorithms and Data Structures 2 // 2021W // 8 A. Ferscha

SEARCHING IN BINARY SEARCH TREES

Binary search tree is a decision tree:

in each internal node v the key to be searched for is compared with the key stored in v

Pseudocode
Algorithm TreeSearch(k, v)
Input: key k to be searched, node v of a binary search tree
Output: node w in subtree of v, for successful search internal node with key k, for unsuccessful
search
external node after all internal nodes smaller than k and before all internal nodes greater than k
if v is an external node then
return v
if k = key(v) then
return v
else if k < key(v) then
return TreeSearch(k, T.leftChild(v))
else
return Treesearch(k, T.rightChild(v))

Algorithms and Data Structures 2 // 2021W // 9 A. Ferscha

EXAMPLES FOR SEARCH IN BINARY SEARCH TREES

Algorithms and Data Structures 2 // 2021W // 10 A. Ferscha

COMPLEXITY OF BINARY SEARCH TREES

For searching, inserting, removal, nodes along a root-leaf-path (plus possibly siblings of such nodes)
needs to be traversed.
• Complexity per node: O(1)
• Therefore the runtime is O(h), where h is the height of the tree
In worst case the height of the binary search tree is h = N
(tree „degenerates“ to sorted sequence)

• Worst case complexity O(N)

• Therefore, the tree must be kept balanced to allow operations in O(log N)

Algorithms and Data Structures 2 // 2021W // 11 A. Ferscha

TREES

Problem: trees can degenerate (e.g. become lists)

Removal of a node

• Very easy: Removal of a leaf

• Easy: Removal of a node p with 1 successor:
replace node with successor

• More difficult: Removal of a node p with 2 successors:

Search for the leftmost node q in right subtree (symmetric successor).

Replace p with q, remove q from its original position.

Algorithms and Data Structures 2 // 2021W // 12 A. Ferscha

BINARY SEARCH TREE :: ACCESS COST

Cost measure: Number of nodes visited or number of search steps or key comparisons required.

Average access cost z of a tree B ...

• is obtained by calculating its total path length PL
as the sum of the lengths of all paths from the root to each node Ki.
$

𝑃𝐿 𝐵 = ( 𝐿𝑒𝑣𝑒𝑙 (𝐾! )
!"#
!"
• The mean path length is calculated by 𝑙 =
#

Maximum access cost We have the longest search path and thus the maximum access cost, when the binary search
tree degenerates into a linear list.
ℎ𝑒𝑖𝑔ℎ𝑡 ℎ = 𝑙!"# + 1 = 𝑁

Algorithms and Data Structures 2 // 2021W // 13 A. Ferscha

BINARY SEARCH TREE :: ACCESS COST

Maximum average access cost:

$)#
1 (𝑁 + 1) (𝑁 + 1)
𝑧%&' = ∗ ( 𝑖+1 ∗1=𝑁∗ = = 𝑂(𝑁)
𝑁 2𝑁 2
!"(

Minimum average access cost:

• They can be expected in an almost complete or balanced tree structure.
• Total number of nodes: 2h-1-1 < N ≤ 2h-1
• height h = [log2N] + 1
• Minimum average access cost: zmin ≈ log2N - 1

Algorithms and Data Structures 2 // 2021W // 14 A. Ferscha

BINARY SEARCH TREE :: ACCESS COST

Average access cost

• Extreme cases of average access costs have only little
significance.
• The difference between the average and the minimum
access cost is a measure of the urgency of balancing.
ZN
Determination of the average access cost: i
Bi Br

i-1 N-i
Zi-1 nodes nodes ZN-i

Algorithms and Data Structures 2 // 2021W // 15 A. Ferscha

BINARY SEARCH TREE :: ACCESS COST

N different keys with values 1, 2, ..., N are given in random order.

$
The probability that the first key has the value i is
%
(Assumption: same access probability to all nodes)

For the tree with i as root we get:

1
𝑍% 𝑖 = ∗ 𝑍&'$ + 1 ∗ 𝑖 − 1 + 1 + 𝑍%'& + 1 ∗ (𝑁 − 𝑖)
𝑁
The recursive equation can be represented in non-recursive, closed form by means of the harmonic function:
%
1
𝐻% = 1
𝑖
&($

Algorithms and Data Structures 2 // 2021W // 16 A. Ferscha

BINARY SEARCH TREE :: ACCESS COST

We get:
𝑁+1
𝑧$ = 2 ∗ ∗ 𝐻$ − 3 = 2𝑙𝑛 𝑁 − 𝑐
𝑁

Relative additional cost:

*! +,- $ ). + /2 $ ).
= ~ = 2 ln 2 = 1,386 …
*"#$ /01% $ )# /01% $

The balanced binary search tree causes the least cost for all basic operations.

However, perfect balancing at any time is very expensive.

Algorithms and Data Structures 2 // 2021W // 17 A. Ferscha

BALANCED TREES

Efficiency of dictionary operations (insert, search, remove) on trees depends directly on the tree height.

Tree with N nodes:

• Minimum height: ëlog2 Nû, maximum height: N-1
• Access on average O(log2 N), but in worst case linear.

Aim:
• Fast access with zmax ~ O (log2 N)
• Insert and remove operations in logarithmic complexity.

Heuristics:
For each node in the tree, the number of nodes in each of its two subtrees should be kept as constant as possible.

Algorithms and Data Structures 2 // 2021W // 18 A. Ferscha

BALANCED TREES

Two different approaches

• Height-balanced trees:

The maximum allowed height difference of the two subtrees is limited

• Weight-balanced trees:

The ratio of the node weights (number of nodes) of both subtrees meets certain conditions.

Algorithms and Data Structures 2 // 2021W // 19 A. Ferscha

HEIGHT-BALANCED SEARCH TREES

• B-Trees
always balanced due to balance sustaining search, insert, remove
number of children per inner node between t (=m/2) and 2t (=m)

access in O(log(n))

• AVL Trees
The heights of each internal node’s children only differ by a maximum of 1

access in O(log(n))

Example from https://de.wikipedia.org/wiki/B-Baum

Algorithms and Data Structures 2 // 2021W // 20 A. Ferscha

B-TREES

Tree structure (m-Tree) that is always balanced

• self-balancing mechanism integrated in access operations (search, insert, remove)
• maintains sorted data (keys)
• search(), insert(), remove() in logarithmic time
• supports ”locality”-principle in memory organization

1. Every node has at most m children (m = 2t).

2. Every non-leaf node (except root) has at
least ⌈t = m/2⌉ child nodes.
3. The root has at least 2 children if
it is not a leaf node.
4. A non-leaf node with k children
contains k−1 keys.
5. All leaves appear in the same level (height h)
Graphics: Flying sheep, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=19226668

Algorithms and Data Structures 2 // 2021W // 21 A. Ferscha

B-TREES

m = 2t ... maximum number of children per node

n ... number of keys stored,

then the height h is:

*+$
ℎ ≤ log ) +1 (therefore access in O(log(n)))
,

t ... minimum number of children per node

n ... th-1

let t = 1024, h = 4 then

10244– 1 = (210)4-1 = 240- 1 = 1.099.511.627.775 keys (i.e. access time 10 times lower)

Algorithms and Data Structures 2 // 2021W // 22 A. Ferscha

B-TREES

Search (key = 9) Insert

Search for insert position
(inner node)
If inner node full
(i.e more than 2t-1 keys)
then preventively split
before stepping down

Graphics: Haui, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=245051, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=245228

Algorithms and Data Structures 2 // 2021W // 23 A. Ferscha

B-TREES

Remove Move
Removing (inner node) could transfer a sibling
destroy balance to expand (minimum) inner node
before stepping down
check whether there are
enough keys in the subtree

If considered inner node in

subtree has just minimum
number of keys (t-1 keys), then
Move / Merge
to prevent from imbalance after
Remove

Graphics: Haui, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=248207

Algorithms and Data Structures 2 // 2021W // 24 A. Ferscha

B-TREES

Merge Remove
before stepping down before deleting key from inner node
merge two a siblings separate/adjust the key ranges
if both (left and right child) of both left and right child
are at minimum number of inner nodes (here: inorder transfer,
i.e. rightmost of left child)

Graphics: Haui, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=248208, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=248209

Algorithms and Data Structures 2 // 2021W // 25 A. Ferscha

B-TREES
Example (t=2) i.e. min 1, max 3 keys per inner node
a–c) insert 5, 13, 27 k–l) remove 7 induces merge
d–e) insert 9 – induces splitting of root m) remove 5 (leaf)
f) insert 7 leaf node n–q) remove 3 induces merge of two children of root
g–h) insert 3 needs splitting empty root replaced by only child
i–j) remove 9, needs induces move of sibling

Example from https://de.wikipedia.org/wiki/B-Baum; Graphics: Haui, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=250188

Algorithms and Data Structures 2 // 2021W // 26 A. Ferscha

AVL TREES (ADELSON-VELSKII, LANDIS 1962)

AVL trees are balanced binary search trees

An AVL tree is a binary search tree, in which the heights of each internal node’s children only differ by a maximum of 1.
Example (numbers next to the nodes indicate their height):
44 4

2 17 78 3

32 1 2 50 88 1

48 1 62 1

Algorithms and Data Structures 2 // 2021W // 27 A. Ferscha

HEIGHT OF AN AVL TREE

Assertion: The height of an AVL tree T that stores n keys is O(log n)

Sketch of a proof:
Find n(h), the smallest possible number of internal nodes in an AVL tree of height h.

• Trivial: n(1) = 1, n(2) = 2

Algorithms and Data Structures 2 // 2021W // 28 A. Ferscha

HEIGHT OF AN AVL TREE

Assertion: The height of an AVL tree T that stores n keys is O(log n)

Sketch of a proof:
Find n(h), the smallest possible number of internal nodes in an AVL tree of height h.
• Trivial: n(1) = 1, n(2) = 2
• n ³ 3: The AVL tree of height h with minimum n(h) consists of a root node, an AVL subtree of height h-1 and an AVL
subtree of height h-2.
i.e. n(h) = 1 + n(h-1) + n(h-2) Fibonacci progression
as n(h-1) > n(h-2) it follows: (exponential!)

• findElement: O(logn) n(h) > 2 n(h-2)

n(h) > 4 n(h-4)
choose i=éh/2ù -1
• findAllElements O(log n+s) n(h) > 8 n(h-6)
(s=number of findings) ...
n(h) > 2i n(h-2i) then n(h) ³ 2 h/2-1

Using the logarithm we get: h < 2 log n(h) + 2

Therefore we have: The height of an AVL tree is O(log n) log n(h) > h/2 –1

Algorithms and Data Structures 2 // 2021W // 29 A. Ferscha

INSERT IN AVL TREES

By inserting a node into an AVL tree, the height of some nodes in this tree changes

Insert operation may cause the AVL tree to be unbalanced

• Go up from the new node in the tree until the first node x is found, whose "grandfather" z is an unbalanced node.
• Since z was unbalanced by inserting a node that lies in a sub-tree with root y (where y is a child of z), we have
height(y) = height(sibling(y)) + 2

Re-balancing of the sub-tree with root z requires restructuring:

• x, y and z are renamed to a, b, c (according to in-order traversing)
• z is replaced by b, whose children are now a and c, whose children are again four subtrees,
which were formerly the children of x, y and z

Algorithms and Data Structures 2 // 2021W // 30 A. Ferscha

INSERT IN AVL TREES
44 4
Insert 54

2 17 78 3

1 32 2 50 1 88

48 1 62 1

Algorithms and Data Structures 2 // 2021W // 31 A. Ferscha

INSERT IN AVL TREES
44 5
After inserting 54:

2 17 78 4

unbalanced!
1 32 3 50 1 88

48 1 62 2

54 1

Algorithms and Data Structures 2 // 2021W // 32 A. Ferscha

INSERT IN AVL TREES
44
After inserting 54:

17 78 z

32 y 50 88

48 62 x

Algorithms and Data Structures 2 // 2021W // 33 A. Ferscha

INSERT IN AVL TREES
44
After inserting 54:

17 78 z

32 y 50 88

48 62 x T3

T0 54 T2

Algorithms and Data Structures 2 // 2021W // 34 A. Ferscha

INSERT IN AVL TREES
44 x, y, z renamed according to
After inserting 54: in-order traversal

17 78 z/c

32 y/a 50 88

48 62 x/b T3

T0 54 T2

Algorithms and Data Structures 2 // 2021W // 35 A. Ferscha

INSERT IN AVL TREES
44 4 replace z by b
After inserting 54: with a and c as children

2 17 3 62 x/b

1 32 y/a 50 2 2 78 z/c

1 48 1 54 88 1

T0 T1 T3

Algorithms and Data Structures 2 // 2021W // 36 A. Ferscha

RESTRUCTURING IN AVL TREES
(SINGLE ROTATIONS)
a=z b=y
b=y a=z c=x
c=x

T0 T3
T1 T3 T0 T1 T2
T2

c=z
b=y
b=y
a=x c=z
a=x

T3
T0
T0 T2
T1 T2 T3
T1 Algorithms and Data Structures 2 // 2021W // 37 A. Ferscha
RESTRUCTURING IN AVL TREES
(DOUBLE ROTATIONS)
a=z b=x

c=y a=z c=y

b=x

T0
T2
T2 T3 T0 T1 T3
T1

c=z b=x

a=y a=y c=z

b=x

T3 T1
T0 T1 T0 T2 T3
T 2
Algorithms and Data Structures 2 // 2021W // 38 A. Ferscha
ALGORITHM FOR RESTRUCTURING

Algorithm restructure (x):

Input:
node x of a binary tree T with parent y and grandparent z
Output:
tree T restructured by (single or double) rotation of nodes x, y, z

1.Let (a,b,c) be the in-order-sequence of the nodes x, y, z and let

(T0, T1, T2, T3) be the in-order-sequence of the subtrees of x, y and z
2.Replace subtree with root z by subtree with root b
3.Assign a as left child of b and assign T0 and T1 as left/right subtree of a
4.Assign c as right child of b and T2, T3 as left/right subtree of c

Algorithms and Data Structures 2 // 2021W // 39 A. Ferscha

CUT/LINK RESTRUCTURING ALGORITHM

Each tree to be balanced can be divided into 7 parts:

• nodes x, y, z and 44 z
• 4 subtrees (T0, T1, T2, T3),
with the children of x, y, z as root
17 62 y

50 78 x
Create a new tree from the 7 parts, which is balanced
and in which the in-order sequence of the parts is retained. T0
48 54 88
T2

T1 T3

Algorithms and Data Structures 2 // 2021W // 40 A. Ferscha

CUT/LINK RESTRUCTURING ALGORITHM

Example 2
number 7 parts according to in-order traversal 44 z
1 4
17 62 y
3 6
50 78 x
T0 5 7
48 54 88
T2

T1 T3

Algorithms and Data Structures 2 // 2021W // 41 A. Ferscha

CUT/LINK RESTRUCTURING ALGORITHM

Example
Create an array with indices 1..7

1 2 3 4 5 6 7
“Cut“ the 4 subtrees and place them into the array
according to their numbering.

“Cut" x, y and z (in the order child, parent, grandparent) T0 T1 T2 T3

and put them in the array according to their numbering
1 2 3 4 5 6 7

T0 z=a T1 y=b T2 x=c T3

1 2 3 4 5 6 7

Algorithms and Data Structures 2 // 2021W // 42 A. Ferscha

CUT/LINK RESTRUCTURING ALGORITHM

Example
• Reassemble the tree again
• Set element
at position 4 (b) as root
4 T0 z=a T1 y=b T2 x=c T3
62 b
1 2 3 4 5 6 7

Algorithms and Data Structures 2 // 2021W // 43 A. Ferscha

CUT/LINK RESTRUCTURING ALGORITHM

Example
• Reassemble the tree again
• Set elements
at position 2 and 6 as children
4 T0 z=a T1 y=b T2 x=c T3
62 b
1 2 3 4 5 6 7
2 6
44 a 78 c

Algorithms and Data Structures 2 // 2021W // 44 A. Ferscha

CUT/LINK RESTRUCTURING ALGORITHM

Example
• Reassemble the tree again
• Set elements at position
1, 3 or 5, 7 as child of 2 and 6
4 T0 z=a T1 y=b T2 x=c T3
62 b
1 2 3 4 5 6 7
2 6
44 a 78 c
1 3 5 7
17 50 T2 88

T0 48 54 T3

Algorithms and Data Structures 2 // 2021W // 45 A. Ferscha

CUT/LINK RESTRUCTURING ALGORITHM

Cut/Link restructuring algorithm has the same effects as the four rotation cases previously considered

Advantage:
• No case distinction necessary
• More „elegant“ solution

Disadvantage:
• May require more code

The two procedures do not differ in terms of complexity (runtime).

Algorithms and Data Structures 2 // 2021W // 46 A. Ferscha

REMOVAL OF NODES

Using removeAboveExternal(w) can unbalance a tree:

• Let z be the first unbalanced node which is visited while traversing up in the tree.

• Let y be the child of z with the largest height and let x be the child of y with largest height.

• The algorithm restructure(x) can be used to restructure and balance the subtree with root z.

• However, restructuring can destroy the balance at higher levels, so that the verification
(and restructuring if necessary) must be continued until the root node is reached.

Algorithms and Data Structures 2 // 2021W // 47 A. Ferscha

REMOVAL OF NODES

Example
44

17 62

32 50 78

48 54 88

remove

Algorithms and Data Structures 2 // 2021W // 48 A. Ferscha

REMOVAL OF NODES

Example
44 z unbalanced!

17 62 y

50 78 x
T0
48 54 88
T2

T1 T3

AVL TREES :: COMPLEXITY

Possible complexity: N = number of keys

size(), isEmpty() Q(1)

get(), insert(), remove() O(log N)
getAll(), removeAll() O(log N + s)

(s = Number of elements in respective enumeration)

Algorithms and Data Structures 2 // 2021W // 53 A. Ferscha

BALANCED TREES
Balanced trees are introduced as a compromise between balanced and natural search trees,
whereby logarithmic search complexity is required in the worst case.

For the height hb of an AVL tree with N nodes we have:

ëlog2 Nû £ hb £ 1,44 * log2 (N+2)
• The upper limit can be derived from Fibonacci trees, a subclass of the AVL trees.
• Let N(h) be the minimum number of nodes of a height-balanced tree with height h. We have:
• N(0)=1, N(1)=2, N(2)=4, N(3)=7, N(4)=12, N(5)=20, …
• N(h) = 1 + N(h-1) + N(h-2) = Fib(h+3) – 1 0,1,1,2,3,5,8,13,21, …
• Fib(h) = 1/Ö5 * ( ((1 + Ö5)/2)h - ((1 - Ö5)/2)h )
• for all h we have: Fib(h) ³ 1/Ö5 * ((1 + Ö5)/2)h – 1
• if N(h)=Fib(h+3)-1 we have: log2 (N(h)+2) ³ log2 (1/Ö5) + (h+3) log2 ((1 + Ö5)/2))

• From this the estimation follows: h £ 1,44 log2 (N(h)+2) à h = O (log N(h))
Minimum number of nodes grows exponential with height
à so vice versa: height grows logarithmically with node number

Algorithms and Data Structures 2 // 2021W // 54 A. Ferscha

TREES (HEIGHT BALANCED)

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at
Trees (Weight Balanced)

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria

Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at

Balanced Trees

Two different approaches

◦ Height-balanced trees:

The maximum allowed height difference of the two subtrees is limited

◦ Weight-balanced trees:

The ratio of the node weights (number of nodes) of both subtrees

meets certain conditions.

Algorithms and Data Structures 2 // 2021W // 2 A. Ferscha

Balanced Trees

Balanced trees are introduced as a compromise between balanced and natural search trees,
whereby logarithmic search complexity is required in the worst case.

For the height hb of an AVL tree with N nodes we have:

◦ ëlog2 Nû £ hb £ 1,44 * log2 (N+2)
◦ The upper limit can be derived from Fibonacci trees, a subclass of the AVL trees.
◦ Let N(h) be the minimum number of nodes of a height-balanced tree with height h. We have:
§ N(0)=1, N(1)=2, N(2)=4, N(3)=7, N(4)=12, N(5)=20, …
§ N(h) = 1 + N(h-1) + N(h-2) = Fib(h+3) – 1
§ Fib(h) = 1/Ö5 * ( ((1 + Ö5)/2)h - ((1 - Ö5)/2)h ) 0,1,1,2,3,5,8,13,21, …
§ for all h we have: Fib(h) ³ 1/Ö5 * ((1 + Ö5)/2)h – 1
§ if N(h)=Fib(h+3)-1 we have: log2 (N(h)+2) ³ log2 (1/Ö5) + (h+3) log2 ((1 + Ö5)/2))
◦ From this the estimation follows: h £ 1,44 log2 (N(h)+2) à h = O (log N(h))
Minimum number of nodes grows exponential with height
à so vice versa: height grows logarithmically with node number

Algorithms and Data Structures 2 // 2021W // 3 A. Ferscha

Weight-Balanced Search Trees

Weight-balanced or BB trees (bounded balance):

Tolerated deviation of the structure from balanced binary tree
is defined as the difference between the number of
nodes in the right and left subtree.

Definition:
Let B be a binary search tree with left subtree Bl and l be the number of nodes in Bl (let N be the
corresponding number of nodes in B)
◦ r ( B ) = ( l + 1 ) / ( N + 1) is the root balance of B.
◦ A tree B is weight-balanced ( BB(a ) ) or of limited balance a,
if for each subtree B‘ of B we have: a £ r ( B‘ ) £ 1 - a

Algorithms and Data Structures 2 // 2021W // 4 A. Ferscha

Weight-Balanced Search Trees

Parameter a as degree of freedom in the tree

◦ a = 1/2 : Balancing criterion only accepts complete binary trees
◦ a < 1/2 : Structural restriction is increasingly relaxed
What effects does the relaxation of the balancing criterion have on costs?

Rebalancing
◦ Use of the same rotation types as for the AVL tree
◦ is guaranteed by the choice of a £ 1 - Ö2 / 2

Search and update costs: O (log2 N)

Algorithms and Data Structures 2 // 2021W // 5 A. Ferscha

Multipath Search Trees

Each internal node of a multipath search tree

◦ has d children, where d ³ 2
◦ Stores a set of records(k, x) where k is the key and
x the element 22
◦ Number of records: d-1
◦ Additionally: 2 pseudo records: k0 = -¥ and kd = ¥
5 10 25

For all children of an internal node we have:

◦ Keys of the children lie between 3 4 6 8 14 23 24 27
the keys of the respective records

External nodes are placeholders 11 13 17 20 21

Algorithms and Data Structures 2 // 2021W // 6 A. Ferscha

Multipath Search

Similar to binary search

• If k < k1 search in the leftmost child 22 8<22 à left subtree
• If k > kd-1 search in the rightmost child
• If d>2:
5 10 5<8<10 à middle subtree 25
find key ki-1 and ki for which:
ki-1 < k < ki and continue 8=8 à found!
search in child vi
3 4 6 8 14 23 24 27

Example: Search 8
11 13 17 20 21

Algorithms and Data Structures 2 // 2021W // 7 A. Ferscha

Multipath Search

Similar to binary search

• If k < k1 search in the leftmost child 22 12<22 à left subtree
• If k > kd-1 search in the rightmost child
• If d>2: 12>10 à right subtree
5 10 25
find key ki-1 and ki for which:
ki-1 < k < ki and continue 12<14 à left subtree
search in child vi
3 4 6 8 14 23 24 27

Example: Search 12
11<12<13 à
11 13 17 20 21
middle subtree
External node à unsuccessful!

Algorithms and Data Structures 2 // 2021W // 8 A. Ferscha

(2,4) Trees
width property
Special case of a multipath tree:
◦ Each node has a minimum/maximum of 2/4 children
◦ All external nodes have the same depth depth property

◦ Height of the tree is O(log N)

5 10 15

3 4 6 8 11 13 14 17

Algorithms and Data Structures 2 // 2021W // 9 A. Ferscha

Insert in (2,4) Trees

Insert key in lowest internal node, that has been reached during search.

insert g
Case 1: d d g
Node has first record

insert f
Case 2: d g d fg
Node has 2 records
fn

Case 3:
n
Node has already 3 records
◦ Node splitting: Split node into two insert e
d fg d e g
1-element nodes and move middle
element into parent node

Algorithms and Data Structures 2 // 2021W // 10 A. Ferscha

Insert in (2,4) trees

Top-Down Insertion
◦ Starting with the root, node-splitting is done for each node with three elements, that is
visited on the way when searching for the insertion position
◦ This ensures that inserting can be done according to case 1 or 2.

split concerns
constant number of nodes à O(1)
Bottom-Up Insertion
◦ Insertion position searched
◦ If the node at the insertion position has already 3 elements, node-splitting is done
◦ If this results in an “overflow” in the parent node (by moving the middle element upwards),
node-splitting is done again.

the maximum number of levels affected is

log N à O(log N)

Algorithms and Data Structures 2 // 2021W // 11 A. Ferscha

Removal in (2,4) Trees

Principle
◦ Find the record to be deleted via key
◦ Remove the entry and merge (inverse operation to split), if node has too few entries
Example 5 10 6 10
remove 3

3 6 8 11 5 8 11

The following special cases must be considered in detail:

◦ If node has internal nodes among its children: reduce this case to the case where node has only
external nodes as successors:
§ Search for previous entry according to in-order traversing
§ Swap entry with this predecessor
§ is repeated until the entry is at the lowest level of the tree.

Algorithms and Data Structures 2 // 2021W // 12 A. Ferscha

Removal in (2,4) Trees

The entry to be deleted is the last entry in the node:

◦ Get entry from parent in this node
◦ Replace "gap" in parent node with entry from sibling (transfer)

5 10 6 10
remove 3
Example: Remove 3

3 6 8 11 5 8 11

underflow!
therefore transfer

Algorithms and Data Structures 2 // 2021W // 13 A. Ferscha

Removal in (2,4) Trees

All siblings have only one records:

◦ Get entry from parent node
◦ “Merge“ 2 siblings
6 10 6
remove 11 merge

3 8 11 5 8 810
10

Example: Remove 11

Algorithms and Data Structures 2 // 2021W // 14 A. Ferscha

Removal in (2,4) Trees

Parent has only one record:

◦ Merging propagates upwards
12

Example: Remove 14
5 10 15

3 4 6 8 11 14 17

Algorithms and Data Structures 2 // 2021W // 15 A. Ferscha

Removal in (2,4) trees

Parent has only one record:

Merging propagates upwards
12
Example: Remove 14
merge

5 10 5 10 12 15
12
merge

3 4 6 8 11 14
15 151717

Algorithms and Data Structures 2 // 2021W // 16 A. Ferscha

(2,4) Trees :: Complexity

Splitting, Transfer, Merge: O(1)

Search
◦ Runtime corresponds to height of the tree, therefore O(log N)
Insert
◦ During search O(log N) nodes are visited
◦ Inserting requires (at maximum) O(log N) node-splitting operations
◦ A node-splitting operation can be done in constant time, O(1)
◦ Therefore the overall complexity for insert is: O(log N)
Remove
◦ During search O(log N) nodes are visited
◦ Remove requires at maximum O(log N) update operations (Transfer, Merge)
◦ Therefore the overall complexity for remove is: O(log N)

Algorithms and Data Structures 2 // 2021W // 17 A. Ferscha

From (2,4) Trees to Red-Black trees

Properties of (2,4) trees:

• balanced
• Search, Insert, Remove: O(log N)
• but: structure!!

Red-Black trees
• balanced
• Search, Insert, Remove: O(log N)
• Binary tree structure!

Algorithms and Data Structures 2 // 2021W // 18 A. Ferscha

Red-black trees
(2,4) tree Red-Black tree
Red-black trees are
another way of
representing (2,4) trees.

Algorithms and Data Structures 2 // 2021W // 19 A. Ferscha

Red-black trees

A red-black tree is a binary search tree with the following properties

◦ Edges are colored red or black
◦ In no path from root to leaf there are two consecutive red edges
◦ The number of black edges is the same for each path from the root to a leaf ("black height")
◦ Edges that lead to leaves are always black.
Root is
4 (always) black

Children of red nodes

are (always) black 2 7

1 3 6 8 External nodes are

(always) black
5 9

Algorithms and Data Structures 2 // 2021W // 20 A. Ferscha

Properties of red-black trees

Let
N be the number of internal nodes
L be the number of leaves (L = N+1)
H be the height
B be the Black Height (height according to black edges)
Property 1:
2B £ N+1 £ 4B
Property 2:
1/2 log2 (N+1) £ B £ log2 (N+1)
Property 3:
Search algorithm in Red-Black trees is
log2 (N+1) £ H £ 2 log2 (N+1) identical to search in binary trees

Implies a search complexity of O(log N)

Height is twice as high as that of the corresponding (2,4) tree

Algorithms and Data Structures 2 // 2021W // 21 A. Ferscha

Insert in red-black trees

• Search for leaf position at which the new key is to be inserted.

• Replace leaf with internal node containing new key
• Color the edge leading into the new node red.
• Attach two new leaves to the new node via black edges

2 2
insert 1

• If the parent of the new node has already an incoming red edge, two red edges would follow
each other. Therefore restructuring by rotation or promotion is required.

Algorithms and Data Structures 2 // 2021W // 22 A. Ferscha

Insert in red-black trees

Restructuring by rotation
(single)

n ... new node

p ... parent
g ... grandparent

Algorithms and Data Structures 2 // 2021W // 23 A. Ferscha

Insert in red-black trees

Restructuring by rotation
(double)

n ... new node

p ... parent
g ... grandparent

Algorithms and Data Structures 2 // 2021W // 24 A. Ferscha

Insert in red-black trees

Restructuring by promotion (if sibling of p is also red)

n ... new node
p ... parent
g ... grandparent
g g
p „recoloring“ p
n n

T2 T2
T1 T1

Can propagate upwards (if parent of g has a red incoming edge)

Algorithms and Data Structures 2 // 2021W // 25 A. Ferscha

Insert in red-black trees :: Summary

If two red edges follow each other after insertion, then

◦ restructure tree by single or double rotation - done!
◦ or recolor edges (if necessary propagate upwards)

z v

Algorithms and Data Structures 2 // 2021W // 29 A. Ferscha

Removal in red-black trees

Elimination of double-black colored edges

◦ Search „nearby“ red edge and change colors from (red, double-black) in (black, black)
Case 2: black sibling with black child;
Recoloring (with possible propagation upwards)
p p

v s v s

p p

v s v s

Algorithms and Data Structures 2 // 2021W // 30 A. Ferscha

Removal in red-black trees

Elimination of double-black colored edges

◦ Search „nearby“ red edge and change colors from (red, double-black) in (black, black)
Case 3: red sibling; Adjustment

p s

v s p

Then proceed according to case 1 or 2.

Algorithms and Data Structures 2 // 2021W // 31 A. Ferscha

Red-black Trees :: Summary

Insert or Remove can cause a local interference (successive red or double-black colored edges)

Resolving the interference

◦ Locally by restructuring
◦ Globally by propagation on higher levels (recoloring)

Complexity
◦ One restructure or recolor step: O(1)
◦ Insert: at maximum 1 restructure step or O(log N) recolor steps
◦ Remove: at maximum 2 restructure steps or O(log N) recolor steps
◦ Overall complexity: O(log N)

Algorithms and Data Structures 2 // 2021W // 32 A. Ferscha

Splay Trees

Binary search tree, in which “splaying” is done after each access operation => adaption to search
queries
◦ Splaying: special move-to-root operation is applied to node x
◦ Three cases for one step in splaying:
§ zick-zick: x is right (left) child of y, y is right (left) child of z
z
x
splaying step
y
y
T1
x T4
z
T2
T3

T3 T4
T1 T2
Algorithms and Data Structures 2 // 2021W // 33 A. Ferscha
Splay Trees

Three cases for one step in splaying:

◦ zick-zack: x is left (right) child of y, y is right (left) child of z

y
splaying step x
T1
x
z y
T4

T2 T3
T1 T2 T3 T4

Algorithms and Data Structures 2 // 2021W // 34 A. Ferscha

Splay Trees

Three cases for one step in splaying:

Algorithms and Data Structures 2 // 2021W // 43 A. Ferscha

Splay Trees

6
8

4 7

3 5 10

Algorithms and Data Structures 2 // 2021W // 47 A. Ferscha

Splay Trees :: Analysis

• Insert, Remove, Search: O(h)

(with h = depth of the tree)

• in worst case: h = N therefore O(N)

(see example of insertion)

• An amortized analysis (using the accounting method) can show that on average we have:
O(log N)

Algorithms and Data Structures 2 // 2021W // 48 A. Ferscha

Splay Trees :: Analysis

Algorithms and Data Structures 2 // 2021W // 49 A. Ferscha

Trees (Weight Balanced)

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria

Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at

Hashing

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at
Motivation

Dictionary data structures until now:

• ordered storage with usage of keys k ∈ K
At a time only a small amount of keys K from the amount of all possible keys K is in use (k ∈ K)
• search, remove, insert always requires a series of key comparisons

Hashing:
• Try to do this without key comparisons, i.e. determine by calculation where a data set with key k ∈ K is stored.

Hashtable:
• Data set are stored in an array A[0..N-1]

Hashfunction:
• h: K → {0, ..., N-1} assigns a hash address to each key k (= index in the hash table)
0 ≤ h(k) ≤ N-1
• Since N is generally much smaller than K, h() is generally not injective
• Example: Symbol table: 51 reserved words in Java with more than 6280 allowed identifiers with ≤ 80 digits.

Algorithms and Data Structures 2 // 2021W // 2 A. Ferscha

Motivation

Synonyms
• Keys k, k‘∈ K are synonymous if h(k) = h(k‘)

Address collision
• The same hash address is assigned to synonyms
• No synonyms, no collision
• Address collision requires special handling

Occupancy factor
• For a hash table of size N, that currently stores n keys, we specify 𝜶 = n/N as the occupancy factor.

Two requirements on hashing methods:

1. Choose h() in a way, so that as few collisions as possible occur = Selection of a “good” hash function
2. Address collisions should be resolved as efficiently as possible

Algorithms and Data Structures 2 // 2021W // 3 A. Ferscha

Hash Tables

Hash tables:
Efficient implementation of a dictionary with regard to storage space and complexity of
search, insert and remove operations (usually better than implementations based on key
comparisons)
• Key-value pairs are stored in an array of size N
• Index is calculated from the hash function value of the key h(k).
Aim: store item(k,e) at A[h(k)]
• Example: Use key k modulo array size as index and use chaining,
if two keys are mapped to the same index (collision)

A[0] 5
A[1] 11 1
Chaining:
A[2]
Keys with same index are stored in a list.
A[3] 13
A[4] 9 24

Algorithms and Data Structures 2 // 2021W // 4 A. Ferscha

Hash Function

h() often consist of two mapping functions: h = h1 ° h2

• hash code: h1 :key ® integer (k now integer = Hashcode)
• compression map: h2 :k ® [0 .. N-1]

k= h1(s) ... Hashcode

(-address)

h2(k) = h2 (k´) Û k and k´

[0 .. N-1] are synonyms
Hash
K‘ code K
K‘ is the key set in any data type (e.g. s Î K‘ string)
K is the key set where k Î K integer
(but often the key k can also be used directly as hash code)

„Good“ hash function:

• Easy to calculate
• Possible keys distributed as evenly as possible across indexes
• Probability of a collision should be minimized

Algorithms and Data Structures 2 // 2021W // 5 A. Ferscha

Hash Codes in Java

First part of the hash function (h1) assigns an integer to any key k
= Hash code or hash value

in Java: hashCode() method returns 32 bit int (!) for each object
• (in many Java implementations, however, this is only the memory address of the object,
i.e. a bad distribution => bad hash codes => overload with better method)

Example: Integer cast

• for numeric data types with 32 bits or less, the bits can be interpreted as int: typecast of byte, short, int, char

Exmaple: Component sum

• for numeric types with more than 32 bits (long, double) add 32-bit components
public static int hashCode(long i) {
return (int) ((i >> 32) + (int) i);
}
(upper 32 bit) + (lower 32 bit)

Algorithms and Data Structures 2 // 2021W // 6 A. Ferscha

Hash Codes: Polynomial Accumulation

Consider binary representation of the key as (x0, x1, x2, ... xk-1):
simple accumulation results in bad hash code because e.g. „spot“, „stop“, „tops“ ... Collide.
for (Java-)Strings therefore:
Consider the character values (ASCII or unicode) x0x1...xn-1 as coefficients of a polynomial
x0 ak-1 + x1 ak-2 + ... + xk-2 a + xk-1
Calculation according to Horner scheme (overflows are ignored) for certain value a ¹ 1
xk-1 + a ( xk-2 + a (xk-3 + ... + a (x1 + a x0)...))
For e.g. a=33, 37, 39, or 41 there are only 6 collisions in a vocabulary of 50.000 (english) words
public static int hashCode(String s) {
int h = 0;
for (int i = 0; i < s.length(); i++) {
h = (h << 5) | (h >>> 27); // 5-bit cyclic shift
h += (int) s.charAt(i); // add next character
}
return h;
}

Algorithms and Data Structures 2 // 2021W // 7 A. Ferscha

Compression Maps – Hash Functions

Division-Reminder-Method
• h(k) = |k| mod N
• Choice of N even (odd), then h(k) also even (odd)
• Bad if e.g. the last bit expresses a fact
(e.g. 0 = male, 1 = female)
• Choice of N = 2p
• h(k) returns the p lowest dual digits of k: bad because remaining bits are neglected
• Choice of N as prime number: N ¹ ri ± j, 0 £ j £ r-1, ... r = radix
(proves best in practice, empirically best results)

MAD - Multiply, Add, and Divide

• h(k) = |ak+b| mod N ... N prime, a, b ³ 0, a mod N ¹ 0
• eliminates „patterns“ in keys of the form iN+j
• Collision probability for two keys £ 1/N
• the same formula is also used in linear congruent (pseudo)random number generators

Algorithms and Data Structures 2 // 2021W // 8 A. Ferscha

Compression Maps – Hash Functions

Multiplicative Method - Requirement [Turan Sos]:

Let Y be an irrational number. If you place n points

Y - ëYû, 2 Y - ë2 Yû, 3 Y - ë3 Yû, ..., n Y - ën Yû

in the interval [0,1], then the resulting n+1 intervals have at most three different lengths.

Algorithms and Data Structures 2 // 2021W // 9 A. Ferscha

Compression Maps – Hash Functions
5−1
Multiplicative Method - Requirement [Turan Sos]: Y= ≈ 0,618
2

0,618 Y − Y

0,236 <- 1,236 2Y − 2Y

0,854 <- 1,854 3Y − 3Y

0,472 <- 2,472 4Y − 4Y

Algorithms and Data Structures 2 // 2021W // 10 A. Ferscha

Compression Maps – Hash Functions

Multiplicative Method - Requirement [Turan Sos]:

Let Y be an irrational number. If you place n points

Y - ëYû, 2 Y - ë2 Yû, 3 Y - ë3 Yû, ..., n Y - ën Yû

in the interval [0,1], then the resulting n+1 intervals have at most three different lengths.

If you divide further, the next point (n+1) Y - ë(n+1) Yû falls into the largest partial interval.

Of all numbers 0 £ Y £ 1 the golden ration Y = (Ö5 -1)/2 leads to the most balanced intervals.

h(k) = ë N (k Y - ë k Y û ) û forms exactly the permutation for N=10

h(1) = 6, h(2) = 2, h(3)= 8, =4, =0, =7, =3, =9, =5, h(10)=1
(.. and always divides exactly in the golden ratio)

Algorithms and Data Structures 2 // 2021W // 11 A. Ferscha

The “Golden Number”

.... 1,618034

Leonardo Pisano Fibonacci

(c. 1170 – c. 1240–50)

Algorithms and Data Structures 2 // 2021W // 12 A. Ferscha

The “Golden Ratio”
Logo Design based ca. 61,8 % ca. 38,2 %
𝒙 𝟏
𝟏
= 𝒙#𝟏
, thus x2 – x = 1 or: x2 -x-1=0 on the Golden Ratio

with the solutions

X1 = (1 + Ö 5 ) /2) = 1,61803
X2 = ( Ö 5 - 1 ) /2) = 0,61803
Google Apple

McDonald’s Toyota
1

National
1.618 Geographic

Parthenon in Athens, Greece

Algorithms and Data Structures 2 // 2021W // 13 A. Ferscha

Compression Maps – Hash Functions

Multiplicative Method
• Choose constant Y with 0 < Y < 1
• Calculate k Y mod 1 = k Y - ëk Y û k
• h(k) = ëN(k Y mod 1)û 0, Y
• Choice of N not critical
with N = 2p the r1 , r0
calculation of h(k) can be accelerated.
p Bits = h(k)
Example:

Y = (Ö5 -1)/2 » 0,6180339...

k = 123456
N = 1024
h(k) = ë1024(123456 × 0,6180339... mod 1)û
= ë1024(76300,0041151... mod 1)û can be implemented as shift
= ë4,213...û = 4
= ë41,151...û = 41 .. Calculation error

Algorithms and Data Structures 2 // 2021W // 16 A. Ferscha

Perfect Hashing

If the number of keys to be stored is known and ½ K ½ £ N, collision-free storing is always possible!

Form the injective mapping h: K ® {0, ..., N-1}:

1. arrange the keys k Î K in lexicographic order
2. assign (unique) order numbers to the keys

Collisions are completely avoided: perfect hashing

Application example:
Keywords of a programming language are assigned to fixed places in a symbol table.

Algorithms and Data Structures 2 // 2021W // 17 A. Ferscha

Analysis of Ideal Hashing

Assumptions
• n data items inserted into a memory with N places
• there have been no deletions
• All configrations of n occupied and N-n nonoccupied storage locations have the same probability
If Pr is the probability that exactly r places must be tested in the unsuccessful search, then we have:

• the first r-1 places are occupied, the rth place is free
• on the remaining m-r places the other n-(r-1) occupied places can be distributed arbitrarily

Algorithms and Data Structures 2 // 2021W // 18 A. Ferscha

Analysis of Ideal Hashing
Expected number of searched items if search failed

Expected number of searched items for successful search

with H(N) = 1 + 1/2 + 1/3 + ... + 1/N » ln N

Algorithms and Data Structures 2 // 2021W // 19 A. Ferscha

Universal Hashing (Randomisation)

Observation:
current key set K Ì K is generally not “equally distributed” from the universe of keys K
(example: programmers’ preference for variables i, i1, i2, i3, ... )

Problem: If fixed h is chosen Þ K Ì K can be constructed with arbitrarily many collisions!

Idea: Universal Hashing

• Choose hash function h randomly from a finite number of hash functions H
h Î H : K ® { 0, ..., N-1}
• Definition: H is universal, if for arbitrary x, y Î K we have:
• Conclusion: x, y Î K arbitrary, H universal, h Î H random

• (Probability that x, y is mapped to the same hash address of a random h Î H)

Algorithms and Data Structures 2 // 2021W // 20 A. Ferscha

Example for Universal Hashing

In other words:
H is universal if the number of hash functions to which h(k)=h(l) applies is at maximum equal to |H|/N for each pair of different keys.
Universal Hash functions exist / and are „easy“ to create:

Hash table A of size N=3 and p=5 (prime number) Keys K = {0, 1, 2, 3, 4}
20 hash functions H = hi,j(x) = ((ix + j) mod 5) mod 3 1 £ i < p, 0 £ j < p
1x+0 2x+0 3x+0 4x+0
1x+1 2x+1 3x+1 4x+1
1x+2 2x+2 3x+2 4x+2
1x+3 2x+3 3x+3 4x+3
1x+4 2x+4 3x+4 4x+4
each (mod 5) (mod 3) For two randomly chosen keys 1, 4
there is one collision in 4 of the 20
Example: Consider e.g. keys 1 and 4 hash functions
h(1) = h(4) occurs in 4 of 20 hash functions (x+0, x+4, 4x+0, 4x+4)
(1×1 + 0) mod 5 mod 3 = 1 = (1×4 + 0) mod 5 mod 3 in the other 16 there are
(1×1 + 4) mod 5 mod 3 = 0 = (1×4 + 4) mod 5 mod 3
0 collisions
(4×1 + 0) mod 5 mod 3 = 1 = (4×4 + 0) mod 5 mod 3
(4×1 + 4) mod 5 mod 3 = 0 = (4×4 + 4) mod 5 mod 3
i.e. PrH (hi,j(x) = hi,j(y)) £ 4/20=1/5 for all hash functions hi,j(x) Î H
i.e. H is universal

Algorithms and Data Structures 2 // 2021W // 21 A. Ferscha

Universal Hashing

Recommended approach:

Known:
The number of keys |K| which has to be mapped to N hash addresses.

Choose:
1. a prime number p which is greater than or equal to |K|
2. two numbers i, j in the range 1 £ i < p, 0 £ j < p

Then:
h(x) = ((ix + j) mod p) mod N
is a “good” hash function

Algorithms and Data Structures 2 // 2021W // 22 A. Ferscha

Universal Hashing

Definition
1 … 𝑖𝑓 ℎ 𝑥 = ℎ 𝑦 𝑎𝑛𝑑 𝑥 ≠ 𝑦
𝛿 𝑥, 𝑦, ℎ = )
0 … 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
d shows if collisions occur for two keys from K regarding h()

Extension of d to a set Y Í K and H

d(x,Y,h) = Sy Î Y d(x,y,h)
d(x,y,H) = Sh Î H d(x,y,h)

H is universal, if for two arbitrary x,y Î K (x¹y) we have

Algorithms and Data Structures 2 // 2021W // 23 A. Ferscha

Universal Hashing
Unknown: number of |K|
Known:
H : K ® { 0, ..., N-1} a universal set of hash functions
h Î H a randomly chosen hash function used for all insertions,
then the place h(x) can already be occupied for the insertion attempt x.
For the insertion attempt x there are already S keys stored:
E[d(x,S,h)] = Sh Î H d(x,S,h) / |H|

= 1/|H| Sh Î H Sy Î S d(x,y,h)
= 1/|H| Sy Î S Sh Î H d(x,y,h)
= 1/|H| Sy Î S d(x,y,H)
£ 1/|H| Sy Î S |H| / N
= |S| / N

i.e. the expected number of already inserted elements which probably have collided with x is |S| / N
This means that an arbitrarily chosen hash function h from a universal set H, will map sequences of keys (no matter
how unilaterally they are) to available hash addresses as evenly as possible.

Algorithms and Data Structures 2 // 2021W // 24 A. Ferscha

Overview :: Collision Handling

Inserting a synonym k‘, if key k is already stored: Collision (place h(k) = h(k‘) is already occupied)
h(k‘) is referred to as overflow

Solution 1: Overflow chaining

• Keys with the same index are stored in an overflow list at the corresponding index position.
• Is also known as closed hash procedure.

Solution 2: Open hashing

• If a key is to be inserted at a position that is already occupied, another free (vacant) position
is selected and the key is stored at this position.
• More details regarding this follow later.

Algorithms and Data Structures 2 // 2021W // 25 A. Ferscha

Chaining

Example:
Insert sequence:
25, 2, 15, 50, 13, 6, 20

Hash Function:
h(k) = k mod 7

Method:
Each element of the hash table is a
Overflows in reference to an overflow chain.
linked list

Algorithms and Data Structures 2 // 2021W // 26 A. Ferscha

Operations in Hash Tables with Chaining

Search for key k

• calculate h(k) and reference A[h(k)] in the overflow list
• search for k in the overflow list until it is found or the end of the list is reached (not found)

Insert a key k
• search for k as described above (ends unsuccessfully – otherwise it will not be inserted)
• create list element for k and insert it in the overflow list

Remove a key k
• search for k as described above
• if successful, remove from overflow list

All operations are based on pure list operations.

Algorithms and Data Structures 2 // 2021W // 27 A. Ferscha

Analysis of Hash Tables with Chaining

Uniform Hashing assumption:

• all hash addresses are chosen with equal probability: Pr ( h(ki) = j ) = 1/N
• Independent from operation to operation (above Pr for each 0 £ j £ N-1)

Average overflow list length for n entries (also: occupancy factor)

• n/N = a

Complexity of the search: (new keys are always added to the end of the overflow list)
C´n expected number of searched position for unsuccessful search
C´n = n/N = a
Cn expected number of searched positions for successful search
Cn = 1/n Sj=1..n (1+(j-1)/N) = 1+ (n-1) / 2m » 1 + a /2

Algorithms and Data Structures 2 // 2021W // 28 A. Ferscha

Open Hashing
Idea:
• Placement of overflows k‘ (h(k‘) = h(k)) at vacant position in the hash table
• According to the rule: if A[h(k)] occupied, search other position for k‘
• Sequence of chosen positions: probing sequence
• Basic problem: Selection of a suitable probing sequence
Example: Consider entry with next smaller index (h(k) – 1) mod N
Problem: Recovery of k‘ if k is removed in the meantime

Generalization: consider entry with

• (h(k) - s(j,k)) mod N j = 0, ..., N-1 for a given function s(j,k)

Common variants of s(j,k):

• linear probing: s(j,k) = j
• quadratic probing: s(j,k) = (-1)j ×éj/2ù2

Algorithms and Data Structures 2 // 2021W // 29 A. Ferscha

Open Hashing

Properties of s(j,k)
Sequence
h(k) - s(0,k) mod N
h(k) - s(1,k) mod N
...
h(k) - s(N-2,k) mod N
h(k) - s(N-1,k) mod N

Is a permutation of the hash addresses 0, ..., N-1

e.g. quadratic probing

h(11) = 4 0 1 2 3 4 5 6
s(j,k) = (-1)j ×éj/2ù2 = 0, -1, 1, -4, 4, -9, 9

Algorithms and Data Structures 2 // 2021W // 30 A. Ferscha

Open Hashing :: Linear Probing

Probe function: s(j,k) = j

Probe sequence: h(k), h(k)-1, h(k) –2, ... 0, N-1, ..., h(k)+1
Example: N=7, K = {0, 1, ..., 500}, h(k) = k mod N, Keys: 12, 53, 5, 15, 2, 19

A[0] A[1] A[2] A[3] A[4] A[5] A[6]

53 12 „coalescing“

5 long (occupied) parts

tend to grow further
A[0] A[1] A[2] A[3] A[4] A[5] A[6]

5 53 12
"primary
accumulation“
A[0] A[1] A[2] A[3] A[4] A[5] A[6]
6 inspections! Efficiency gets worse
15 2 5 53 12
drastically near a=1

19
Algorithms and Data Structures 2 // 2021W // 31 A. Ferscha
Open Hashing :: Quadratic Probing

Aim: avoid „primary accumulation“

Probe function: s(j,k) = (-1)j ×éj/2ù2 = 0, -1, 1, -4, 4, -9, 9
Probe sequence: h(k), h(k)+1, h(k)-1, h(k)+4, h(k)-4, ...
Example: N=7, K = {0, 1, ..., 500}, h(k) = k mod N, Keys: 12, 53, 5, 15, 2, 19

A[0] A[1] A[2] A[3] A[4] A[5] A[6]

53 12
Two synonyms always
traverse through the same
5 probe sequence
A[0] A[1] A[2] A[3] A[4] A[5] A[6]
= interfere each other
53 12 5

A[0] A[1] A[2] A[3] A[4] A[5] A[6] A[2]

„secondary accumulation“
6 inspections! 15 2 53 12 5 2

Algorithms and Data Structures 2 // 2021W // 32

19 A. Ferscha
Open Hashing :: Uniform Probing

Aim: avoid “primary“ and “secondary accumulation“

Reason for accumulation: Probing function is independent of k !
(probe function is the same for all synonyms)
Probing function: s(j,k) for j = 0, ..., N-1 is a permutation of the hash addresses that depends only on k,
where each of the N! possible permutations is used with equal probability
(Uniform probing)
+ asymptotically optimal!
- practically very difficult to realize

Random Probing: s(j,k) chooses a hash address randomly

Contrary to Uniform Probing: A value chosen for s(j,k)

can be “drawn” again later (j‘ > j)

Algorithms and Data Structures 2 // 2021W // 33 A. Ferscha

Analysis Open Hashing

Linear Probing
• Probe sequence: h(k), h(k)-1, h(k-2), ...
• Problem: primary clustering
• Cń » ( 1 + 1/(1-a)2) Cn » ( 1 + 1/(1-a))
Quadratic Probing
• Probe sequence: h(k), h(k)-1, h(k)+1, h(k)-4, h(k)+4, ...
• Permutation, if N = 4i+3, prime
• Problem: secondary clustering
• Cń » 1/(1-a) - a + ln( 1/(1-a) ) Cn » 1 - a/2 + ln( 1/(1-a) )
Uniform Probing
• s(j,k) = pk(j) pk one of N! permutations of {0,...,N-1}
• Each permutation has equal probability
• Cń £ 1/(1-a) Cn » 1/a × ln( 1/(1-a) )
Random Probing
• s(j,k) = random number dependent on k
• s(j,k) = s(j´,k) possible, but unlikely

Algorithms and Data Structures 2 // 2021W // 34 A. Ferscha

Generalization: consider entry with

• (h(k) - s(j,k)) mod N j = 0, ..., N-1 for a given function s(j,k)

Common variants of s(j,k):

• Linear Probing: s(j,k) = j
• Quadratic Probing: s(j,k) = (-1)j ×éj/2ù2
• Double Hashing: s(j,k) = j × h2(k)

Algorithms and Data Structures 2 // 2021W // 35 A. Ferscha

Hashing

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at
Double Hashing

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at
Open Hashing :: Double Hashing

Uniform probing efficiency is already achieved if a second hash function

is used instead of random permutation: Double Hashing
• Use two hash functions h1(k), h2(k)
double_hash_insert(k)
if(table is full) error
probe = h1(k)
offset = h2(k)
while(table[probe] occupied)
probe = (probe + offest) mod m
table[probe] = k

• Keys are distributed more equally than with linear probing

• (») same efficiency as uniform probing (if h1(k), h2(k) independent)

• Disadvantage of all open hashing procedures:

• Operations become slower and more complex (e.g. selecting or moving elements in remove operation)

Algorithms and Data Structures 2 // 2021W // 2 A. Ferscha

Open Hashing :: Double Hashing

Probing function: s(j,k) = j × h2(k)

Probing sequence: h1(k), h1(k) - h2(k), h1(k) - 2 × h2(k), ..., h1(k) - (N-1) × h2(k)
(each mod N)
h2 must be chosen so that the probing sequence forms a permutation of the hash addresses Þ N prime
Example: N=7, K = {0, 1, ..., 500}, Keys: 12, 53, 5, 15, 2, 19
h1(k) = k mod N, h2(k) = 1 + k mod(N-2) Insert of 5
h1(k) = k mod 7
h1(5) = 5 mod 7 = 5
A[0] A[1] A[2] A[3] A[4] A[5] A[6]
h2(5) = 1+ 5 mod 5 = 1
h2(k) = 1 + k mod 5
53 12
Probe sequence: 5, 4, 3
5
A[0] A[1] A[2] A[3] A[4] A[5] A[6]
Insert of 19
2 inspections! 5 53 12
h1(19) = 19 mod 7 = 5
h2(19) = 1+ 19 mod 5 = 5
19 15 2 5 53 12
Probe sequence: 5, 0
19
Algorithms and Data Structures 2 // 2021W // 3 A. Ferscha
Further Example for Double Hashing

Example
• h1(k) = k mod 7, h2(k) = 1 + k mod 5
Key sequence 15, 22, 1, 29, 26

h1(15) = 1
A[0] A[1] A[2] A[3] A[4] A[5] A[6] h1(22) = 1 h2(22) = 3
15 29 26 22 1 h1(1) = 1 h2(1) = 2
h1(29) = 1 h2(29) = 5
h1(26) = 5 h2(26) = 2

• Average search time: (1+2+2+2+5)/5 = 2.4

Average search time (1+1+3+1+1+1+2)/6 = 1.5

if in the order 53, 5, 15, 2, 19, 12 is inserted!!!

Algorithms and Data Structures 2 // 2021W // 4 A. Ferscha

Improvement of the Successful Search

When inserting:
• k encounters kold in A[i], i.e. i = h(k) - s(j,k) = h(kold) - s(j´,kold)
• kold already stored in A[i]
• Idea: Search vacant position for k or kold

Two options:
M1 : kold remains in A[i] and k tries insert position h(k)- s(j+1,k)
M2 : k pushes kold to h(kold) - s(j´+1, kold)

If M1 or M2 finds a vacant position:

then insert the appropriate key Þ done
otherwise proceed with M1 or M2

Algorithms and Data Structures 2 // 2021W // 5 A. Ferscha

Brent’s Algorithm
k encounters k´

Only follow M1 k yields k´ yields

k encounters k´´ done

k yields k´´ yields

k encounters k´´´ done

k yields k´´´ yields

done
k encounters k´´´´

Time for unsuccessful search remains unchanged

Cn´ » 1 / (1-a)

Time for successful search is reduced to

CnBrent » 1 + a/2 + a3/4 + a4/15 + a5/18 + ... < 2.5

Algorithms and Data Structures 2 // 2021W // 6 A. Ferscha

Example for Brent’s Algorithm
Insert of 5
Example: N=7, K = {0, 1, ..., 500}, Keys: 12, 53, 5, 15, 2, 19
h1(5) = 5 mod 7 = 5 occupied,
h1(k) = k mod 7, h2(k) = 1 + k mod 5
calculate
A[0] A[1] A[2] A[3] A[4] A[5] A[6] h2(5) = 1+ 5 mod 5 = 1
and
53 12 h2(12) = 1 + 12 mod 5 = 3

5: probe (5-1)mod7=4 occupied,

Insert of 2
5 12: probe (5-3)mod7=2 vacant,
A[0] A[1] A[2] A[3] A[4] A[5] A[6] therfore 12 yields to 5
h1(2) = 2 mod 7 = 2 occupied,

calculate
19 15 12 53 5 2 Insert of 19
h2(2) = 1+ 2 mod 5 = 3
and h1(19) = 19 mod 7 = 5 occupied,
h2(12) = 1 + 12 mod 5 = 3 2 19
calculated
2: probe (2-3)mod7=6 h2(19) = 1+ 19 mod 5 = 5
vacant and
h2(5) = 1 + 5 mod 5 = 1

19: probe (5-5)mod7=0 vacant

Algorithms and Data Structures 2 // 2021W // 7 A. Ferscha
Further Example for Brent’s Algorithm

Example
h1(k) = k mod 7, h2 (k) = 1 + k mod 5
key sequence 15, 22, 1, 29, 26 h1(15) = 1

A[0] A[1] A[2] A[3] A[4] A[5] A[6] h1(22) = 1 h2(22) = 3

15 22 29 22 1
26
h1(1) = 1 h2(1) = 2

h1(29) = 1 h2(29) = 5

h1(26) = 5 h2(26) = 2 occupied

h2(22) = 3 vacant
◦ Durchschnittliche Suchzeit = (1+2+2+2+2)/5 = 9/5 = 1.8

Algorithms and Data Structures 2 // 2021W // 8 A. Ferscha

Binary Tree Probing

Follow both M1 as well as M2 simultaneously, until a vacant position is found in a subbranch

k encounters k´

k yields k´ yields

k encounters k´´ k´ encounters k´´´

k yields k´´ yields k´ yields k´´´ yields

k encounters k´´´´ k´´ encounters k´´´´´ k´ encounters k´´´´´´ k´´ encounters k´´´´´´´
. . . .
. . . .
• Time for unsuccessful. search remains unchanged
. . .
• Cn´ » 1 / (1-a)
• Time for successful search is reduced to
• Cnbinary tree < 2.2

Algorithms and Data Structures 2 // 2021W // 9 A. Ferscha

Example for Binary Tree Probing
Insert of 5
Example:
A[0] A[1] A[2] A[3] A[4] A[5] A[6]
N=7, K = {0, 1, ..., 500}, h1(5) = 5 mod 7 = 5 occupied,
Keys: 12, 53, 5, 15, 2, 19, 21 53 12 calculate
h2(5) = 1+ 5 mod 5 = 1
Insert of 2 and
h1(k) = k mod 7,
5 h2(12) = 1 + 12 mod 5 = 3
h2(k) = 1 + k mod 5 h1(2) = 2 mod 7 = 2 occupied,
calculate 5: probe (5-1)mod7=4 occupied,
A[0] A[1] A[2] A[3] A[4] A[5] A[6]
h2(2) = 1+ 2 mod 5 = 3 12: probe (5-3)mod7=2 vacant,
and 19 15 12 53 5 2 therefore 12 yields to 5
h2(12) = 1 + 12 mod 5 = 3

2: probe (2-3)mod7=6 Insert of 19

vacant 21 2 19
h1(19) = 19 mod 7 = 5 occupied,
calculate
Insert of 21 h2(19) = 1+ 19 mod 5 = 5
and
h1(21) = 21 mod 7 = 0 occupied,
h2(5) = 1 + 5 mod 5 = 1
calculate
h2(21) = 1+ 21 mod 5 = 2 and h2(19) = 1 + 19 mod 5 = 5
19: probe (5-5)mod7=0 vacant
probe 21: (0-2)mod7=5 occupied and 19: (0-5)mod7=2 occupied,
continue probing for 21 AND 19
Algorithms and Data Structures 2 // 2021W // 10 A. Ferscha
Example for Binary Tree Probing

Example:
A[0] A[1] A[2] A[3] A[4] A[5] A[6] A[0] A[1] A[2] A[3] A[4] A[5] A[6]
N=7, K = {0, 1, ..., 500},
Keys: 12, 53, 5, 15, 2, 19, 21 19 15 12 53 5 2 21 15 12 53 5 2
h1(k) = k mod 7,
h2(k) = 1 + k mod 5 21 19

Subbranch for 21 Subbranch for 19

(assumption: 21 would have displaced 19)
calculate
h2(21) = 1 + 21 mod 5 = 2 calculate
and h2(19) = 1 + 19 mod 5 = 5
h2(5) = 1 + 5 mod 5 = 1 and
h2(12) = 1 + 12 mod 5 = 3
probe 21: (5-2)mod7=3 and 5: (5-1)mod7=4
Position 3 vacant, insert 21 probe 19: (2-5)mod7=4 and 12: (2-3)mod7=6
both occupied
A[0] A[1] A[2] A[3] A[4] A[5] A[6]

19 15 12 21 53 5 2

Algorithms and Data Structures 2 // 2021W // 11 A. Ferscha

Further Example for Binary Tree Probing

Example
§ h1(k) = k mod 7, h2 (k) = 1 + k mod 5
key sequence 15, 22, 1, 29, 26
h1(15) = 1

A[0] A[1] A[2] A[3] A[4] A[5] A[6] h1(22) = 1 h2(22) = 3

15 22 29 22 1
26
h1(1) = 1 h2(1) = 2

h1(29) = 1 h2(29) = 0

h1(26) = 5 h2(26) = 2
h2(22) = 3
§ Durchschnittliche Suchzeit = (1+2+2+2+2)/5 = 9/5 = 1.8
§ hier identisch mit Brent‘s Algorithmus, da jeweils im ersten Sondierungsschritt Lücke gefunden wird

Algorithms and Data Structures 2 // 2021W // 12 A. Ferscha

Ordered Double Hashing

Aim: Improvement of unsuccessful search

Search for k:
• k´> k in probe sequence => search unsuccessful
Rule for inserting:
• Smaller keys displace larger keys (ordered hashing)
Invariant:
• all keys in the probe sequence before k are smaller than k
(but not necessarily sorted in ascending order)

Problems:
• Displacement can trigger "chain reaction”
• k´ displaced by k: Position of k´ in probe sequence? => (s(j,k) - s(j-1,k) = s(1,k)) 1£j£N

Algorithms and Data Structures 2 // 2021W // 13 A. Ferscha

Example :: Ordered Double Hashing

Hash functions:
h1(k) = k mod 7, h2(k) = 1 + k mod 5
Key sequence: 15, 2, 43, 4, 8

A[0] A[1] A[2] A[3] A[4] A[5] A[6]

15 15
43 8 2 43 43
4
h1(15) = 1
h1(2) = 2
h1(43) = 1 43>15 h2(43) = 4 (1-4) mod 7 = 4
h1(4) = 4 4<43 h2(43) = 4 (4-4) = 0
h1(8) = 1 8<15 h2(15) = 1 (1-1) = 0
15<43 h2(43) = 4 (0-4) mod 7 = 3

Algorithms and Data Structures 2 // 2021W // 14 A. Ferscha

Hashing Complexity :: Summary

Occupancy factor a = average number of keys per array index

# 𝑠𝑡𝑜𝑟𝑒𝑑 𝑘𝑒𝑦𝑠 |𝑆| 𝑛

= = =
𝑠𝑖𝑧𝑒 𝑜𝑓 ℎ𝑎𝑠ℎ 𝑡𝑎𝑏𝑙𝑒 𝑁 𝑁

Complexity after probabilistic analysis

Expected number of probes
unsuccessful successful
Chaining a 1 + a/2
Linear Probing 1 + 1/(1-a)2 1 + 1/(1-a)
Double Hashing 1/(1-a) 1/a × ln 1/(1-a)

Algorithms and Data Structures 2 // 2021W // 18 A. Ferscha

Double Hashing

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at
Monte Carlo Tree Search

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at
Introduction
Search trees
• Search (no adversary), analyse traversing strategies, evaluation of cost
Game trees
• Games (adversary), game states (board configurations) have utility, tree defines decision process (decision tree),
find strategy
• Example: 2-Player Games: Two players, fully observable environments, deterministic, turn-taking, zero-sum
games of perfect information (e.g., go, chess, backgammonm, tic-tac-toe, etc.)

Consideration of a Game as a Search Problem:

• States = board configurations
• Operators = legal moves
• Initial State = current configuration
• Goal = find winning configuration
• payoff function (utility) = gives numerical value of outcome of the game

• Two players, MIN and MAX taking turns

◦ MIN/MAX use search tree to find next move

Algorithms and Data Structures 2 // 2021W // 2 A. Ferscha

Game Trees and the Minimax Algorithm
• How can MIN/MAX determine
which move to pick to win the game?

• We know for each terminal state the outcome of the

game – this is called the utility.

• In each turn, both players want to select

a node which results in the best utility for them.
1. Generate whole game tree to leaves
2. Apply utility function to leaves
3. Back up values from leaves to root
- MAX nodes compute maximum of children
- MIN nodes compute minimum of children
4. When value reaches root: choose max value
and the corresponding move Deterministic, perfect information Tic-Tac-Toe game tree
of 2 players (5,478 valid game states).
Games and Adversarial Search - Marco Chiarandini

Algorithms and Data Structures 2 // 2021W // 3 A. Ferscha

Minimax Algorithm

Minimax value
• Is the best utility that can be reached from a current node n onwards,
assuming that both players play optimally from n to the end of the game:

𝑈𝑡𝑖𝑙𝑖𝑡𝑦 𝑛 𝑖𝑓 𝑛 𝑖𝑠 𝑎 𝑡𝑒𝑟𝑚𝑖𝑛𝑎𝑙 𝑛𝑜𝑑𝑒

MINIMAX-VALUE (n)=! min 𝑠∈𝑆𝑢𝑐𝑐𝑒𝑠𝑠𝑜𝑟 𝑛 𝑀𝐼𝑁𝐼𝑀𝐴𝑋 − 𝑉𝐴𝐿𝑈𝐸 𝑠 𝑖𝑓 𝑛 𝑖𝑠 𝑎 𝑀𝐼𝑁 𝑛𝑜𝑑𝑒
m𝑎𝑥 𝑠∈𝑆𝑢𝑐𝑐𝑒𝑠𝑠𝑜𝑟 𝑛 𝑀𝐼𝑁𝐼𝑀𝐴𝑋 − 𝑉𝐴𝐿𝑈𝐸 𝑠 𝑖𝑓 𝑛 𝑖𝑠 𝑎 𝑀𝐴𝑋 𝑛𝑜𝑑𝑒

• MAX will try to move to states with maximum values.

• MIN will try to move to states with minimum values.

Games and Adversarial Search - Marco Chiarandini

Algorithms and Data Structures 2 // 2021W // 4 A. Ferscha

Minimax Algorithm :: Example

Step 1:
• The entire decision tree is
A MAX generated (meaning we
expand every possible
move).
B C MIN • The utility function is applied
to get the terminal values for
each node.
D E F G MAX

H I J K L M N O Terminal Nodes

4 8 9 3 2 -2 9 -1 Utility

m ... maximum length of any path in the state space (may be infinite).

Games and Adversarial Search - Marco Chiarandini

Algorithms and Data Structures 2 // 2021W // 10 A. Ferscha

Alpha-Beta Pruning

Main disadvantage of Minimax

• Minimax has to look into every node of the game tree.

A MAX

8 B 2 C MIN

8 D 9 E 2 F 9 G MAX

H I J K L M N O Terminal Nodes

4 8 9 3 2 -2 9 -1 Utility
The entire right branch of node A
will never lead to success

Algorithms and Data Structures 2 // 2021W // 11 A. Ferscha

Alpha-Beta Pruning

Method
Propagate two parameters along the expansion of a path, and update them when backing up: [α, β].
• α ... best (largest) value found so far for MAX.
• β ... best (smallest) value found so far for MIN.

Pruning
• Whenever a Minimax value as a child of a MIN node is less than or equal to the current α:
➔ ignore remaining nodes (subtrees) below this MIN node.
• Whenever a Minimax value as a child of a MAX node is greater than or equal to the current β:
➔ ignore remaining nodes (subtrees) below this MAX node.

Games and Adversarial Search - Marco Chiarandini

Algorithms and Data Structures 2 // 2021W // 12 A. Ferscha

Alpha-Beta Pruning

Basic algorithm outline

Max-Value(s, ⍺, β): Min-Value(s, ⍺, β):

if terminal(s): return U(s) if terminal(s): return U(s)
v = -∞ v = +∞
for c in next-states(s): for c in next-states(s):
v’ = min-value(c, ⍺, β) v’ = max-value(c, ⍺, β)
if v’ > v: v = v’ if v’ < v: v = v’
if v’ ≥ β: return v if v’ ≤ ⍺: return v
if v’ > ⍺: ⍺ = v’ if v’ < β: β = v’
return v return v

Based on an example from John Levine (https://www.youtube.com/watch?v=zp3VMe0Jpf8)

Algorithms and Data Structures 2 // 2021W // 13 A. Ferscha

Alpha-Beta Pruning :: Principle

⍺=
A β= MAX

⍺= ⍺=
B β= C β= MIN

⍺= ⍺= ⍺= ⍺=
D β= E F G MAX
β= β= β=

H I J K L M N O Terminal Nodes

4 8 9 3 2 -2 9 -1 Utility

Alpha-Beta Pruning :: Example
This is a
MIN node
Step 4:
so we
update β ⍺ = -∞ • In node B MIN finds the value 8
A β = +∞ MAX of node D.
• 8 < β(+∞): β is updated to 8
⍺ = -∞ and the value of B is updated to
8 B β=8 C MIN 8.
• β is passed down to node E.
⍺=8 ⍺ = -∞
8 D β = +∞ E β=8 F G MAX

H I J K L M N O Terminal Nodes

4 8 9 3 2 -2 9 -1 Utility

Based on an example from John Levine (https://www.youtube.com/watch?v=zp3VMe0Jpf8)

Algorithms and Data Structures 2 // 2021W // 20 A. Ferscha

Alpha-Beta Pruning :: Example

Step 5:

⍺ = -∞ • In node E MAX finds the value

A β = +∞ MAX 9 of node J.
• 9 > β (8): the remaining
⍺ = -∞ branches of E are pruned.
8 B β=8 C MIN

⍺=8 ⍺ = -∞
8 D β = +∞ 9 E β = 8 F G MAX

H I J K L M N O Terminal Nodes

4 8 9 3 2 -2 9 -1 Utility

9 > β(8)
Based on an example from John Levine (https://www.youtube.com/watch?v=zp3VMe0Jpf8)

Algorithms and Data Structures 2 // 2021W // 21 A. Ferscha

Alpha-Beta Pruning :: Example

Step 6:
8 > ⍺(-∞) • In node A MAX finds the value
⍺=8
8 A β = +∞ MAX 8 of node B.
• 8 > ⍺(-∞): ⍺ is updated to 8
⍺ = -∞ ⍺=8 and the value of node A is
8 B β=8 C β = +∞ MIN updated to 8.
• ⍺ is down propagated to node
⍺=8 ⍺ = -∞ ⍺=8
8 D β = +∞ 9 E β = 8 F G MAX F.
β = +∞

H I J K L M N O Terminal Nodes

4 8 9 3 2 -2 9 -1 Utility

Based on an example from John Levine (https://www.youtube.com/watch?v=zp3VMe0Jpf8)

Algorithms and Data Structures 2 // 2021W // 22 A. Ferscha

Alpha-Beta Pruning :: Example

Step 7:

⍺=8 • In node F MAX finds the value

8 A β = +∞ MAX 2 of node L.
• 2 < ⍺(8): ⍺ is not updated.
⍺ = -∞ ⍺=8
8 B β=8 C β = +∞ MIN

⍺=8 ⍺ = -∞ ⍺=8
8 D β = +∞ 9 E β=8 2 F β = +∞ G MAX

H I J K L M N O Terminal Nodes

4 8 9 3 2 -2 9 -1 Utility
2 < ⍺(8)
Based on an example from John Levine (https://www.youtube.com/watch?v=zp3VMe0Jpf8)

Algorithms and Data Structures 2 // 2021W // 23 A. Ferscha

Alpha-Beta Pruning :: Example

Step 8:

⍺=8 • Since we have not found a value

8 A β = +∞ MAX >= ⍺,
MAX looks into node M to find a
⍺ = -∞ ⍺=8 value of -2.
8 B β=8 C β = +∞ MIN • -2 < ⍺(8): ⍺ is not updated.

⍺=8 ⍺ = -∞ ⍺=8
8 D β = +∞ 9 E β=8 2 F β = +∞ G MAX

H I J K L M N O Terminal Nodes

4 8 9 3 2 -2 9 -1 Utility
-2 < ⍺(8)
Based on an example from John Levine (https://www.youtube.com/watch?v=zp3VMe0Jpf8)

Algorithms and Data Structures 2 // 2021W // 24 A. Ferscha

Alpha-Beta Pruning :: Example

Step 9:
⍺=8 • In node C MIN finds the
8 A β = +∞ MAX
value 2 of node F.

8
⍺ = -∞
2
⍺=8 • 2 < ⍺(8): the remaining
B β=8 C β = +∞ MIN
branches of C are
pruned.
⍺=8 ⍺ = -∞ ⍺=8
8 D β = +∞ 9 E β=8 2 F β = +∞ G MAX

H I J K L M N O Terminal Nodes

4 8 9 3 2 -2 9 -1 Utility
2 < ⍺(8)
Based on an example from John Levine (https://www.youtube.com/watch?v=zp3VMe0Jpf8)

Algorithms and Data Structures 2 // 2021W // 25 A. Ferscha

Alpha-Beta Pruning :: Example

Result

⍺=8 • The resulting path is the same

8 A β = +∞ MAX as in Minimax,
but fewer nodes had to be
⍺ = -∞ ⍺=8 analyzed.
8 B β=8 2 C β = +∞ MIN

⍺=8 ⍺ = -∞ ⍺=8
8 D β = +∞ 9 E β=8 2 F β = +∞ G MAX

H I J K L M N O Terminal Nodes

4 8 9 3 2 -2 9 -1 Utility

Based on an example from John Levine (https://www.youtube.com/watch?v=zp3VMe0Jpf8)

Algorithms and Data Structures 2 // 2021W // 26 A. Ferscha

Alpha-Beta Pruning

Properties of Alpha-Beta pruning

• Does not affect the final result.
• With perfect ordering, time complexity would be O(bm/2).

Limitations of Minimax and Alpha-Beta Pruning

• Minimax traverses the entire game tree.
• Alpha-Beta pruning still has to search all the way to terminal states of many nodes.

Can we do better?
• While both algorithms have many applications,
in certain scenarios they might reach their limits.
• This is where we can apply Monte Carlo Tree Search.

Games and Adversarial Search - Marco Chiarandini

Algorithms and Data Structures 2 // 2021W // 27 A. Ferscha

Game Trees and the Minimax Algorithm
• How can MIN/MAX determine
which move to pick to win the game?

• We know for each terminal state the outcome of the

game – this is called the utility.

• In each turn, both players want to select

Algorithms and Data Structures 2 // 2021W // 28 A. Ferscha

Monte Carlo Tree Search

Basic Algorithm Outline

The basic algorithm involves iteratively building a search tree until some predefined
computational budget – typically a time, memory or iteration constraint – is reached.

At this point the search is halted and the best performing root action is returned.

Each node in the search tree represents a state of the domain.

Directed links to child nodes represent actions leading to subsequent states.

A Survey of Monte Carlo Tree Search Methods, Cameron Browne, 2012

Algorithms and Data Structures 2 // 2021W // 29 A. Ferscha

Monte Carlo Tree Search

Motivation: the game of Go

• Played on a 19x19 board by two players in alternating
moves by placing black stones and white stones
respectively on the board.
• Opponent’s stones can be captured once they are fully
surrounded by own stones.
• No move can lead to a game state that has
been present in the move directly before (no immediate
repetitions).
• The goal is to occupy a larger area of the game board
than the opponent. There are 2.08*10170 valid game sates in the game of Go.

Algorithms and Data Structures 2 // 2021W // 31 A. Ferscha

Monte Carlo Tree Search

Basic algorithm Outline

• Selection: Find a leaf node from which to
traverse next.
• Expansion: Child nodes are added to expand
the tree.
• Simulation: A rollout from the new node(s) to a
terminal node is done.
• Backpropagation: The simulation result is
“backed up” through the selected nodes to
update their statistics.

A Survey of Monte Carlo Tree Search Methods, Cameron Browne, 2012

Algorithms and Data Structures 2 // 2021W // 32 A. Ferscha

Monte Carlo Tree Search
Basic algorithm Outline
Rollout(Si):
loop forever:
if Si is a terminal state:
return value(Si)
Ai = random(available_actions(Si))
Si = simulate(Ai, Si)

Si v = 10

Random decisions Backpropagation

throughout the tree of value estimate
down to a terminal node

A Survey of Monte Carlo Tree Search Methods, Cameron Browne, 2012 v = 10

Algorithms and Data Structures 2 // 2021W // 33 A. Ferscha

Monte Carlo Tree Search

How to select a leaf node?

Similarly to Minimax, we need to find some value that gives the branch a score,
determining the most promising path.

Tree Policy
In MCTS the most widely used utility function is called Upper Confidence Bound (UCB1).

UCB1 =

A value of 2 for the tunable parameter C has been used in the past to yield promising results.

A Survey of Monte Carlo Tree Search Methods, Cameron Browne, 2012

Algorithms and Data Structures 2 // 2021W // 34 A. Ferscha

Monte Carlo Tree Search

How to choose which route to follow in the expanded node?

Once we decided which branch to expand, we need to find some strategy to traverse through that
branch down to its terminal node.

Default Policy
Play out the domain from a given non-terminal state to produce a value estimate (simulation). In the
simplest case these are just random moves.

A Survey of Monte Carlo Tree Search Methods, Cameron Browne, 2012

Algorithms and Data Structures 2 // 2021W // 35 A. Ferscha

Monte Carlo Tree Search :: Example

Iteration 1 (Simulation Phase) v=0

s0 n = 0
• Since s1 is a leaf node that has not been
visited yet, we perform a rollout to a v=0 v=0
terminal state. s1 s2
n=0 n=0
• Via simulation we get a value estimate of
20. Remember, this is the result when Random decisions
playing out this branch to the end. throughout the tree
down to a terminal node

v = 20

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

Algorithms and Data Structures 2 // 2021W // 36 A. Ferscha

Monte Carlo Tree Search :: Example

Iteration 1 (Backpropagation Phase) v = 20

s0 n = 1
• The value estimate is backpropagated up to
the root node. v = 20 v=0
s1 s2
• This concludes the first iteration. n=1 n=0

Backpropagating and
summing up the value
estimate

v = 20

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

Algorithms and Data Structures 2 // 2021W // 37 A. Ferscha

Monte carlo tree search :: Example

Iteration 2 (Selection Phase) v = 20

s0 n = 1
• In the second iteration, again starting in s0,
we calculate the UCB1 (Upper Confidence v = 20 v=0
Bound) scores for s1 and s2. s1 s2
n=1 n=0
)*(,)
• UCB1(s1) = 20 +2 ,
= 20

• UCB1(s2) is still infinite.

• Since the UCB1 score of s2 is higher we
select this branch.

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

Algorithms and Data Structures 2 // 2021W // 38 A. Ferscha

Monte Carlo Tree Search :: Example

Iteration 2 (Simulation Phase) v = 20

s0 n = 1
• Since s2 is a leaf node which has not been
visited yet, we perform a rollout to a v = 20 v=0
terminal state. s1 s2
n=1 n=0
• Via simulation we get a value estimate of
10.

v = 10

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

Algorithms and Data Structures 2 // 2021W // 39 A. Ferscha

Monte Carlo Tree Search :: Example

Iteration 2 (Backpropagation Phase) v = 30

s0 n = 2
• The value estimate is backpropagated up to
the root node. v = 20 v = 10
s1 s2
• This concludes the second iteration. n=1 n=1

v = 10

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

Algorithms and Data Structures 2 // 2021W // 40 A. Ferscha

Monte Carlo Tree Search :: Example

Iteration 3 (Selection Phase) v = 30

s0 n = 2
• In the third iteration, again starting in s0,
we calculate the UCB1 scores for s1 and v = 20 v = 10
s2. s1 s2
n=1 n=1
)*(.)
• UCB1(s1) = 20 +2 ,
= 21.67

)*(.)
• UCB1(s2) = 10 +2 ,
= 11.67

• Since the UCB1 score of s1 is higher we

expand this branch.

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

Algorithms and Data Structures 2 // 2021W // 41 A. Ferscha

Monte Carlo Tree Search :: Example

Iteration 3 (Expansion Phase) v = 30

s0 n = 2
• Since s1 is a leaf node but has already
been visited, we expand the tree. v = 20 v = 10
s1 s2
n=1 n=1

s3 v = 0 s4 v = 0
n=0 n=0

Expand the tree by

adding two new nodes

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

Algorithms and Data Structures 2 // 2021W // 42 A. Ferscha

Monte Carlo Tree Search :: Example

Iteration 3 (Simulation Phase) v = 30

s0 n = 2
• We calculate the UCB1 score for both nodes,
s3 and s4. v = 20 v = 10
s1 s2
• Since the UCB1 score for both children infinite n=1 n=1
(no one has been visited before) we can again
choose to start with the leftmost child which is
s3.
s3 v = 0 s4 v = 0
• We perform the rollout and via simulation get n=0 n=0
a value estimate of 0.

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

v=0

Algorithms and Data Structures 2 // 2021W // 43 A. Ferscha

Monte Carlo Tree Search :: Example

Iteration 3 (Backpropagation Phase) v = 30

s0 n = 3
• The value estimate is now backpropagated up
to the root node. v = 20 v = 10
s1 s2
• This concludes the third iteration. n=2 n=1

s3 v = 0 s4 v = 0
n=1 n=0

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U) v=0

Algorithms and Data Structures 2 // 2021W // 44 A. Ferscha

Monte Carlo Tree Search :: Example

Iteration 4 (Selection Phase) v = 30

s0 n = 3
• In the fourth iteration, again starting in s0,
we calculate the UCB1 scores for s1 and v = 20 v = 10
s2. s1 s2
n=2 n=1
)*(/)
• UCB1(s1) = 20 +2 .
= 11.48

)*(/) s3 v = 0 s4 v = 0
• UCB1(s2) = 10 +2 ,
= 12.10 n=1 n=0
• Since the UCB1 score of s2 is higher we
explore this branch.

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

Algorithms and Data Structures 2 // 2021W // 45 A. Ferscha

Monte Carlo Tree Search :: Example

Iteration 4 (Expansion Phase) v = 30

s0 n = 3
• Since s2 is a leaf node but has already
been visited, we expand the tree. (s5, s6) v = 20 v = 10
s1 s2
n=2 n=1

s3 v = 0 s4 v = 0 s5 v = 0 s6 v = 0
n=1 n=0 n=0 n=0

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

Algorithms and Data Structures 2 // 2021W // 46 A. Ferscha

Monte Carlo Tree Search :: Example

Iteration 4 (Simulation Phase) v = 30

s0 n = 3
• We calculate the UCB1 score for both
nodes, s5 and s6. v = 20 v = 10
s1 s2
• Since the UCB1 score for both children is n=2 n=1
infinite (both have not been visited before)
we can again choose to start with the
leftmost node which is s5.
s3 v = 0 s4 v = 0 s5 v = 0 s6 v = 0
• We perform a rollout and via simulation n=1 n=0 n=0 n=0
get a value estimate of 14.

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

v = 14

Algorithms and Data Structures 2 // 2021W // 47 A. Ferscha

Monte Carlo Tree Search :: Example

Iteration 4 (Backpropagation Phase) v = 44

s0 n = 4
• The value estimate is backpropagated up to
the root node. v = 20 v = 24
s1 s2
• This concludes the fourth iteration. n=2 n=2

s3 v = 0 s4 v = 0 s5 v = 14 s6 v = 0
n=1 n=0 n=1 n=0

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

v = 14

Algorithms and Data Structures 2 // 2021W // 48 A. Ferscha

Monte Carlo Tree Search :: Example

Result v = 44
s0 n = 4
• Following the design of the MCTS algorithm
we could do as many iterations as we want. v = 20 v = 24
s1 s2
• However, if we were to stop now, the branches n=2 n=2
with the highest total scores would be optimal
to choose (which is s2 followed by s5).
• More iterations often improve results. s3 v = 0 s4 v = 0 s5 v = 14 s6 v = 0
n=1 n=0 n=1 n=0

Based on an example from John Levine (https://www.youtube.com/watch?v=UXW2yZndl7U)

Algorithms and Data Structures 2 // 2021W // 49 A. Ferscha

Algorithms and Data Structures 2 // 2021W // 50 A. Ferscha
Solving the Game of Go

Background
• Remember, in a 19x19 Go board there are 2.08*10170 valid game states.
• While boards with the size of 5x5 have successfully been solved in 2002,
19x19 boards have long been assumed unsolvable.

Approach by DeepMind via AlphaGo

• In 2015 DeepMind realized their idea of
solving Go via machine learning and MCTS.
• They combine two approaches in their implementation:
• Value networks to evaluate board positions
and policy networks to select moves.
• A search algorithm that combines Monte Carlo simulation
with value and policy networks.

Silver, D., Huang, A., Maddison, C. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).

Algorithms and Data Structures 2 // 2021W // 51 A. Ferscha

Solving the Game of Go

Training pipeline of AlphaGo

Rollout policy and Supervised Learning (SL) policy

• A rollout policy trained on 8 million human expert moves (accuracy of 24.2% in just 2𝝁s).
• A 13-layer convolutional neural network trained on 30 million moves of human experts (accuracy of 57% while
best result of other research groups was 44.4%).

Reinforcement Learning (RL) policy

• Aims to improve the Supervised Learning (SL-)policy through self-play by having the same architecture as the SL-
policy but initializing it with the final RL-weights.
• This adjusts the policy towards the correct goal of winning games rather than maximizing predictive accuracy.

Silver, D., Huang, A., Maddison, C. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).

Algorithms and Data Structures 2 // 2021W // 52 A. Ferscha

Solving the Game of Go
Training pipeline of AlphaGo (cont’d)

Value Networks
• A value network approximates the optimal value function.
• Trained by 30 million moves sampled from distinct games of self-play from RL-policy.
• While the policy networks reveal which moves are promising,
the value network determines how good a board position is.

Network overview

Silver, D., Huang, A., Maddison, C. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).

Algorithms and Data Structures 2 // 2021W // 53 A. Ferscha

Solving the Game of Go

MCTS during live play

• Up until now the models are trained but still have to be processed.
• Right now the network does not play any better than any state-of-the-art MCTS algorithm.
• The key factor is combing the neural networks with MCTS in what is called asynchronous policy and value MCTS
(APV-MCTS).

Evaluation of terminal nodes

(1) by the value network 𝑣𝜃(sL).
(2) by the outcome of MCTS simulations zL.
• These evaluations are combined into a terminal node evaluation using a mixing parameter
𝜆 tuned to 0.5 : 𝑉(sL) = (1- 𝜆)𝑣𝜃(sL) + 𝜆zL

Silver, D., Huang, A., Maddison, C. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).

Algorithms and Data Structures 2 // 2021W // 54 A. Ferscha

Solving the Game of Go

Evaluation of terminal nodes

Play strength of AlphaGo

When released in 2015, AlphaGo won against European Go champion Fan Hui
followed by winning 4 out of 5 matches against 18 times world champion Lee Se-dol.

Silver, D., Huang, A., Maddison, C. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).

Algorithms and Data Structures 2 // 2021W // 55 A. Ferscha

Monte Carlo Tree Search

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at
Graphs
Part: Structure

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at
Definition

A graph G = (V, E) consists of

V ... a quantity of vertices/nodes and

E ... a quantity of edges/arcs.

An edge e = (u,v) is a pair of vertices.

Example: a b

V = { a, b, c, d, e}
E = { (a,b), (a,c), (a,d), (b,e), (c,d), (c,e), (d,e)}
c

d e

Algorithms and Data Structures 2 // 2021W // 2 A. Ferscha

Definitions of Terms

Two vertices are adjacent to each other, if they are connected by an edge.
• Example: a and c are adjacent to each other

An edge connecting two adjacent vertices is called incident to these vertices.

The degree of a vertex deg(v) is defined as the number of vertices adjacent to it.

• We have: ! deg 𝑣 = 2 ∗ (#𝑒𝑑𝑔𝑒𝑠)

!∈# a b
• Example: deg(c) = 3

A path is a sequence of vertices (v1, v2, ..., vk) in which successive c

vertices vi and vi+1 are adjacent to each other.

• Example: path (a, b, e, d, c, e)

d e

Algorithms and Data Structures 2 // 2021W // 3 A. Ferscha

Definitions of Terms
In a simple path no vertex occurs more than once.
a b
• Example: a, b, e
c

d e

A cyclic path (short: cycle) is a simple path with the exception, that the first vertex in the path is identical to the last
vertex in the path.
• Example: a, b, e, c, a a b

d e
A graph is connected if any two of its vertices are connected to each other by a path.
A subgraph is a subset of vertices and edges of a graph, which in turn form a graph.

Algorithms and Data Structures 2 // 2021W // 4 A. Ferscha

Definitions of Terms

A connected component in a graph is a maximum connected subgraph.

• Example:
Graph with 3 connected components a b

c c c c

A tree is a connected graph without cycles. d e e

A forest is a set of trees.

tree

a b

c c c c

d e e

Algorithms and Data Structures 2 // 2021W // 5 A. Ferscha

Connectivity

In a complete graph each pair of vertices is adjacent to each other.

Let
n ... number of vertices and
m ... number of edges

1 1 𝑛(𝑛 − 1)
• in a complete graph we have: 𝑚= ! deg 𝑣 = ! 𝑛 − 1 =
2 2 2
!∈# !∈#

• in a non-complete graph we have: 𝑛(𝑛 − 1)

𝑚<
2
• in a tree we have: 𝑚 =𝑛−1

If m < n-1, then the graph is not connected (consists of more than one connected component).

Algorithms and Data Structures 2 // 2021W // 6 A. Ferscha

Spanning Tree

A spanning tree ST in a graph G is a subgraph of G for which we have:

• ST is a tree not
• ST contains all vertices of G unambiguous!

a b a b

c c

d e d e

G Spanning tree of G

If a single edge is removed, the graph is no longer connected.

Algorithms and Data Structures 2 // 2021W // 7 A. Ferscha

Directed Graph

An edge (v,w) is defined as directed, if it leads v to w, but not vice versa. Then (v,w) is an ordered pair.
• Illustration: as arrow

v w
a b
A graph is directed (digraph), if it contains directed edges.
c

d e

A directed graph without cycles is called a directed acyclic graph (DAG).

A digraph is strongly connected, if each vertex can be reached from any other vertex.
A strongly connected component in a digraph is a subgraph, in which each vertex can be reached from any other vertex.

Algorithms and Data Structures 2 // 2021W // 8 A. Ferscha

Data Structures for Storing Graphs
Edge list
BA901
Edges are stored in a list, from each edge a
reference leads to the end vertices stored in a BRU
separate data structure. OS122
SN4051 FRA
LHR SN3640 SN4054
SR533
LH2829
• reference to edge ZRH LNZ OS931 VIE
• (un)directed
• reference to vertex IW700 OS205
• position in container
CDG
AF1539

E LH2829 OS931 OS122 SR533 BA901 OS205 AF1539 SN4054 IW700 SN4051 SN3640

V
LNZ FRA VIE ZRH LHR BRU CDG
• in-degree
• out-degree

Algorithms and Data Structures 2 // 2021W // 9 A. Ferscha

Data Structures for Storing Graphs
Adjacency list (traditional)
• For each vertex its adjacent vertices are stored in a list.
• All vertices with adjacency lists are again stored in a list.

BRU

FRA
LHR

ZRH LNZ VIE

CDG

V LNZ FRA VIE ZRH LHR BRU CDG

FRA VIE ZRH BRU ZRH BRU

VIE ZRH CDG CDG

LHR

Algorithms and Data Structures 2 // 2021W // 10 A. Ferscha

Data Structures for Storing Graphs
Adjacency list (modern)
• Extension of the edge list by a list of the incident edges for each vertex

E LH2829 OS931 OS122 SR533 BA901 OS205 AF1539 SN4054 IW700 SN4051 SN3640

V
LNZ FRA VIE ZRH LHR BRU CDG

in out in out in out in out in out in out in out

LH2829 LH2829 OS122 OS931 AF1539 SR533 IW700 BA901 SN3640 SN4051 IW700 SN3640

OS931 SR533 OS122 OS205 SN4051 SN4054 SN4054 AF1539

BA901 OS205

• Incidency container
• (in/out count)

Algorithms and Data Structures 2 // 2021W // 11 A. Ferscha

Data Structures for Storing Graphs
Adjacency matrix (traditional)
• Matrix M where M[i,j] is true if an edge from vertex i to vertex j exists, or false otherwise.

to
from LNZ FRA VIE ZRH LHR BRU CDG

LNZ F T T F F F F

FRA F F T T T F F
VIE F F F T F F T

ZRH F F F F F T T

LHR F F F F F F F
BRU F F F T F F F
CDG F F F F F T F

Algorithms and Data Structures 2 // 2021W // 12 A. Ferscha

Data Structures for Storing Graphs
Adjacency matrix (modern)
• Matrix elements store a reference to an edge object

to
from LNZ FRA VIE ZRH LHR BRU CDG

LNZ F LH2829 OS931 F F F F

FRA F F OS122 SR533 BA901 F F

VIE F F F OS205 F F AF1539

ZRH F F F F F SN4054 IW700

LHR F F F F F F F
BRU F F F SN4051 F F F
CDG F F F F F SN3640 F

Algorithms and Data Structures 2 // 2021W // 13 A. Ferscha

Complexity Comparison Assumption:
• Edges, vertices as doubly linked list
• Container with position information

Complexity
Edge list Adjacency list Adjacency matrix
Space O(m+n) O(m+n) O(n2)
Operations:
size, isEmpty O(1) O(1) O(1)
vertices O(n) O(n) O(n)
edges O(m) O(m) O(m)
endVertices, opposite, isDirected O(1) O(1) O(1)
incidentEdges O(m) O(deg(v)) O(n)
areAdjacent O(m) O(min(deg(u),deg(v))) O(1)
insertVertex O(1) O(1) O(n2)
removeVertex O(m) O(deg(v)) O(n2)
insertEdge, insertDirectedEdge, removeEdge O(1) O(1) O(1)

Algorithms and Data Structures 2 // 2021W // 14 A. Ferscha

Graph Structure Analysis

Algorithms and Data Structures 2 // 2021W // 15 A. Ferscha

Graph Traversal – Depth-First Search (DFS)

Algorithm DFS(v);

Input: A vertex v in a graph

Output: A labeling of edges as "discovery" and "back" edges

for each edge e incident to v do

if edge e is unexplored then

let w be the other endpoint of e

if vertex w is unexplored then

label e as a discovery edge
recursively call DFS(w)

else
label e as a back edge

Algorithms and Data Structures 2 // 2021W // 16 A. Ferscha

DFS :: Example

A B C D

E F G H

I J K L

M N O P

discovery edge
back edge
Algorithms and Data Structures 2 // 2021W // 17 A. Ferscha
Properties of DFS

For an undirected graph G in which a DFS starting with vertex s is executed, we have:
• The traversal visits all vertices in the connected component (=maximum connected subgraph), which contains s.
• The set of discovery edges form a spanning tree for this connected component.

Complexity of DFS
• If ns is the number of vertices in a connected component with s and if ms is the number of edges in the connected
component of s, then the complexity of DFS is O(ns + ms) on the assumptions:
• Graph is stored so that access to the vertices and edges is O(1)
• Marking and testing of the edges is O(1)
• There is a mechanism that systematically searches the edges of a node without looking at an edge more than
once.

Algorithms and Data Structures 2 // 2021W // 18 A. Ferscha

DFS :: Finding a Path

Each vertex is labeled initially as UNEXPLORED Algorithm pathDFS(G,v,z);

and can be relabeled as VISITED. Input: A graph G and the vertices v (start) and z
(destination) in graph G
Output: Path as a sequence of vertices
Each edge is labeled initially as UNEXPLORED
and can be relabeled as DISCOVERY or BACK. setLabel(v, VISITED)
S.push(v)
if v = z
A stack S is used to keep track of the path return S.elements() and terminate algorithm
for all e Î G.incidentEdges(v)
between the start vertex and the current
if getLabel(e) = UNEXPLORED
vertex. w ← opposite(v,e)
if getLabel(w) = UNEXPLORED
setLabel(e, DISCOVERY)
When the destination vertex z is reached, the S.push(e)
stack content is returned as the path between pathDFS(G, w, z)
v and z. S.pop(e)
else
setLabel(e, BACK)
Based on Goodrich et al., Data Structures & Algorithms in Python S.pop(v)

Algorithms and Data Structures 2 // 2021W // 19 A. Ferscha

DFS :: Finding a Cycle
Algorithm cycleDFS(G,v);
Input: A vertex v (start) in a graph G
Each vertex is labeled initially as UNEXPLORED Output: Path of in the graph representing the cycle
and can be relabeled as VISITED. setLabel(v, VISITED)
S.push(v)
for all e ∈ G.incidentEdges(v)
Each edge is labeled initially as UNEXPLORED if getLabel(e) = UNEXPLORED
w ← opposite(v,e)
and can be relabeled as DISCOVERY. S.push(e)
if getLabel(w) = UNEXPLORED
setLabel(e, DISCOVERY)
A stack S is used to keep track of the path cycleDFS(G, w)
S.pop(e)
between the start vertex and the current vertex. else
T ← new empty stack
repeat
A cycle is detected when a back edge to an already o ← S.pop()
visited vertex is recognized, which is not a parent. T.push(o)
until o = w
return T.elements() and terminate
Based on Goodrich et al., Data Structures & Algorithms in Python algorithm
S.pop(v)

Algorithms and Data Structures 2 // 2021W // 20 A. Ferscha

Graph Traversal – Breadth-First Search (BFS)

Principle:
• Let s be the start node of BFS, set s to level 0
• In the first step, visit all (not yet visited) vertices that are adjacent to s and set them to level 1.
• In the next step visit for all vertices on level 1 all not yet visited adjacent vertices and set them to level 2.
• Repeat this step as long as all vertices have been reached.

Result:
• Traversal of the graph
• Level of a vertex v shows the length of the shortest path from v to s.

Algorithms and Data Structures 2 // 2021W // 21 A. Ferscha

Graph Traversal – Breadth-First Search (BFS)

Pseudo code example: Algorithm BFS(v);

Input: A vertex s in a graph

Output: A labeling of edges as "discovery" and "cross" edges

initialize container L0 to contain vertex s

i¬0
while Li is not empty do
create container Li+1 to initially be empty
for each vertex v in Li do
for each edge e incident on v do
if edge e is unexplored then
let w be the other endpoint of e
if vertex w is unexplored then
label e as discovery edge
insert w into Li+1
else
label e as cross edge
i ¬ i+1

Algorithms and Data Structures 2 // 2021W // 22 A. Ferscha

BFS :: Example

A B C D

E F G H

Level 1
I J K L
Level 2
Level 3
M N O P Level 4
Level 5

cross edge

Algorithms and Data Structures 2 // 2021W // 23 A. Ferscha

Properties of BFS

Let G be an undirected graph in which BFS is executed starting with vertex s, then we have:
• The traversal visits all vertices in the connected component, which contains s.
• The set of discovery edges form a spanning tree (BFS tree) for this connected component.
• For each vertex v on level i, the path to s along the BFS tree has length i
and any other path from v to s has at least length i.
• If (u,v) is an edge that is not in the BFS tree, then the levels of u and v differ by 1 at maximum.

Complexity of BFS (analogue DFS)

If n is the number of vertices of G and m the number of edges, then the complexity of BFS in G is O(n + m)
With the same complexity the following tasks can be solved:
• Test, if G is connected
• Calculation of a spanning tree in G
• Calculation of the connected components in G
• Calculation of a cycle in G (or that G has no cycles)

Algorithms and Data Structures 2 // 2021W // 24 A. Ferscha

Graph Traversal – DFS for Directed Graphs

Same algorithms as for undirected graphs

• The result can also be a set of unconnected DFS trees (forest)

Accessibility in directed graphs

• A DFS tree with root s contains all vertices that can be reached from s via directed edges.

Transitive closure
• The transitive closure G* to a graph G is obtained by inserting a directed edge (v,w) if v is reachable from w
(there is a directed path from v to w)

Calculation of the transitive closure for graph G:

• Apply DFS on each vertex of G
• Complexity: O( n (n+m) ) A B C
• Alternative: Floyd-Warshall Algorithm

Algorithms and Data Structures 2 // 2021W // 25 A. Ferscha

Floyd-Warshall Algorithm

Assumption: Operations areAdjacent and insertDirectedEdge have complexity O(1) (Graph is e.g. as adjacency matrix stored)

Algorithm FloydWarshall(G);
let v1 ... vn be an arbitrary ordering of the vertices of G0 = G
for k = 1 to n do
// consider all possible routing vertices vk
Gk = G k-1
for each ( i, j = 1,..,n ) ( i != j ) ( i,j != k ) do
// for each pair of vertices vi and vj
if Gk-1.areAdjacent(vi,vk) and Gk-1.areAdjacent(vk,vj) then
Gk. insertDirectedEdge(vi,vj,null)
return Gn

Digraph Gk is the subdigraph of the transitive closure of G,

which results from the paths to the intermediate vertices of set {v1, ..., vk }
Runtime: O(n3)

Algorithms and Data Structures 2 // 2021W // 26 A. Ferscha

Weighted Graph

An edge is weighted, if a numerical value is assigned to it.

• Value can represent for example a distance, travel time or cost.

110 weight(u,z)
u = FRA
z = VIE
BRU
85
80 FRA
LHR 55
65 75
ZRH 45
LNZ VIE
75 80
CDG
130

Algorithms and Data Structures 2 // 2021W // 27 A. Ferscha

Shortest Path – Dijkstra’s Algorithm

BFS finds path with minimum number of edges

• Corresponds to the shortest path if all edges have the same weight

Dijkstra‘s algorithm
• Finds shortest path for all vertices z to start vertex s in a graph
• with undirected edges and
• with non-negative edge weights
• based on greedy method

Algorithmic idea
• Set of nodes for which a path has already been found is stored in set C.
• D(z) denotes the shortest path from v to s found so far.
• When a new node u is visited, the system checks whether a route via this node to an already visited node z has
a shorter distance than the shortest route found so far, i.e. whether D[u] + weight(u,z) < D[z]
• If yes, then this path is saved via b as the new shortest path to z and D[z] is updated (relaxation)

Algorithms and Data Structures 2 // 2021W // 28 A. Ferscha

Dijkstra’s Algorithm

Algorithm ShortestPath(G,s);

Input: A weighted Graph G and distinguished vertex s of G

Output: labels D[u] for each vertex u of G giving the length of the shortest
path from u to s in G

initialize D[s] ¬ 0 and D[u] ¬ + ¥ for each vertex u ¹ s

let Q be a priority queue that contains all of the vertices of G

using the D labels as keys
while Q ¹ Æ do
// pull u into cloud C
u ¬ Q.removeMinElement()
for each vertex z adjazent to u such that z is in Q do
// perform relaxation operation on edge (u,z)
if D[u] + w((u,z)) < D[z] then
D[z] = D[u] + w((u,z))
change the key value of z in Q to D[z]
return label D[u] of each vertex u

Algorithms and Data Structures 2 // 2021W // 29 A. Ferscha

Dijkstra’s Algorithm :: Example

Search shortest path starting from LNZ

Initialization
110

¥
BRU
¥
85
¥ 80 FRA
LHR 55
65 75 0 ¥
¥ ZRH 45
LNZ VIE
75 80
CDG
¥ 160

Algorithms and Data Structures 2 // 2021W // 30 A. Ferscha

Dijkstra’s Algorithm :: Example

Search shortest path starting from LNZ

110

¥ 75
BRU
185 205 ¥
85
¥ 80 FRA
LHR 55
125 65 75 0 ¥
¥ ZRH
ZRH 45
LNZ VIE 45
75 80
CDG
¥ 160
205
200

Algorithms and Data Structures 2 // 2021W // 31 A. Ferscha

Runtime of Dijkstra’s Algorithm

Assumption: G is stored as adjacency list

Vertices that are adjacent to u can be visited in O(j),
where j is the number of vertices that are adjacent to u

Variation 1: Priority Queue Q implemented as heap

Initialization of all vertices in QO(log n) To be preferred when
number of edges is small:
Within the loop:
• removeMin O(log n) m < n2 / log n
• key updates O(log n) (if locators are used in the heap)
• Relaxation O(degree(u) log n )
Runtime of the loop
! 1 + 𝑑𝑒𝑔𝑟𝑒𝑒 𝑢 log(𝑛)
!∈#

Therefore total runtime of Dijkstra: O((n+m) log n)

Algorithms and Data Structures 2 // 2021W // 32 A. Ferscha

Runtime of Dijkstra’s Algorithm

Assumption: G is stored as adjacency list

Vertices that are adjacent to u can be visited in O(j),
where j is the number of vertices that are adjacent to u

Variation 2: Priority Queue as unsorted list To be preferred if the

number of edges is large:
Within the loop
• removeMin O(n) m > n2 / log n
• key updates O(1)
Therefore total runtime of Dijkstra: O(n2 + m)

Algorithms and Data Structures 2 // 2021W // 33 A. Ferscha

Minimum Spanning Tree (MST)

MST is a spanning tree with minimum total edge weight

Prim-Jarnik algorithm (similar to Dijkstra)

• Start with (any) start vertex v
• Tree is constructed vertex by vertex
• Insert edge (v,u) and vertex u, which are:
• Vertex v is already in the tree
• Vertex u is not yet in the tree
• The weight of (v,u) is the minimum of the weights of all possible candidates u.
• For each vertex D[u] is stored
• Indicates for a vertex that has not yet been visited, the minimum of the weights of all edges
with which u could be connected to the tree.
• Advantage: shortened search for minimum

Algorithms and Data Structures 2 // 2021W // 34 A. Ferscha

Prim-Jarnik Algorithm

Algorithm PrimJarnik(G):

Input: A weighted graph G.

Output: A minimum spanning tree T for G.

pick any vertex v of G // grow the tree starting with vertex v

T ¬ {v}
D[v] ¬ 0
E[v] ¬ Æ
for each vertex u ¹ v do D[u] ¬ +¥
let Q be a priority queue that contains vertices, using the D labels as keys
while Q ¹ Æ do // pull u into the cloud C
u¬ Q.removeMinElement()
add vertex u and edge E[u] to T
for each vertex z adjacent to u do
if z is in Q // perform the relaxation operation on edge (u, z)
if weight(u, z) < D[z] then
D[z] ¬ weight(u, z)
E[z] ¬ (u, z)
change the key of z in Q to D[z]
return tree T
runtime
O((n+m) log n)

Algorithms and Data Structures 2 // 2021W // 35 A. Ferscha

Prim-Jarnik Algorithm :: Example

110 Initialization

BRU
¥ ¥
¥ 85
80 FRA
LHR 55
¥ 65 75 0 ¥
ZRH 45
LNZ VIE
¥ 75 80
CDG
160

Algorithms and Data Structures 2 // 2021W // 36 A. Ferscha

Prim-Jarnik Algorithm :: Example

110

BRU 55
80
¥ ¥75
¥110 FRA
85
LHR 55 80
65
¥80 65 75 0 45
¥
ZRH 45
75
160¥
LNZ VIE
75 80
CDG
160

PQ: LNZ VIE FRA ZRH CDG BRU LHR

Algorithms and Data Structures 2 // 2021W // 37 A. Ferscha

Kruskal’s Algorithm

Put one edge after the other into the MST under the following conditions:
• Select the edge with the lowest weight
• An edge is only inserted, if no cycle will result from the insertion

Data structure:
• Algorithm manages a set of trees (forest)
• An edge is accepted if it connects vertices from different trees

Therefore the data structure must manage disjunctive subsets and support the following operations:
• find(u) returns the set, which u contains
• union(u,v) merges the sets, which u and v contain

Algorithms and Data Structures 2 // 2021W // 38 A. Ferscha

Kruskal’s Algorithm
Algorithm Kruskal(G):

Input: A weighted graph G.

Output: A minimum spanning tree T for G.

let P be a partition of the vertices of G, where each

vertex forms a separate set

let Q be a priority queue storing the edges of G, sorted

by their weights

T ¬Æ
while Q ¹ Æ do
(u,v) ¬ Q.removeMinElement()
if P.find(u) ¹ P.find(v) then
add edge (u,v) to T
P.union(u,v)

return T

Algorithms and Data Structures 2 // 2021W // 39 A. Ferscha

Kruskal’s Algorithm :: Example

110

BRU
85
80 FRA
LHR 55
65 75
ZRH 45
LNZ VIE
75 80
CDG
160

Edge in MST

Edge not in MST

Algorithms and Data Structures 2 // 2021W // 40 A. Ferscha

Distance and Centrality

Distance between two vertices u,v in a Graph G (V,E)

• shortest path between them
• G unweighted: d(u,v) number of hops of the shortest path
• G weighted: d(u,v) sum of the weights of the shortest path

Eccentricity
• Maximum (shortest) distance from the vertex to any other vertex in a connected graph

Diameter
• is the maximum (shortest) distance between two vertices of a connected graph

Algorithms and Data Structures 2 // 2021W // 41 A. Ferscha

Distance and Centrality

Radius
Minimum eccentricity among all vertices

Relation: diameter-radius

Algorithms and Data Structures 2 // 2021W // 42 A. Ferscha

Distance and Centrality :: Example

a b c d • h-a-g-f-e-d = 5

eccentricities a, b, c, d, e, f, g, h
h g f e • 4,4,3,5,4,3,3,5

radius:
• 3 (min eccentricities )

Algorithms and Data Structures 2 // 2021W // 43 A. Ferscha

Distance and Centrality

Given a complete graph Kn where n ≥ 2:

• the diameter and radius is 1 a b

c d

Given a complete bipartite graph Km,n where either m or n ≥ 2

• the diameter and radius is 2
c

a
d

b
e

Algorithms and Data Structures 2 // 2021W // 44 A. Ferscha

Distance and Centrality

Average distance of a vertex

• The average distance dv(av) of a vertex v is defined as the arithmetic mean
of the distance of v to all other vertices:

Average distance of a graph

• The average distance dv(av) of a graph G is defined as the arithmetic mean
of the distance among all vertices:

Algorithms and Data Structures 2 // 2021W // 45 A. Ferscha

Distance and Centrality

Average distance of a graph cont’d.

substituting in equations above:

Interpretation
• low average distance => short paths between most of the vertices
• high average distance => general difficult to reach one vertex from another

Algorithms and Data Structures 2 // 2021W // 46 A. Ferscha

Short Distance Graphs

… vertex degree
… (minimum) distance (between vertex i and vertex j)
… diameter

… average distance

… number of distance vertices

Moore Bound

d=4

Short Distance Graphs

Moore Bound

Algorithms and Data Structures 2 // 2021W // 51 A. Ferscha

Short Distance Graphs

Moore Bound

Algorithms and Data Structures 2 // 2021W // 52 A. Ferscha

Short Distance Graphs
Generalized Chordal Rings
• A graph G is a generalized chordal ring if nodes can be labelled with integers modulo M and there exists a
divisor p of M such that node i is joined to node j if node i + p is joined to node j + p.
• G is a chordal ring if all nodes (i,i+1) appear

Distance and Centrality
(Degree) Centrality
Identify the importance of specific nodes or edges in the network as a significantly more important node or an edge may
be joining two distinctive parts of the network.
X
CD (i) = aij
j2V

The equation CD = A x [1] where rows

and columns are ordered a,…,f :
a b c

f e d

Algorithms and Data Structures 2 // 2021W // 59 A. Ferscha

Distance and Centrality
k-path centrality
of a vertex v, k-CD(v) is the number of paths of length k or less emanating from vertex v.
When k=1, k-path centrality is equal to degree centrality.

Edge disjoint paths (with common source/sink) do not have any common edges
Vertex disjoint paths do not have any common vertices

p1 p2
p1 c d p1 c d
p1
b p2 b p2 p1
p2 a e p2 a e
p2

Vertex disjoint paths Edge disjoint paths

(messages from source to sink
never have to use the same link)
Algorithms and Data Structures 2 // 2021W // 60 A. Ferscha
Distance and Centrality

Edge disjoint k-path centrality

• of a vertex v is the number of edge disjoint paths of length k or less that start or end at vertex v. The Ford-
Fulkerson Theorem states that the number of edge disjoint paths between two nodes u and v of a graph is equal to
the minimum number of vertices that must be removed to disconnect u and v.

Vertex disjoint k-path centrality

• of a node v is the number of vertex disjoint paths of length k or less that start or end at vertex v. Menger showed
that the number of vertex disjoint paths between two nodes u and v equals the number of nodes that must be
removed to disconnect u and v.

à Major drawback with degree centrality: only considers local information.

(a node having few important neighbors may be more influential globally than a node which has many less important
neighbors.)

Algorithms and Data Structures 2 // 2021W // 61 A. Ferscha

Distance and Centrality

k-hop degree (k-hop-CD)

• number of neighbors a node has in its k-hop neighborhood.
For k=1, k-hop-CD equals CD.
k-rank of a node as its position in the descended sorted degree list of neighbors in its k-hop neighborhood.

3 1 2
a: a,b,c,d,g à d,a,c,{b,g} à a=2
c d e b: a,b,c à a,b,c àb=2
3
c: a,b,c,d à d,a,c,b à c=3
b d: a,c,d,e,f,g à d,a,c,g,e,f à d=1
e: d,e à d,e à e=2
a g f f: d,f àd,f à f=2
g: a,d,g à d,a,g à g=3
2 3 2
1-hop-degree

One-rank centrality example

Algorithms and Data Structures 2 // 2021W // 62 A. Ferscha

Distance and Centrality
Closeness Centrality
• based on global rather than local knowledge
• defined as the reciprocal of the total distance from this vertex to all other vertices in a graph G(V,E)

a b c a b c a b c

f e d f e d f e d

a b c a b c a b c

f e d f e d f e d

BFS trees for vertex distance calculation

Algorithms and Data Structures 2 // 2021W // 63 A. Ferscha

Distance and Centrality
Closeness Centrality
a b c a b c a b c

f e d f e d f e d

a b c a b c a b c

f e d f e d f e d

Highest closeness centrality

Algorithms and Data Structures 2 // 2021W // 64 A. Ferscha

Distance and Centrality

Stress Centrality
total number of all pairs shortest paths that pass through a vertex v

It is an estimation of the stress that a vertex in a network bears, assuming all communication
will be carried along the shortest path.

Algorithms and Data Structures 2 // 2021W // 65 A. Ferscha

Distance and Centrality

Betweenness Centrality
If the paths for shortest paths between nodes of a network pass through some vertices
more often than others, then these vertices are significantly more important than others for communication purposes.

Simple procedure to find CB is to calculate all shortest paths in the graph G and count the number of paths that pass
through v by excluding the paths that start or end at v.

Perform modified BFS on each node à

Algorithms and Data Structures 2 // 2021W // 66 A. Ferscha

Distance and Centrality

Betweenness Centrality :: Newman’s Algorithm

forall v in V
bv ß 1
find BFS paths from v to all other nodes
end

forall s in V
starting from the farthest nodes, move from u towards s along paths
using vertex v.
bv ß bv+bu

If v has more than one predecessor, then bv is devided equally

between them.
end

Algorithms and Data Structures 2 // 2021W // 67 A. Ferscha

Distance and Centrality
Eigenvalue Centrality
• a vertex in a network may be considered important if it has important neighbors.
• Given a graph G(V,E) with an adjacency matrix A, the score of a vertex i can be defined as
proportional to the sum of all its neighbors’ scores:

• The score of a node can simply be its degree. We are adding the degrees of the neighbors of a
vertex i to find its new score.

which is the eigenvalue equation

• To find the eigenvalue centrality of a graph G, we need to find the eigenvalues of the adjacency matrix A. We
select the largest eigenvalue and its associated eigenvector. This eigenvector contains the eigenvalue
centralities of all vertices.

Algorithms and Data Structures 2 // 2021W // 68 A. Ferscha

Appendix

Algorithms and Data Structures 2 // 2021W // 69 A. Ferscha

Generic DFS in Java (1)

/** Abstract super class for all Depth-First-Search algorithms. */

public abstract class DFS<V, E> {

/** The graph we are operating on. */

protected Graph<V, E> g;

/** List of all visited vertices and edges. */

protected List<Object> visited;

public DFS(Graph<V, E> g) {

this.g = g;
}

/** Perform a depth-first-search traversal on the graph.

* @param start the vertex to begin with
* @return result of the traversal */
protected void execute(Vertex<V> start) {
if (start != null) {
visited = new ArrayList<Object>();
dfsTraversal(start);
}
} à at.jku.pervasive.ad2.vo11.DFS.java

Algorithms and Data Structures 2 // 2021W // 70 A. Ferscha

Generic DFS in Java (2)

/** Recursive template method for a generic DFS traversal.

* @param v start vertex of the traversal */
protected void dfsTraversal(Vertex<V> v) {
startVisit(v);
visited.add(v);
for (Iterator<Edge<E>> itEdges = g.incidentEdges(v);
itEdges.hasNext();) {
Edge<E> nextEdge = itEdges.next();
if (!visited.contains(nextEdge)) {
visited.add(nextEdge);
Vertex<V> w = g.opposite(v, nextEdge);
if (!visited.contains(w)) {
traverseDiscovery(nextEdge, v);
if (!isDone()) dfsTraversal(w);
} else {
traverseBack(nextEdge, v);
}
}
}
finishVisit(v);
}
à at.jku.pervasive.ad2.vo11.DFS.java

Algorithms and Data Structures 2 // 2021W // 71 A. Ferscha

Generic DFS in Java (3)

/** Called when we first visit v. Empty method stub.

* @param v vertex */
protected void startVisit(Vertex<V> v) {}

/** Called when we finish with v. Empty method stub.

* @param v vertex */
protected void finishVisit(Vertex<V> v) {}

/** Found a discovery edge. Empty method stub.

* @param e edge
* @param from vertex */
protected void traverseDiscovery(Edge<E> e, Vertex<V> from) {}

/** Visiting a back edge. Empty method stub.

* @param e edge
* @param from vertex */
protected void traverseBack(Edge<E> e, Vertex<V> from) {}

/** Is DFS done early? Empty method stub.

* @return true if traversal should end now */
protected boolean isDone() {
return false; à at.jku.pervasive.ad2.vo11.DFS.java
}
}
Algorithms and Data Structures 2 // 2021W // 72 A. Ferscha
Connectivity Test

/**
* Determine wether the graph is connected.
*/
public class ConnectivityTesterDFS<V, E> extends DFS<V, E> {
protected int reached;

public ConnectivityTesterDFS(Graph<V, E> g) {

super(g);
}
/**
* Check if graph is connected.
* @return true if all vertices in the graph are connected
*/
public boolean isConnected() {
reached = 0;
execute(g.vertices().next()); // start with any vertex of graph
return reached == g.numberOfVertices();
}
@Override
protected void startVisit(Vertex<V> v) {
reached++;
} à at.jku.pervasive.ad2.vo11.ConnectivityTesterDFS.java
}

Algorithms and Data Structures 2 // 2021W // 73 A. Ferscha

Path between two vertices?

/** Find a path between the start vertex and a given target vertex. */
public class FindPathDFS<V, E> extends DFS<V, E> {
protected LinkedList<Vertex<V>> path;
protected boolean done;
protected Vertex<V> target;

public FindPathDFS(Graph<V, E> g) { super(g); }

/** @return path or empty List if no such path exists */
public List<Vertex<V>> findPath(Vertex<V> start, Vertex<V> end) {
path = new LinkedList<Vertex<V>>();
done = false;
target = end;
execute(start);
return path;
}
protected void startVisit(Vertex<V> v) {
path.addLast(v);
if (v == target) done = true;
}
protected void finishVisit(Vertex<V> v) {
if (!done) path.removeLast();
}
protected boolean isDone() {
return done;
à at.jku.pervasive.ad2.vo11.FindPathDFS.java
}
}

Algorithms and Data Structures 2 // 2021W // 74 A. Ferscha

Find cycle (1)

/** Find cycles i a graph. */

public class FindCycleDFS<V, E> extends DFS<V, E> {
protected LinkedList<Edge<E>> cycle; // sequence of edges of the cycle
protected boolean done;
protected Vertex<V> cycleStart;

public FindCycleDFS(Graph<V, E> g) { super(g); }

/** Find a cycle in the connected component of start vertex.
* @param start vertex
* @return sequence of edges, or empty list if there is no cycle */
public List<Edge<E>> findCycle(Vertex<V> start) {
cycle = new LinkedList<Edge<E>>();
done = false;
cycleStart = null;
execute(start);
if (!cycle.isEmpty()) {
// if cylce does not start with start vertice, remove edges from
// cyle until cycleStart is reached
while (start != cycleStart)
start = g.opposite(start, cycle.removeFirst());
}
return cycle; à at.jku.pervasive.ad2.vo11.FindCycleDFS.java
}

Algorithms and Data Structures 2 // 2021W // 75 A. Ferscha

Find cycle (2)

@Override
protected void finishVisit(Vertex<V> v) {
if (!cycle.isEmpty() && !done) cycle.removeLast();
}
@Override
protected void traverseDiscovery(Edge<E> e, Vertex<V> from) {
if (!done) cycle.addLast(e);
}
@Override
protected void traverseBack(Edge<E> e, Vertex<V> from) {
if (!done) {
cycle.addLast(e); // back edge e creates a cycle
cycleStart = g.opposite(from, e);
done = true;
}
}
@Override
protected boolean isDone() {
return done;
}
}
à at.jku.pervasive.ad2.vo11.FindCycleDFS.java

Algorithms and Data Structures 2 // 2021W // 76 A. Ferscha

Graphs
Part: Structure

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at
Graphs
Part: Flows

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at
Flow Analysis

Algorithms and Data Structures 2 // 2021W // 2 A. Ferscha

Flows in Networks

Consider a directed, weighted graph, also called network N, where:

2 s 3
The weights are defined as capacity of the edge
1
In the graph there is one distinct vertex, which has no incoming edges, the source s 2 2
In the graph there is one distinct vertex which has no outgoing edges, the sink t 1 2
2 4
2
A flow(u,v) " edge(u,v) in this graph is a function on the edges with 2 1
t
• 0 £ flow(u,v) £ capacity(u,v) (Capacity constraint)

• S uÎin(v) flow(u,v) = S wÎout(v) flow(v,w) (Flow conservation)

• |flow| = S wÎout(s) flow(s,w) = S uÎin(t) flow(u,t) (Value of the flow)

Algorithms and Data Structures 2 // 2021W // 3 A. Ferscha

Maximum Flow

flow
Find the flow f in a given network N with maximum value
capacity
Example: maxFlow = 5 s
2/2 2/3

1/1
1/2 1/2

1/1 1/2
1/2 0/4

2/2

2/2 1/1
t

Algorithms and Data Structures 2 // 2021W // 4 A. Ferscha

Maximum flow

Maximum value of the flow is not only determined by s and t,

but also by each cut that separates s from t : cut

s
2/2 2/3
S
1/1
1/2 1/2

1/1 1/2
1/2 0/4
c(X)=1+2+2+4+2=11
2/2

2/2 1/1
t T

Algorithms and Data Structures 2 // 2021W // 5 A. Ferscha

Cuts in Networks

A cut X = (Vs, Vt) is a cut through the network that separates the set of vertices into two partitions.

The capacity of a cut is the sum of the capacities of the “cut” edges

𝑐 𝑋 = $ 𝑐𝑎𝑝𝑎𝑐𝑖𝑡𝑦(𝑣, 𝑤)
!∈#! ,%∈#"
We have:
Value of the maximum flow = capacity of the minimum cut

Determination of the minimum cut:

Determine Vs as the set of vertices that are reached on the augmenting path in the network, and Vt as the set of all other
vertices (variant of the Ford-Fulkerson algorithm)

Algorithms and Data Structures 2 // 2021W // 6 A. Ferscha

min cut = 4

Maximum Flow
2/2 s 1/2

Maximum Flow Theorem (Ford/Fulkerson 1956)

1/1
max flow ó min cut
1/2 t 2/2

Each path s->t contains at least flow = 3

one saturated edge

Algorithms and Data Structures 2 // 2021W // 7 A. Ferscha

min cut = 4

Maximum Flow
2/2 s 1/2

Maximum Flow Theorem (Ford/Fulkerson 1956)

1/1
max flow ó min cut
1/2 t 2/2

Each path s->t contains at least flow = 3

one saturated edge

Definition: augmenting path

• Forward-edge:
s 1
flow(u,v) < capacity(u,v)
à Flow can be increased
1/2

• Backward-edge: 1
flow(u,v) > 0
à Flow can be decreased 1/1

Algorithms and Data Structures 2 // 2021W // 8 A. Ferscha

min cut = 4

Maximum Flow
2/2 s 1/2

Maximum Flow Theorem (Ford/Fulkerson 1956)

1/1
The flow in a network is at maximum, if and only
if the network has no augmenting path. 1/2 t 2/2

Each path s->t contains at least flow = 3

one saturated edge

Definition: augmenting path

• Forward-edge:
flow(u,v) < capacity(u,v) s 1
à Flow can be increased s 1
1/2
1 1/2
• Backward-edge: 1 1 1/1
flow(u,v) > 0
à Flow can be decreased 1/1 1/2 t

Algorithms and Data Structures 2 // 2021W // 9 A. Ferscha

Ford-Fulkerson Algorithm

initialize network with null flow;

Method FindFlow
if augmenting paths exist then
find augmenting path;
increase flow;
recursive call to FindFlow;

A path s à t
(regardless of the direction of the arrow)
on which you can increase the flow is
called augmenting path

Algorithms and Data Structures 2 // 2021W // 10 A. Ferscha

Ford-Fulkerson :: Example
Capacities Initialize flow with zero
2 s 2
0 0
0
2 1 2
0 0
t
|f|=0
2 s 2 Increase flow by 1 unit
0
1
2 1
0 Increase flow by 1 unit
1
0
2 1 2 Increase flow by 1 unit
10 2
0
1
t
|f|=3
1
2
2 s 2
Send further unit over
2 0
1 1
2 augmented path
2 1 2
1
2 2
t
|f|=4
3
Algorithms and Data Structures 2 // 2021W // 11 A. Ferscha
Determination of the Augmenting Path
„Residual graph“: describes all
possibilities to increase the flow
Consider the Residual Network Nf = (V, Ef, cf, s, t) to a network N = (V, E, c, s, t)
N Nf
cf(u,v) = c(u,v) - f(u,v)
v v
f(u,v) / c(u,v)

u u cf(v,u) = f(u,v)
Edges with capacity 0 are removed
An augmented path in N corresponds to a directed path from s to t in Nf and can therefore be determined using DFS in Nf

2 s
2/2 s

Algorithms and Data Structures 2 // 2021W // 12 A. Ferscha

Determination of the Augmenting Path
„Residual graph“: describes all
possibilities to increase the flow
Consider the Residual Network Nf = (V, Ef, cf, s, t) to a network N = (V, E, c, s, t)
N Nf
cf(u,v) = c(u,v) - f(u,v)
v v
f(u,v) / c(u,v)

u u cf(v,u) = f(u,v)
Edges with capacity 0 are removed

An augmented path in N corresponds to a directed path from s to t in Nf and can therefore be determined using DFS in Nf
2 s 1
2/2 s 1/2
1
1/1 1
1
1/2 2/2 1 2
t t

Algorithms and Data Structures 2 // 2021W // 13 A. Ferscha

AUGMENTED PATH :: EXAMPLE STEP 1/7
capacity Current flow Augmenting path Residual graph

flow 0/3 0/3 3

0/5 0/3 0/5 0/3 5 0 3
0 0
S 0/3 T 0/3 3
S T S T
4 0 5
0/4 0/5 0/4 0/5 0 3 0
0/3 0/3
0

increase flow by 1
+1 Updated residual graph
+1 +1 2
1/3
1/5 1/3 4 1 2
1 1
0/3 3
S T S T
4 0
5
0/4 0/5 0 3 0
0/3
0
Algorithms and Data Structures 2 // 2021W // 14 A. Ferscha
AUGMENTED PATH :: EXAMPLE STEP 2/7
capacity Current flow Augmenting path Residual graph

flow 1/3 1/3 2

1/5 1/3 1/5 1/3 4 1 2
1 1
S 0/3 T 0/3 3
S T S T
4 0
5
0/4 0/5 0/4 0/5 0 3 0
0/3 0/3
0

increase flow by 1
Updated residual graph

1/3 2
1/5 1/3 4 1 2
1 1
0/3 3
S T S T
3 0
+1 +1 4
1/4 1/5 1 2 1
1/3 1
+1
Algorithms and Data Structures 2 // 2021W // 15 A. Ferscha
AUGMENTED PATH :: EXAMPLE STEP 3/7
capacity Current flow Augmenting path Residual graph

flow 1/3 1/3 2

1/5 1/3 1/5 1/3 4 1 2
1 1
S 0/3 T 0/3 3
S T S T
3 0
4
1/4 1/5 1/4 1/5 1 2 1
1/3 1/3
1

increase flow by 1
+1 Updated residual graph
+1 1
2/3
2/5 1/3 3 2 2
2 1
1/3 +1 2
S T S T
3 1
3
1/4 2/5 1 1 2
2/3 2
+1 +1
Algorithms and Data Structures 2 // 2021W // 16 A. Ferscha
AUGMENTED PATH :: EXAMPLE STEP 4/7
capacity Current flow Augmenting path Residual graph

flow 2/3 2/3 1

2/5 1/3 2/5 1/3 3 2 2
2 1
S 1/3 T 1/3 2
S T S T
3 1
3
1/4 2/5 1/4 2/5 1 1 2
2/3 2/3
2

increase flow by 1
Updated residual graph
+1 1
2/3
2/5 2/3 3 2 1
-1 2
3 2
S 0/3 T S T
2 0
+1 3
2/4 2/5 2 1 2
2/3
2
Algorithms and Data Structures 2 // 2021W // 17 A. Ferscha
AUGMENTED PATH :: EXAMPLE STEP 5/7
capacity Current flow Augmenting path Residual graph

flow 2/3 2/3 1

2/5 2/3 2/5 2/3 3 2 1
2 2
S 0/3 T 0/3 3
S T S T
2 0
3
2/4 2/5 2/4 2/5 2 1 2
2/3 2/3
2

increase flow by 1
+1 Updated residual graph
+1 0
3/3
3/5 2/3 2 3 1
3 2
1/3 2
S +1 T S T
2 1
2
+1 2
2/4 3/5 0 3
3/3 3
+1
Algorithms and Data Structures 2 // 2021W // 18 A. Ferscha
AUGMENTED PATH :: EXAMPLE STEP 6/7
capacity Current flow Augmenting path Residual graph

flow 3/3 3/3 0

3/5 2/3 3/5 2/3 2 3 1
3 2
S 1/3 T 1/3 2
S T S T
2 1
2
2/4 3/5 2/4 3/5 2 0 3
3/3 3/3
3

increase flow by 1
Updated residual graph
+1
3/3 0
3/5 3/3 2 3 0
-1 3 3
0/3 3
S T S T
1 0
2
+1 3/4 3/5 3 0 3
3/3
3
Algorithms and Data Structures 2 // 2021W // 19 A. Ferscha
AUGMENTED PATH :: EXAMPLE STEP 7/7

No augmenting path from source S to sink T exists where flow could be increased
(when flow = capacity).

Residual graph
flow = capacity
3/3 0
3/5 3/3 2 3 0
3 3
0/3 3
S T S T
1 0
2
3/4 3/5 3 0 3
3/3
3

Max flow is: |f| = 6

Algorithms and Data Structures 2 // 2021W // 20 A. Ferscha

AUGMENTED PATH :: EXAMPLE STEP 7/7

No augmenting path from source S to sink T exists where flow could be increased
(when flow = capacity).

Residual graph
flow = capacity
3/3
3/5 3/3 2 3
3 3
0/3 3
S T S T
1
2
3/4 3/5 3 3
3/3
3

Max flow is: |f| = 6

Algorithms and Data Structures 2 // 2021W // 21 A. Ferscha

Maximum Flow Algorithm
Part I: Setup Algorithm: MaxFlow(N)
Input: network N
Output: network Nf with maximum flow
Start with null flow:
f(u,v) ¬ 0 " (u,v) Î E;
Initialize residual network:
Nf ¬ N;

Part II: Loop repeat

search for directed path p in N f from s to t
if (path p found)
Df ¬ min {cf (u,v), f(u,v) Î p};
for (each (u,v) Î p) do
Runtime if (forward (u,v))
O( F × (n+m) ) f(u,v) ¬ f(u,v) + Df ;
if (backward (u,v))
f(u,v) ¬ f(u,v) - Df ;
update N f ;
until (no augmenting path exists);

Algorithms and Data Structures 2 // 2021W // 22 A. Ferscha

Improvement of Maximum Flow

Theorem [Edmonds & Karp 1972]

By using BFS the maximum flow can be determine in a runtime of

O( (n+m) × n × m ) = O(n5)

Advantage:
Runtime is independent from the value of the maximum flow.

Algorithms and Data Structures 2 // 2021W // 23 A. Ferscha

Graphs
Part: Flows

Algorithms and Data Structures 2, 340300 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2021W Altenberger Straße 69
4040 Linz, Austria
Univ.-Prof. Dr. Alois Ferscha, [email protected] jku.at
Social Graphs

Univ.-Prof. Dr. Alois Ferscha

Institute of Pervasive Computing
Johannes Kepler University Linz
[email protected]

Algorithms and Data Structures 2 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2020W Altenberger Straße 69
4040 Linz, Austria
jku.at
Networks

Erdös Number: “Collaborative Distance” among mathematical authors

Erdős Pál
(1913-1996)

Algorithms and Data Structures 2 // 2020W // 2 A. Ferscha

„Small-World“ Networks

Small-world network
• most nodes are not neighbors of one another
• but most nodes can be reached from every other by
a small number of hops or steps.

distance L between two randomly chosen nodes grows

proportionally to the logarithm of the number of nodes N
in the network (=society)

human society is a small-world-type network

Erdős Pál characterized by short path-lengths.
(1913-1996) Stanley Milgram

Algorithms and Data Structures 2 // 2020W // 3 A. Ferscha

Connect with probability p

p=1/6 N=10
<k> ~ 1.5

Algorithms and Data Structures 2 // 2020W // 7 A. Ferscha

Random Networks (1959)

Degree distribution
(in very large networks)

Erdős Pál Alfréd Rényi Connect with probability p

(1913-1996) (1921-1970)
0 < p < 1/N disconnected trees O(log n)
p = 1/N large connected component
p > 1/N connected giant component

The emergence of a network is NOT a GRADUAL process !

Algorithms and Data Structures 2 // 2020W // 8 A. Ferscha
World Wide Web
Nodes: WWW documents
Links: URL links

Expected
Over 1 Trillion documents

Collect all URL‘s found in

Found
a document and follow
them recursively

Albert, Jeong & Barabási, Nature, 401 130 (1999).

Algorithms and Data Structures 2 // 2020W // 10 A. Ferscha

Scale-Free Networks (1999)

Scale free networks

(large hubs network)

Albert-László
Barabási
(1967-)

Albert, Jeong & Barabási, Nature, 401 130 (1999).

Algorithms and Data Structures 2 // 2020W // 11 A. Ferscha

Erdős-Rényi (ER) Networks ::
Barabasi-Albert (BA) Networks
Degree distribution

Gaussian
Random
Network

Power-Law
Scale-free
Network

Albert, Jeong & Barabási, Nature, 401 130 (1999).

Algorithms and Data Structures 2 // 2020W // 13 A. Ferscha

Networks Held together by Hubs

Power Law Distribution

Number of nodes with k links

Very many nodes
with only a few links

A few hubs with

Large number of links

Number of links ( k )
Costa, Evsukoff et al (2010) Complex Networks

Algorithms and Data Structures 2 // 2020W // 14 A. Ferscha

Barabasi Networks :: Examples

1985 2003
The Network Behind an Organisation :: Shareholder-Network of the Japanese Automotive Industry
https://www.quantamagazine.org/20130904-evolution-as-opportunist/
Krischke & Röpcke (2014) Graphen und Netzwerktheorie

Algorithms and Data Structures 2 // 2020W // 15 A. Ferscha

Barabasi Networks :: The Matter of Life

Metabolic Network Social Network

https://www.quantamagazine.org/20130904-evolution-as-opportunist/
Krischke & Röpcke (2014) Graphen und Netzwerktheorie

Algorithms and Data Structures 2 // 2020W // 17 A. Ferscha

Barabasi Networks :: Examples

Airline Network Protein Interactions

https://www.quantamagazine.org/20130904-evolution-as-opportunist/
Krischke & Röpcke (2014) Graphen und Netzwerktheorie

Algorithms and Data Structures 2 // 2020W // 18 A. Ferscha

SF Networks :: Growth and Preferential Attachment

(1) Networks continuously expand by the addition of Growth:

new nodes Add a new node with m links
www: addition of new documents
Preferential Attachment:
The probability that a node connects to a node
(2) New nodes prefer to link to highly connected with k links is proportional to k.
nodes.
www: linking to well known sites

Barabási & Albert (1999) Science 286, 509.

Algorithms and Data Structures 2 // 2020W // 19 A. Ferscha

Robustness of Scale-free Networks

Scale-free Networks are

Failures
(i) ultra-resilient against
failures

Attacks (ii) but suffer from direct

attacks on hubs

Krischke & Röpcke (2014) Graphen und Netzwerktheorie

Algorithms and Data Structures 2 // 2020W // 20 A. Ferscha

Contagion :: Erdős-Rényi Networks
Biology: Economics:
If a virus is not too infectious If a product/idea is not too ‚sticky‘
> it dies out. > it does not succeed.

spreading rate
of a virus

Pastor-Satorras & Vespignani, Physical Review Letters (2001) DOI: 10.1103/PhysRevLett.86.3200

density of infected individuals
Algorithms and Data Structures 2 // 2020W // 21 A. Ferscha
Contagion :: Erdős-Rényi Networks
Biology: Economics:
If a virus is not too infectious If a product/idea is not too ‚sticky‘
> it dies out. > it does not succeed.

spreading rate
of a virus

In the presence of hubs,

weakly infectious viruses
persist in the population.
Pastor-Satorras & Vespignani, Physical Review Letters (2001) DOI: 10.1103/PhysRevLett.86.3200
density of infected individuals
Algorithms and Data Structures 2 // 2020W // 22 A. Ferscha
Spreading Processes :: Airport‘s Role in a Pandemic

Algorithms & Datastructures 2 // 2019W // 23 A. Ferscha A. Ferscha

How Information Spreads// Media+Lifestyle Summit 2015 // Zürs, Austria // March 15, 2015 // 23
http://newsoffice.mit.edu/2012/spread-of-disease-in-airports-0723
Algorithms & Datastructures 2 // 2019W // 24 A. Ferscha
Social Graphs

Univ.-Prof. Dr. Alois Ferscha

Institute of Pervasive Computing
Johannes Kepler University Linz
[email protected]

Algorithms and Data Structures 2 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2020W Altenberger Straße 69
4040 Linz, Austria
jku.at
Community Analysis

Univ.-Prof. Dr. Alois Ferscha

Institute of Pervasive Computing
Johannes Kepler University Linz
[email protected]

Algorithms and Data Structures 2 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2020W Altenberger Straße 69
4040 Linz, Austria
jku.at
Community Analysis

Small-World Effect
• Six Degrees of Separation
• A famous experiment conducted by Travers and Milgram (1969)
- Subjects were asked to send a chain letter to his acquaintance in order to reach a target person
- The average path length is around 5.5

Verified on a planetary-scale IM network (Microsoft Messenger) of

240 million people (Leskovec and Horvitz 2008) and 30 billion conversations
• 180 million nodes
• 1.3 billion undirected edges
• The average path length is 6.6

J. Leskovec and E. Horvitz. Planetary-scale views on a large instant-messaging network. In Proceedings of the 17th international conference on World Wide Web, WWW ’08, pages 915–924, New York, NY, USA, 2008. ACM

Algorithms and Data Structures 2 // 2020W // 2 A. Ferscha

Community Analysis

Verified on a planetary-scale IM network (Microsoft Messenger) of

240 million people (Leskovec and Horvitz 2008) and 30 billion conversations
• 180 million nodes
• 1.3 billion undirected edges
• The average path length is 6.6

Algorithms and Data Structures 2 // 2020W // 3 A. Ferscha

Community

Measures used to calibrate the small world effect:

Diameter
Measures used to calibrate the small world effect
• Diameter: the (maximum) longest shortest path in a network
• Average shortest path length

• The shortest path between two nodes is (also) called geodesic.

• The number of hops in the geodesic is the geodesic distance.

• The geodesic distance between node 1 and node 9 is 4.

• The diameter of the network is 5, corresponding to the
geodesic distance between nodes 2 and 9.

Algorithms and Data Structures 2 // 2020W // 4 A. Ferscha

Community
A community are people in a group interacting with each other more frequently than those outside the group

Link prediction

Network-Based Classification
Algorithms and Data Structures 2 // 2020W // 5 A. Ferscha
Community Analysis

Density of connections
• Friends of a friend are likely to be friends as well
- density of connections among one’s friends

d … number of neighbours
k … connections among neighboured friends

Example:
d6=4, N6= {4, 5, 7,8}
k6=4 as e(4,5), e(5,7), e(5,8), e(7,8)
C6 = 4/(4*3/2) = 2/3

Average clustering coefficient

C = (C1 + C2 + … + Cn)/n

C = 0.61 for the given network

Algorithms and Data Structures 2 // 2020W // 6 A. Ferscha

Community Analysis

Degree of Centrality
The importance of a node is determined by the number of nodes adjacent to it
• The larger the degree, the more import the node is
• Only a small number of nodes have high degrees in many real-life networks

Degree Centrality

Normalized Degree Centrality

• For node 1, degree centrality is 3;

• Normalized degree centrality is 3/(9-1)=3/8

Algorithms and Data Structures 2 // 2020W // 7 A. Ferscha

Community Analysis

Community Analysis

Eigenvalue (or Eigenvector) Centrality

• One’s importance is determined by his friends’
• If one has many important friends, he should be important as well.

• The centrality corresponds to the top eigenvector of the adjacency matrix A.

• A variant of this eigenvector centrality is the PageRank score.

Algorithms and Data Structures 2 // 2020W // 12 A. Ferscha

Community Analysis :: Importance (Closeness)

(A) Example of a hypothetical social network illustrating the individual level metrics. Letters label individuals in the network.

(B) Network is restructured in a hierarchy such that the node with the highest relevant centrality measure is on top
following the arrow). For example, node C has the highest Eigenvector centrality, but node E had the highest
betweenness centrality.

Algorithms and Data Structures 2 // 2020W // 13 A. Ferscha

Community Analysis :: Ties and Influence
Weak and Strong Ties
• In practice, connections are not of the same strength

• Interpersonal social networks are composed of strong

ties (close friends) and weak ties (acquaintances).

• Strong ties and weak ties play different roles for

community formation and information diffusion
• strong ties are transitive with high probability
• weak ties are not transitive (or with low probability)
Weak Ties
Strong Ties
• The Strength of Weak Ties (Granovetter, 1973)
• Occasional encounters with distant acquaintances can
provide important information about new opportunities
for job search
M. Granovetter. The Strength of Weak Ties. The American Journal of Sociology, 78(6):1360–1380, 1973.

Algorithms and Data Structures 2 // 2020W // 14 A. Ferscha

Community Analysis

Connections in Social Media

Social Media allows users to connect to each other more easily than ever
• One user might have thousands of friends online
• Who are the most important ones among your Facebook friends?

Imperative to estimate the strengths of ties for advanced analysis

• Analyze network topology
• Learn from User Profiles and Attributes
• Learn from User Activities

Algorithms and Data Structures 2 // 2020W // 15 A. Ferscha

Community Analysis :: Network Topology

Learning from Network Topology

• Bridges connecting two different communities are weak ties
• An edge is a bridge if its removal results
in disconnection of its terminal nodes

e(2,5) is a bridge e(2,5) is NOT a bridge

Algorithms and Data Structures 2 // 2020W // 16 A. Ferscha

Community Analysis :: Network Topology

“shortcut” Bridge
• Bridges are rare in real-life networks
• Alternatively, one can relax the definition by checking if the distance
between two terminal nodes increases if the edge is removed
• The larger the distance, the weaker the tie is

• d(2,5) = 4 if e(2,5) is removed

• d(5,6) = 2 if e(5,6) is removed
• e(5,6) is a stronger tie than e(2,5)

Algorithms and Data Structures 2 // 2020W // 17 A. Ferscha

Community Analysis :: Network Topology

Neighbourhood Overlap
Tie strength can be measured based on neighborhood overlap; the larger the overlap, the stronger the tie is.

-2 in the denominator is to exclude vi and vj

Algorithms and Data Structures 2 // 2020W // 18 A. Ferscha

Community Analysis :: User Activities

Learning from User Activities

• One might learn how one influences his friends
if the user activity log is accessible
• Depending on the adopted influence model
- Independent cascading model
- Linear threshold model

Maximizing the likelihood of user activity given an influence model

Algorithms and Data Structures 2 // 2020W // 20 A. Ferscha

Community Analysis :: Influence Modeling

Influence Modeling
Influence modeling is one of the fundamental questions in order to understand the information diffusion,
spread of new ideas, and word-of-mouth (viral) marketing

Well known methods:

• Linear threshold model (LTM)

• Independent cascade model (ICM)

Common properties of influence modeling methods

• A social network is represented by a directed graph, with each actor being one node;

• Each node is started as active or inactive;

• A node, once activated, will activate his neighboring nodes;

• Once a node is activated, this node cannot be deactivated.

Algorithms and Data Structures 2 // 2020W // 21 A. Ferscha

Community Analysis :: Influence Modeling

Linear Threshold Model (LTM)

An actor would take an action if the number of his friends who have taken the action exceeds (reaches)
a certain threshold

• Each node v chooses a threshold ϴv randomly from a uniform distribution in an interval between 0 and 1.
• In each discrete step, all nodes that were active in the previous step remain active
• The nodes satisfying the following condition will be activated

Algorithms and Data Structures 2 // 2020W // 22 A. Ferscha

Community Analysis :: Influence Modeling

Linear Threshold Model – Diffusion Process (Threshold = 50%)

Algorithms and Data Structures 2 // 2020W // 23 A. Ferscha

Community Analysis :: Influence Modeling

Independent Cascade Model (ICM)

The independent cascade model focuses on the sender’s rather than the receiver’s view

• A node w, once activated at step t , has one chance to activate each of its neighbors randomly
- For a neighboring node (say, v), the activation succeeds with probability pw,v (e.g. p = 0.5)

• If the activation succeeds, then v will become active at step t + 1

• In the subsequent rounds, w will not attempt to activate v anymore.

• The diffusion process, starts with an initial activated set of nodes,

then continues until no further activation is possible

Algorithms and Data Structures 2 // 2020W // 24 A. Ferscha

Community Analysis :: Influence Modeling

Independent Cascade Model (ICM)

Algorithms and Data Structures 2 // 2020W // 25 A. Ferscha

Community Analysis :: Influence Modeling

Influence Maximization
• Given a network and a parameter k, which k nodes should be selected to be in the activation set B in order to
maximize the influence in terms of active nodes at the end?

• Let σ(B) denote the expected number of nodes that can be influenced by B, the optimization problem can be
formulated as follows:

Algorithms and Data Structures 2 // 2020W // 26 A. Ferscha

Community Analysis :: Influence Modeling

Influence Maximization – A greedy approach

• Maximizing the influence is an NP-hard problem but it is proven that the greedy approach gives a solution that is
63 % of the optimal.
• A greedy approach
• Start with B = Ø
• Evaluate σ(v) for each node, and pick the node with maximum σ as the first node v1 to form B = {v1}
• Select a node which will increase σ(B) most if the node is included in B

• Essentially, we greedily find a node v ∈ V \B such that

Algorithms and Data Structures 2 // 2020W // 27 A. Ferscha

Community Analysis ::
Distinguishing Between Influence & Correlation
Correlation
It has been widely observed that user attributes and behaviors tend to correlate with their social networks

§ Suppose we have a binary attribute with each node (say, whether or not being smoker)

§ If the attribute is correlated with the network, we expect actors sharing the same attribute value
to be positively correlated with social connections

§ That is, smokers are more likely to interact with other smokers,
and non-smokers with non-smokers

Algorithms and Data Structures 2 // 2020W // 28 A. Ferscha

Community Analysis ::
Distinguishing Between Influence & Correlation
Test For Correlation
If the fraction of edges linking nodes with different attribute values are significantly less than the expected
probability, then there is evidence of correlation

Example; if connections are independent of the smoking behavior:

p fraction are smokers (1-p non-smoker)

• one edge is expected to connect two smokers with probability p × p,

• two non-smokers with probability: (1 − p) × (1 − p)
• A smoker and a non-smoker: 2 p (1-p)

Algorithms and Data Structures 2 // 2020W // 29 A. Ferscha

Community Analysis ::
Distinguishing Between Influence & Correlation
Test For Correlation

• Red nodes denote non-smokers, and green ones are smokers. If there is no correlation, then the probability
of one edge connecting a smoker and a non-smoker is 2 × 4/9 × 5/9 = 49%.

• In this example the fraction is 2/14 = 14% < 49%, so this network demonstrates some degree of correlation
with respect to the smoking behavior.

• A more formal way is to conduct a χ2 test for independence of social connections and attributes [1]

[1] T. La Fond and J. Neville. Randomization tests for distinguishing social influence and homophily effects. In Proceedings of the 19th international conference on World wide web, WWW ’10, pages 601–610, New York, NY, USA, 2010.
ACM.

Algorithms and Data Structures 2 // 2020W // 30 A. Ferscha

Community Analysis :: Clustering

Taxonomy of Community Criteria

Criteria vary depending on the tasks

Roughly, clustering methods can be divided into 4 categories (not exclusive):

• Node-Centric Community
• Each node in a group satisfies certain properties

• Group-Centric Community
• Consider the connections within a group as a whole.
The group has to satisfy certain properties without zooming into node-level

• Network-Centric Community
• Partition the whole network into several disjoint sets

• Hierarchy-Centric Community
• Construct a hierarchical structure of communities

Algorithms and Data Structures 2 // 2020W // 31 A. Ferscha

Community Analysis :: Clustering

Taxonomy of Community Criteria

Criteria vary depending on the tasks

Roughly, clustering methods can be divided into 4 categories (not exclusive):

• Node-Centric Community
• Each node in a group satisfies certain properties

• Group-Centric Community
• Consider the connections within a group as a whole.
The group has to satisfy certain properties without zooming into node-level

• Network-Centric Community
• Partition the whole network into several disjoint sets

• Hierarchy-Centric Community
• Construct a hierarchical structure of communities

Algorithms and Data Structures 2 // 2020W // 32 A. Ferscha

Community Analysis :: Clustering

Node Centric clustering

Nodes satisfy different properties:

Complete Mutuality
• Cliques

Reachability of members
• k-clique, k-clan, k-club

Nodal degrees
• k-plex, k-core

Relative frequency of Within-Outside Ties

• LS sets, Lambda sets
• Commonly used in traditional social network analysis
• Here, we discuss some representative ones

Algorithms and Data Structures 2 // 2020W // 33 A. Ferscha

Community Analysis :: Clustering

Clique
A maximum complete subgraph in which all nodes are adjacent to each other

Cliques with 1, 2, 3, 4, 5 and 6 vertices

Algorithms and Data Structures 2 // 2020W // 34 A. Ferscha

Community Analysis :: Clustering

Clique
A maximum complete subgraph in which all nodes are adjacent to each other

Nodes 5, 6, 7 and 8 form a clique

• NP-hard to find the maximum clique in a network

• Straightforward implementation to find cliques is very expensive in time complexity

Algorithms and Data Structures 2 // 2020W // 35 A. Ferscha

Community Analysis :: Clustering

Finding the Maximum Clique

• In a clique of size k, each node maintains degree >= k-1

• Nodes with degree < k-1 will not be included in the maximum clique

• Recursively apply the following pruning procedure

• Sample a sub-network from the given network, and find a clique in the sub-network, say, by a greedy approach
• Suppose the clique above is size k, in order to find out a larger clique, all nodes with degree <= k-1 should be removed.

• Repeat until the network is small enough

• Many nodes will be pruned as social media networks follow a power law distribution for node degrees

Algorithms and Data Structures 2 // 2020W // 36 A. Ferscha

Community Analysis :: Clustering

Maximum Clique Example

• Suppose we sample a sub-network with nodes {1, 2, 3, 4, 5} and find a clique {1, 2, 3} of size 3

• In order to find a clique >3, remove all nodes with degree <=3-1=2
• Remove nodes 2 and 9
• Remove nodes 1 and 3
• Remove node 4

Algorithms and Data Structures 2 // 2020W // 37 A. Ferscha

Community Analysis :: Clustering

Clique Percolation Method (CPM)

• Clique is a very strict definition, unstable

• Normally use cliques as a core or a seed to find larger communities

CPM is such a method to find overlapping communities

Input
• A parameter k, and a network
Procedure
• Find out all cliques of size k in a given network
• Construct a clique graph. Two cliques are adjacent if they share k-1 nodes
• Each connected components in the clique graph form a community

Algorithms and Data Structures 2 // 2020W // 38 A. Ferscha

Community Analysis :: Clustering

CPM Example

Cliques of size 3:
{1, 2, 3}, {1, 3, 4}, {4, 5, 6}, {5, 6, 7},
{5, 6, 8}, {5, 7, 8}, {6, 7, 8}

Communities:
{1, 2, 3, 4}
{4, 5, 6, 7, 8}

Algorithms and Data Structures 2 // 2020W // 45 A. Ferscha

Community Analysis :: Clustering

2-cliques: {1, 2, 3, 4, 5}, {2, 3, 4, 5, 6}

Algorithms and Data Structures 2 // 2020W // 46 A. Ferscha

Community Analysis :: Clustering

• Commonly used in traditional SNA
• Often involves combinatorial optimization

Algorithms and Data Structures 2 // 2020W // 50 A. Ferscha

Community Analysis :: Clustering

Group-Centric clustering: Density Based Groups

The group-centric criterion requires the whole group to satisfy a certain condition
• E.g., the group density >= a given threshold

A subgraph Gs (Vs , Es ) is a dense quasi-clique if

|Es |
⇥
|Vs |(|Vs | 1)/2

A similar strategy to that of cliques can be used

• Sample a subgraph, and find a maximal dense quasi-clique (say, of size k)
• Remove nodes with degree < k

Algorithms and Data Structures 2 // 2020W // 51 A. Ferscha

Community Analysis :: Clustering

Network-Centric clustering

Network-centric criterion needs to consider the connections within a network globally

Goal: partition nodes of a network into disjoint sets

Approaches:
• Clustering based on vertex similarity
• Latent space models
• Block model approximation
• Spectral clustering
• Modularity maximization

Algorithms and Data Structures 2 // 2020W // 53 A. Ferscha

Community Analysis :: Clustering

Clustering based on Vertex Similarity

• Apply k-means or similarity-based clustering to nodes

• Vertex similarity is defined in terms of the similarity of their neighborhood
• Structural equivalence: two nodes are structurally equivalent if they are connecting to the same set of actors

Nodes 1 and 3 are

structurally equivalent;

• Structural equivalence is too restrictive for practical use.

Algorithms and Data Structures 2 // 2020W // 54 A. Ferscha

Community Analysis :: Clustering

Vertex Similarity |Ni Nj |

|{5}| 1
Jaccard(4, 6) = =
|{1, 3, 4, 5, 6, 7, 8}| 7
1 1
cosine(4, 6) = =
4·4 4

Algorithms and Data Structures 2 // 2020W // 55 A. Ferscha

Community Analysis :: Clustering
Latent Space Models
• Map nodes into a low-dimensional space such that the proximity between nodes based on network connectivity is
preserved in the new space, then apply k-means clustering
• Multi-dimensional scaling (MDS)
• Given a network, construct a proximity matrix P representing the pairwise distance between nodes (e.g., geodesic distance)
• Let S ∈ R n×denote the coordinates of nodes in the low-dimensional space

1 1 T 1 T
SS T ⇡ (I 11 )(P P )(I 11 ) = Pe
€ 2 n n

• Objective function: min kSS T Pek2F

1
• Solution: S=V 2

• V is the top ` eigenvectors of Pe , and ⇤ is a diagonal matrix of top eigenvalues = diag( 1, 2, · · · , `)

Sosa, J. and Buitrago, L., 2021. A Review of Latent Space Models for Social Networks.
Revista Colombiana de Estadística, 44(1), pp.171-200.
Algorithms and Data Structures 2 // 2020W // 56 A. Ferscha
Community Analysis :: Clustering

MDS Example 2
0 1 1 1 2 2 3 3 4
3
6 1 0 1 2 3 3 4 4 5 7
6 7
6 1 1 0 1 2 2 3 3 4 7
6 7
6 1 2 1 0 1 1 2 2 3 7
geodesic 6
P =6
6 2 3 2 1 0 1 1 1 2
7
7
7
0.8
distance 6
6
6
2 3 2 1 1 0 1 1 2 7
7
7
9 6 3 4 3 2 1 1 0 1 1 7
0.6
4 3 4 3 2 1 1 1 0 2 5
4 5 4 3 2 2 1 2 0
0.4

2 3
0.2 2 2.46 3.96 1.96 0.85 0.65 0.65 2.21 2.04 3.65
7
6 3.96 6.46 3.96 1.35 1.15 1.15 3.71 3.54 6.15 7
1 3 6 7
0 4 6 1.96 3.96 2.46 0.85 0.65 0.65 2.21 2.04 3.65 7
5 6 6 7
S2

6 0.85 1.35 0.85 0.23 0.27 0.27 0.82 0.65 1.27 7

−0.2 6 7
Pe = 6
6 0.65 1.15 0.65 0.27 0.23 0.27 0.68 0.85 1.23 7
7
6 0.65 1.15 0.65 0.27 0.27 0.23 0.68 0.85 1.23 7
−0.4 6 7
6 2.21 3.71 2.21 0.82 0.68 0.68 2.12 1.79 3.68 7
6 7
4 2.04 3.54 2.04 0.65 0.85 0.85 1.79 2.46 2.35 5
−0.6
3.65 6.15 3.65 1.27 1.23 1.23 3.68 2.35 6.23
−0.8
8
−1
−3 −2 −1 0 1 2 3
S
1

Two communities:
{1, 2, 3, 4} and {5, 6, 7, 8, 9}

Algorithms and Data Structures 2 // 2020W // 57 A. Ferscha

Community Analysis :: Clustering

Block Models

min ||A S S T ||2F

• S is the community indicator matrix

• Relax S to be numerical values, then the optimal solution corresponds to the top eigenvectors of A

Two communities:
{1, 2, 3, 4} and {5, 6, 7, 8, 9}

Algorithms and Data Structures 2 // 2020W // 58 A. Ferscha

Community Analysis :: Clustering

Cut
• Most interactions are within a group, whereas interactions between groups are few
• community detection (clustering) à minimum cut problem
• Cut: A partition of vertices of a graph into two disjoint sets
• Minimum cut problem: find a graph partition such that the number of edges between the two sets is minimized

Algorithms and Data Structures 2 // 2020W // 59 A. Ferscha

Community Analysis :: Clustering

Ratio Cut & Normalized Cut

• Minimum cut often returns an imbalanced partition, with one set being a singleton (e.g. node 9)
• Change the objective function to consider community size

k
1 X cut(Ci , C̄i )
Ratio Cut( ) = , Ci,: a community
k i=1 |Ci |
cut(A, B): number of edges induced by cut
k
1 X cut(Ci , C̄i ) |Ci|: number of nodes in Ci (=community size)
Normalized Cut( ) = vol(Ci): sum of degrees in Ci
k i=1 vol(Ci )

Algorithms and Data Structures 2 // 2020W // 60 A. Ferscha

Community Analysis :: Clustering

Ratio Cut & Normalized Cut Example

For partition in red: ⇡1
✓ ◆
1 1 1
Ratio Cut( 1 ) = + = 9/16 = 0.56
2 1 8
✓ ◆
1 1 1
Normalized Cut( 1 ) = + = 14/27 = 0.52
2 1 27
For partition in green:
⇡2
✓ ◆
1 2 2
Ratio Cut( 2 ) = + = 9/20 = 0.45 < Ratio Cut( 1 )
2 4 5
✓ ◆
1 2 2
Normalized Cut( 2 ) = + = 7/48 = 0.15 < Normalized Cut( 1)
2 12 16

Both ratio cut and normalized cut prefer a balanced partition

Algorithms and Data Structures 2 // 2020W // 61 A. Ferscha

Community Analysis :: Clustering

Spectral Clustering
• Both ratio cut and normalized cut can be reformulated as

min e
T r(S T LS)
S {0,1}n⇥k
• Where
⇢ graph Laplacian for ratio cut
e= D A
L 1/2 1/2 normalized graph Laplacian
I D AD
D = diag(d1 , d2 , · · · , dn ) diagonal matrix of degrees

• Spectral relaxation: e
min T r(S T LS) s.t. S T S = Ik
S

• Optimal solution: top eigenvectors with the smallest eigenvalues

Algorithms and Data Structures 2 // 2020W // 62 A. Ferscha

Community Analysis :: Clustering

Spectral Clustering Example

Two communities:
{1, 2, 3, 4} and {5, 6, 7, 8, 9}

The 1st eigenvector k-means

D = diag(3, 2, 3, 4, 4, 4, 4, 3, 1) means all nodes belong
to the same cluster,
2 3 no use 2 3
3 1 1 1 0 0 0 0 0 0.33 0.38
6 1 2 1 0 0 0 0 0 0 7 6 0.33 0.48 7
6 7 6 7
6 1 1 3 1 0 0 0 0 0 7 6 0.33 0.38 7
6 7 6 7
6 1 0 1 4 1 1 0 0 0 7 6 0.33 0.12 7
6 7 6 7
e=D
L A=6
6 0 0 0 1 4 1 1 1 0 7
7 S=6
6 0.33 0.16 7
7
6 0 0 0 1 1 4 1 1 0 7 6 0.33 0.16 7
6 7 6 7
6 0 0 0 0 1 1 4 1 1 7 6 0.33 0.30 7
6 7 6 7
4 0 0 0 0 1 1 1 3 0 5 4 0.33 0.24 5
0 0 0 0 0 0 1 0 1 0.33 0.51

Algorithms and Data Structures 2 // 2020W // 63 A. Ferscha

Community Analysis :: Clustering

Modularity Maximation
• Modularity measures the strength of a community partition by taking into account the degree distribution
• Given a network with m edges, the expected number of edges between two nodes with di and dj is

• Strength of a community:

• Modularity

• A larger value indicates a good community structure

Algorithms and Data Structures 2 // 2020W // 64 A. Ferscha

Community Analysis :: Clustering

Modularity Maximation

• Modularity matrix: B=A ddT /2m (Bij = Aij di dj /2m)

• Similar to spectral clustering, Modularity maximization can be reformulated as

1
max Q = T r(S T BS) s.t. S T S = Ik
2m
• Optimal solution: top eigenvectors of the modularity matrix
• Apply k-means to S as a post-processing step to obtain community partition

Algorithms and Data Structures 2 // 2020W // 65 A. Ferscha

Community Analysis :: Clustering

Modularity Maximation Example

Two communities:
{1, 2, 3, 4} and {5, 6, 7, 8, 9}

k-means

2 3 2 3
0.32 0.79 0.68 0.57 0.43 0.43 0.43 0.32 0.11 0.44 0.00
6 0.79 0.14 0.79 0.29 0.29 0.29 0.29 0.21 0.07 7 6 0.38 0.23 7
6 7 6 7
6 0.68 0.79 0.32 0.57 0.43 0.43 0.43 0.32 0.11 7 6 7
6 7 6 0.44 0.00 7
6 0.57 0.29 0.57 0.57 0.43 0.43 0.57 0.43 0.14 7 6 7
6 7 6 0.17 0.48 7
B=6 0.43 0.29 0.43 0.43 0.57 0.43 0.43 0.57 0.14 7
6
6
7
7 S=6
6 0.29 0.32 7
7
6 0.43 0.29 0.43 0.43 0.43 0.57 0.43 0.57 0.14 7 6 7
6 0.43 0.29 0.43 0.57 0.43 0.43 0.57 0.57 0.86 7 6 0.29 0.32 7
6 7 6 0.38 0.34 7
4 0.32 0.21 0.32 0.43 0.57 0.57 0.57 0.32 0.11 5 6 7
0.11 0.07 0.11 0.14 0.14 0.14 0.86 0.11 0.04 4 0.34 0.08 5
0.14 0.63
Modularity Matrix

Algorithms and Data Structures 2 // 2020W // 66 A. Ferscha

Community Analysis :: Clustering

A Unified View for Community Platform

Latent space models, block models, spectral clustering, and modularity maximization can be unified as

8
>
> modified proximity matrix Pe if latent space models
<
adjacency matrix A if block models
Utility Matrix M = e
>
> graph Laplacian L if spectral clustering
:
modularity maximization B if modularity maximization

Algorithms and Data Structures 2 // 2020W // 67 A. Ferscha

Community Analysis :: Clustering

Hierarchy-Centric Clustering

• Goal: build a hierarchical structure of communities

based on network topology

• Allow the analysis of a network at different resolutions

• Representative approaches:
• Divisive Hierarchical Clustering
• Agglomerative Hierarchical Clustering

Algorithms and Data Structures 2 // 2020W // 68 A. Ferscha

Community Analysis :: Clustering

Divisive Hierarchical Clustering

Divisive clustering
• Partition nodes into several sets
• Each set is further divided into smaller ones
• Network-partition can be applied for the partition

One particular example: recursively remove the “weakest” tie

• Find the edge with the least strength
• Remove the edge and update the corresponding strength of each edge

Recursively apply the above two steps until a network is discomposed

into desired number of connected components.

Each component forms a community

Algorithms and Data Structures 2 // 2020W // 69 A. Ferscha

Community Analysis :: Clustering

Edge Betweeness

• The strength of a tie can be measured by edge betweenness

• Edge betweenness: the number of shortest paths that pass along with the edge

st (e)
edge-betweenness(e) = s<t
s,t
The edge betweenness of e(1, 2) is 4, as
all the shortest paths from 2 to {4, 5, 6, 7,
8, 9} have to either pass e(1, 2) or e(2,
3), and e(1,2) is the shortest path
between 1 and 2

• The edge with higher betweenness tends to be the bridge between two communities.

Algorithms and Data Structures 2 // 2020W // 70 A. Ferscha

Community Analysis :: Clustering

Divisive clustering based on Edge Betweenness

Initial betweenness value

After remove e(4,5), the betweenness of e(4, 6)

becomes 20, which is the highest;

After remove e(4,6), the edge e(7,9) has the highest

betweenness value 4, and should be removed.

Algorithms and Data Structures 2 // 2020W // 71 A. Ferscha

Community Analysis :: Clustering

Agglomerative Hierarchical Clustering

• Initialize each node as a community

• Merge communities successively into larger communities
following a certain criterion
(E.g., based on modularity increase)

Algorithms and Data Structures 2 // 2020W // 72 A. Ferscha

Social Computing :: Community Detection

Summary of Community Detection

Node-Centric Community Detection

§ cliques, k-cliques, k-clubs

Group-Centric Community Detection

§ quasi-cliques

Network-Centric Community Detection

§ Clustering based on vertex similarity
§ Latent space models, block models, spectral clustering, modularity maximization

Hierarchy-Centric Community Detection

§ Divisive clustering
§ Agglomerative clustering

Algorithms and Data Structures 2 // 2020W // 73 A. Ferscha

Community Analysis

Univ.-Prof. Dr. Alois Ferscha

Institute of Pervasive Computing
Johannes Kepler University Linz
[email protected]

Algorithms and Data Structures 2 JOHANNES KEPLER

UNIVERSITY LINZ
Lecture – 2020W Altenberger Straße 69
4040 Linz, Austria
jku.at

Long Distance Messages
100% (16)
Long Distance Messages
8 pages
1.BIT 02105 DATA STRUCTURES AND ALGORITHMS Course Outline
No ratings yet
1.BIT 02105 DATA STRUCTURES AND ALGORITHMS Course Outline
3 pages
An Introduction to Information Theory
From Everand
An Introduction to Information Theory
Fazlollah M. Reza
No ratings yet
Unit One - Hebrew Letters
0% (1)
Unit One - Hebrew Letters
4 pages
AADS-Course Handout
No ratings yet
AADS-Course Handout
30 pages
CS F232 - Fdsa
No ratings yet
CS F232 - Fdsa
4 pages
INF210 Lec1 Data Structures
No ratings yet
INF210 Lec1 Data Structures
32 pages
Bits F232
No ratings yet
Bits F232
3 pages
DSA Syllabus
No ratings yet
DSA Syllabus
3 pages
DSAD Regular HO
No ratings yet
DSAD Regular HO
6 pages
Bits F232
No ratings yet
Bits F232
4 pages
BITS F232 - FDSA - Prof. Hota
No ratings yet
BITS F232 - FDSA - Prof. Hota
4 pages
Advance Algo
No ratings yet
Advance Algo
376 pages
Intro Frist Day 1
No ratings yet
Intro Frist Day 1
18 pages
DataStruc MSCS
No ratings yet
DataStruc MSCS
4 pages
Dsad Ho
No ratings yet
Dsad Ho
6 pages
PPS Course Fie R18
No ratings yet
PPS Course Fie R18
291 pages
Data Structure - CS 221 - Course Outline
No ratings yet
Data Structure - CS 221 - Course Outline
4 pages
DSA Lec 1 Spring 2022
No ratings yet
DSA Lec 1 Spring 2022
47 pages
Course Code:22DS102
No ratings yet
Course Code:22DS102
2 pages
CS 213 2022
No ratings yet
CS 213 2022
20 pages
Bits F232 Foundations of Data Structures and Algorithms - Handout
No ratings yet
Bits F232 Foundations of Data Structures and Algorithms - Handout
5 pages
Ics 2105 Jan 2025PDF
No ratings yet
Ics 2105 Jan 2025PDF
2 pages
ADS Syllabus
No ratings yet
ADS Syllabus
4 pages
Birla Institute of Technology & Science, Pilani Course Handout Part A: Content Design
No ratings yet
Birla Institute of Technology & Science, Pilani Course Handout Part A: Content Design
8 pages
Dsa L1
No ratings yet
Dsa L1
11 pages
Data Structures and Algorithms Roadmap
No ratings yet
Data Structures and Algorithms Roadmap
31 pages
Fds Book Book
No ratings yet
Fds Book Book
133 pages
Instructor: Course Title: Course Code: Course Objectives
No ratings yet
Instructor: Course Title: Course Code: Course Objectives
3 pages
5 B Tree
No ratings yet
5 B Tree
61 pages
CS192, REviewed Advanced Data Structures - Feb 2024
No ratings yet
CS192, REviewed Advanced Data Structures - Feb 2024
9 pages
PCCST303 - Data Structures and Algorithms
100% (1)
PCCST303 - Data Structures and Algorithms
4 pages
CPCS204 01
No ratings yet
CPCS204 01
16 pages
CS F211
No ratings yet
CS F211
3 pages
IARE Data Structures Notes
No ratings yet
IARE Data Structures Notes
133 pages
ADASA Syllabus
No ratings yet
ADASA Syllabus
4 pages
CS F211 - Data Structures and Algorithms - Chittaranjan Hota
No ratings yet
CS F211 - Data Structures and Algorithms - Chittaranjan Hota
3 pages
Circular 20250122151006 Cho 22cs036 (Advanced Data Structures)
No ratings yet
Circular 20250122151006 Cho 22cs036 (Advanced Data Structures)
18 pages
Syllabus SYDE223
No ratings yet
Syllabus SYDE223
2 pages
Advanced Data Structure
No ratings yet
Advanced Data Structure
3 pages
IT3206 - Data Structures and Algorithms
No ratings yet
IT3206 - Data Structures and Algorithms
5 pages
School of Engineering and Technology: Departmental of Electrical Engineering Imperial College of Business Studies
No ratings yet
School of Engineering and Technology: Departmental of Electrical Engineering Imperial College of Business Studies
2 pages
CS482 Data Structures
No ratings yet
CS482 Data Structures
3 pages
CS F211 Handout
No ratings yet
CS F211 Handout
3 pages
FirstYear-TM (First Semester)
No ratings yet
FirstYear-TM (First Semester)
39 pages
Bece208e Data-structures-And-Algorithms Ela 1.0 0 Bece208e
No ratings yet
Bece208e Data-structures-And-Algorithms Ela 1.0 0 Bece208e
3 pages
CS202
No ratings yet
CS202
2 pages
7 Searching
No ratings yet
7 Searching
48 pages
Dcap201 Fundamentals of Data Structures PDF
100% (1)
Dcap201 Fundamentals of Data Structures PDF
270 pages
DSA Report
No ratings yet
DSA Report
33 pages
CS482 Data Structures (Ktustudents - In)
No ratings yet
CS482 Data Structures (Ktustudents - In)
3 pages
DSA Course Outline
No ratings yet
DSA Course Outline
4 pages
Cse2002 Data-structures-And-Algorithms LTP 2.0 2 Cse2002 Data-structures-And-Algorithms LTP 2.0 1 Cse2002-Data Structures and Algorithms
No ratings yet
Cse2002 Data-structures-And-Algorithms LTP 2.0 2 Cse2002 Data-structures-And-Algorithms LTP 2.0 1 Cse2002-Data Structures and Algorithms
3 pages
01 - Introduction + Sorting
No ratings yet
01 - Introduction + Sorting
49 pages
Dsa 2
No ratings yet
Dsa 2
4 pages
E0 251 Aug 3:1 Data Structures and Algorithms: Instructor
No ratings yet
E0 251 Aug 3:1 Data Structures and Algorithms: Instructor
2 pages
CS 202-EE 202-Data Structures-Ihsan Qazi
No ratings yet
CS 202-EE 202-Data Structures-Ihsan Qazi
2 pages
Course Contents
No ratings yet
Course Contents
1 page
Bcse202l Data-structures-And-Algorithms TH 1.0 67 Bcse202l
No ratings yet
Bcse202l Data-structures-And-Algorithms TH 1.0 67 Bcse202l
3 pages
CC104 - Module 1
No ratings yet
CC104 - Module 1
10 pages
Data Structure and Algorithm (Updated)
No ratings yet
Data Structure and Algorithm (Updated)
2 pages
Cse205: Data Structures and Algorithms: Session 2020-2021 Page: 1/2
No ratings yet
Cse205: Data Structures and Algorithms: Session 2020-2021 Page: 1/2
2 pages
Checklist For Writing Strong Paragraphs
No ratings yet
Checklist For Writing Strong Paragraphs
4 pages
Tata Steelium Case Study
No ratings yet
Tata Steelium Case Study
13 pages
Have You Been Waiting Have Become I've Always Has Been Doing Have Been Having Have Made
No ratings yet
Have You Been Waiting Have Become I've Always Has Been Doing Have Been Having Have Made
4 pages
Master Tung Acupuncture Points - Page 2
No ratings yet
Master Tung Acupuncture Points - Page 2
6 pages
DLL Template Sexual g2
No ratings yet
DLL Template Sexual g2
6 pages
Sample Lesson Plan
No ratings yet
Sample Lesson Plan
3 pages
SPECIMEN COLLECTION and Processing For BACTERIOLOGY PDF
No ratings yet
SPECIMEN COLLECTION and Processing For BACTERIOLOGY PDF
19 pages
GATE Aerospace 2016 - GATE Aerospace Syllabus - Are You Gateing
No ratings yet
GATE Aerospace 2016 - GATE Aerospace Syllabus - Are You Gateing
2 pages
Battle Belongs-Phil Wickham-Hymn of Heaven - Db.numerals
No ratings yet
Battle Belongs-Phil Wickham-Hymn of Heaven - Db.numerals
3 pages
The Passive Voice - April
No ratings yet
The Passive Voice - April
22 pages
7378
No ratings yet
7378
19 pages
Jurnal Harian Pai Kelas 4 TW 4
No ratings yet
Jurnal Harian Pai Kelas 4 TW 4
6 pages
Lesson 1: - Competencies Required of An Effective Entrepreneur
No ratings yet
Lesson 1: - Competencies Required of An Effective Entrepreneur
24 pages
Śrī K A Caitanya Book Compilation by His Holiness Jayapatākā Swami Mahārāja On 20 June 2020 in Śrīdhāma Māyāpur, India
No ratings yet
Śrī K A Caitanya Book Compilation by His Holiness Jayapatākā Swami Mahārāja On 20 June 2020 in Śrīdhāma Māyāpur, India
7 pages
Educ 102 Midterm
No ratings yet
Educ 102 Midterm
3 pages
You Are My Mother - Inspired by P - D - Eastman's Are You My - P - D - Eastman - First Edition, New York, 2020 - Random House Children's Books
No ratings yet
You Are My Mother - Inspired by P - D - Eastman's Are You My - P - D - Eastman - First Edition, New York, 2020 - Random House Children's Books
32 pages
Lesson 5 Shape Poems 2
No ratings yet
Lesson 5 Shape Poems 2
6 pages
Real Estate Mortgage
No ratings yet
Real Estate Mortgage
2 pages
Background Check Police
No ratings yet
Background Check Police
1 page
481 Midterm W05
100% (1)
481 Midterm W05
9 pages
Full PRE-TT-1S Term Test 1 Standard With Answers A
No ratings yet
Full PRE-TT-1S Term Test 1 Standard With Answers A
13 pages
Fun Eq-Tp 2
No ratings yet
Fun Eq-Tp 2
15 pages
5 Ethical Approaches
100% (1)
5 Ethical Approaches
5 pages
YOU SAY CHORDS (Ver 2) by Lauren Daigle @
0% (1)
YOU SAY CHORDS (Ver 2) by Lauren Daigle @
1 page
Freedom On The Net 2021
No ratings yet
Freedom On The Net 2021
43 pages
Animal Tissue: Haroen Rasyid
100% (1)
Animal Tissue: Haroen Rasyid
41 pages
Middle Child Syndrome Everything You Need To Know
No ratings yet
Middle Child Syndrome Everything You Need To Know
1 page
Rule of Congruence On Triangles
No ratings yet
Rule of Congruence On Triangles
4 pages