Probability

Tuesday, February 4, 14

What is Probability?

Probability is a mathematical discipline akin to geometry.

Formal Logic Content. Intuitive Background. Applications.

Tuesday, February 4, 14

What is Probability?

The actual science of logic is conversant at present only with things either certain, impossible, or entirely doubtful, none of which (fortunately) we have to reason on. Therefore the true logic for this world is the calculus of Probabilities, which takes account of the magnitude of the probability which is , or ought to be, in a reasonable man's mind James Clerck Maxwell (1850)

Tuesday, February 4, 14

Why to study Probability?

CS Theory

Alg

M Le ach ar ine nin g

ori

thm

Probability

ion t a y m or eor f In Th

N w et ks or

Tuesday, February 4, 14

Probabilistic Model
The sample space , which is the set of all possible outcomes of an experiment. The probability law, which assigns to a set A of possible outcomes (also called an event) a nonnegative number P(A) (called probability of A) that encodes our knowledge or belief about the collective likelihood of the elements A. The probability law must satisfy certain properties to be introduced shortly.

Tuesday, February 4, 14

Probabilistic Model
Probability law B

P( B )
A

Experiment Sample Space

P ( A)

(Set of possible outcomes)

Events

Tuesday, February 4, 14

Set Theory

Finite, Countable and Uncountable Sets. Set Operations. The Algebra of Sets.

Tuesday, February 4, 14

Sets
A set is a collection of objects, which are the elements of the set. xS x /S

Tuesday, February 4, 14

Specifying Sets
As a list of elements: S = {x1 , x2 , . . . , xn } With words: the set of even natural numbers Specify a rule or algorithm: S = {r Q : r < 2}
2

Tuesday, February 4, 14

Subsets
If every element of a set S is also an element of a set T , we say that S is a subset of T , and we write S T or T S If S T and T S the two sets are equal , and we write S=T The universal set, denoted by , contains all the objects of interest in a particular context.

Tuesday, February 4, 14

Set Operations
The complement of a set S , with respect to the universe is the set of elements in that do not belong to S : S = {x : x / S}
c

Tuesday, February 4, 14

Set Operations
The union of two sets S and T is the set of all elements that belong to S or T (or both): S T = {x : x S orx T } The intersection of two sets S and T is the set of all elements that belong to both S and T : S T = {x : x S andx T }

Tuesday, February 4, 14

Algebra of Sets

Tuesday, February 4, 14

De Morgans laws

Tuesday, February 4, 14

Functions

Domain

Range

Tuesday, February 4, 14

Cardinality
{Thorin,Balin, Bifur, Bofur, Bombur, Dori, Fili, Gloin, Kili} {Gruon, Mocoso, Tmido, Mudito, Dormiln, Felz, Sabio}
Tuesday, February 4, 14

Cardinality

Tuesday, February 4, 14

Cardinality

Tuesday, February 4, 14

Cardinality

Tuesday, February 4, 14

Cardinality

Tuesday, February 4, 14

Cardinality

Tuesday, February 4, 14

Cardinality

Tuesday, February 4, 14

Power Sets and Cantors Theorem

Tuesday, February 4, 14

Probabilistic Model
The sample space , which is the set of all possible outcomes of an experiment. The probability law, which assigns to a set A of possible outcomes (also called an event) a nonnegative number P(A) (called probability of A) that encodes our knowledge or belief about the collective likelihood of the elements A. The probability law must satisfy certain properties to be introduced shortly.

Tuesday, February 4, 14

Probabilistic Model
Probability law B

P( B )
A

Experiment Sample Space

P ( A)

(Set of possible outcomes)

Events

Tuesday, February 4, 14

Sample Space

Different elements of the sample space should be distinct and mutually exclusive. The sample space must be collectively exhaustive.

Tuesday, February 4, 14

Sample Space (Examples)

For a single toss coin the sample space consist of two points: = {H, T } (We exclude possibilities like the coin stands on edge, the coin disappears, etc.)

Tuesday, February 4, 14

Sample Space (Examples)

For n tosses of a coin the sample space is = {w : w = (a1 , . . . , an ), ai = H or T } and the general number N () of outcomes is 2n .

Tuesday, February 4, 14

Sample Space (Examples)

For n tosses of a coin the sample space is = {w : w = (a1 , . . . , an ), ai = H or T } and the general number N () of outcomes is 2n .

Tuesday, February 4, 14

Sample Space (Examples)

ue Tre nc eia ba ld s es ed cr ip tio n

(1,1) 1 (1,2) (1,3) (1,4)

Sample space for pair of 4-sided rolls 4

2nd roll

se q

3 root 2 1 1 2 3 1st roll 4

Tuesday, February 4, 14

Probability Laws
1. (Non-negativity) P(A) 0, for every event A.

Tuesday, February 4, 14

Probability Laws
2. (Additivity) If A and B are two disjoint events, then the probability of their union satises P ( A B ) = P ( A) + P ( B ) More generally, if the sample space has an innite number of elements and A1 , A2 , . . . is a sequence of disjoint events then the probability of their union satises P ( A1 A2 ) = P ( A1 ) + P ( A2 ) +

Tuesday, February 4, 14

Probability Laws
3. (Normalization) The probability of the entire sample space is equal to 1, that is P() = 1

Tuesday, February 4, 14

Properties
Let A, B and C be events. 1. P() = 0. 2. If A B , then P(A) P(B ) 3. P(A B ) = P(A) + P(B ) P(A B ) 4. P(A B ) P(A) + P(B )

Tuesday, February 4, 14

Discrete Probability Laws

If the sample space consists of a nite number of possible outcomes, then the probability law is specied by the probabilities of the events that consists of a single element. In particular, the probability of any event A = {s1 , s2 , . . . , sn } is the sum of the probabilities of its elements: P ( A) = P ( s 1 ) + P ( s 2 ) + + P ( s n )

Tuesday, February 4, 14

Discrete Uniform Probability Laws

If the sample space consists of n possible outcomes which are equally likely (i.e. all single-element events have the same probability), then the probability of any event A is given by N ( A) P ( A) = n

Tuesday, February 4, 14

Discrete Uniform Probability Laws

Sample space for pair of 4-sided rolls 4

2nd roll

3 2 1 1 2 3 1st roll 4

{at least one roll is a 4}

{the rst roll is equal to the second}

Tuesday, February 4, 14

Counting

Tuesday, February 4, 14

Cardinality

Tuesday, February 4, 14

Product Rule
Theorem (Pairs). With n elements a1 , a2 , . . . , an and m elements b1 , b2 , . . . , bm , it is possible to form nm pairs (ai , bj ) containing one element for each group.

Tuesday, February 4, 14

Generalized Product Rule

Theorem. Given n1 elements a1 , a2 , . . . , an1 and n2 elements b1 , b2 , . . . , bn2 , etc., up to nr elements x1 , x2 , . . . , xnr ; it is possible to form n1 n2 nr ordered r-tuples (ai1 , ai2 , . . . , air ) containing one element for each group.

Tuesday, February 4, 14

Generalized Product Rule

n1 n2 n3 n4

choices Stage 1

choices Stage 2

choices Stage 3

choices Stage 4

Tuesday, February 4, 14

Examples
Throwing a dice r times. No 1 in r throws? Display of ags. Suppose r ags of dierent colors are to be displayed on n poles in a row. In how many ways can this be done? Loops, Recursions, ...

Tuesday, February 4, 14

Ordered Samples
Consider a set or population of n elements a1 , . . . , an . Any ordered arrangement (ai1 , . . . , aik ) of k symbols is called an ordered sample of size k drawn from our population. Sampling with replacement (repetitions are allowed). Sampling without replacement (repetitions are not allowed).

Tuesday, February 4, 14

Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Ordered Sample with Replacement

Tuesday, February 4, 14

Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Ordered Sample with Replacement

Tuesday, February 4, 14

Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Ordered Sample with Replacement

Tuesday, February 4, 14

Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Ordered Sample with Replacement

Tuesday, February 4, 14

Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Ordered Sample with Replacement

N () = nk

Tuesday, February 4, 14

Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Ordered Sample with Replacement

N () = nk

Tuesday, February 4, 14

Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Ordered Sample without Replacement

Tuesday, February 4, 14

Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Ordered Sample without Replacement

Tuesday, February 4, 14

Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Ordered Sample without Replacement

Tuesday, February 4, 14

Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Ordered Sample without Replacement

Tuesday, February 4, 14

Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Ordered Sample without Replacement

N () = (n)k

Tuesday, February 4, 14

Ordered Samples
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Ordered Sample without Replacement

N () = (n)k

Tuesday, February 4, 14

Ordered Samples
Theorem. For a population of n elements and a prek scribed sample size k , there exist n dierent samples with replacement and (n)k samples without replacement.

Tuesday, February 4, 14

Ordered Samples
When k = n that sample is called a permutation of the elements of the population and (n)n = n! Whenever we speak of random samples of xed size k , the adjective random is to imply that all samples have the same probability, namely, nk in sampling with replacement and 1/(n)k in sampling without replacement. If n is large and k is relatively small, the ratio (n)k /nk is near unity i.e. the two ways of sampling are practically equivalent.

Tuesday, February 4, 14

Examples (Balls and Bins)

Probability an element is not included in the sample? If n balls are randomly placed into n cells, what is the probability every cell is occupied?

Tuesday, February 4, 14

Birthday Paradox
Throw a dice 6 times, what is the probability all six faces appear? Elevator (10 oors, 7 people)

Tuesday, February 4, 14

The Sum Rule

Theorem. If A1 , A2 , . . . , An are disjoint sets, then: | A1 A2 An | = | A1 | + | A2 | + + | An |

Tuesday, February 4, 14

Counting Passwords
On a certain computer system, a valid password is a sequence between six and eight symbols. The rst symbol must be a letter which can be upper case or lower case and the remaining symbols must be either letter or numbers.

Tuesday, February 4, 14

Subpopulations
Two populations are considered dierent only if one contains an element not contained in the other.

Tuesday, February 4, 14

Subpopulations
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Unordered Sample without Replacement

Tuesday, February 4, 14

Subpopulations
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Unordered Sample without Replacement

Tuesday, February 4, 14

Subpopulations
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Unordered Sample without Replacement

Tuesday, February 4, 14

Subpopulations
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Unordered Sample without Replacement

Tuesday, February 4, 14

Subpopulations
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Unordered Sample without Replacement

n N () = k

Tuesday, February 4, 14

Subpopulations
We now consider examples involving the selection of k balls from an urn containing n distinguishable balls.

Unordered Sample without Replacement

n N () = k

Tuesday, February 4, 14

Subpopulations
Theorem. A population of n elements possesses ferent sub-populations of size r n. n n = r nr n = 1, 0! = 1 0
n r

dif-

Tuesday, February 4, 14

The Division Rule

Theorem. If f : A B is k -to-1, then: | A| = k | B | Unlike as it may seem, many counting problems are made much easier by initially counting every item multiple times and the correcting the answer using the division rule.

Tuesday, February 4, 14

Examples
Hands of poker. What is the probability that a hand of poker contains contains ve dierent face values? Occupancy problem. Probability that a specied cell contains exactly k balls?

Tuesday, February 4, 14

Examples (Balls and bins)

Tuesday, February 4, 14

Partitions
Theorem. Let k1 , . . . , kr be integers such that k1 + k2 + + kr = n, ki 0,

The number of ways in which a population of n elements can be divided into k ordered parts (partitioned into k subpopulations) of which the rst contains k1 elements, the second k2 elements, etc., is n! k1 ! k2 ! kr !

Tuesday, February 4, 14

Example: Sequences with repetitions

How many sequences can be formed by permuting the letters in the 10-letter word BOOKKEEPER?

Tuesday, February 4, 14

Example: Balls and Bins

Tuesday, February 4, 14

Example: Binomial Theorem

Theorem. For all n N and a, b R: ( a + b) =
n

k=0

n X

n n k k a b k

Tuesday, February 4, 14

Example: Multinomial Theorem

Theorem. For all n N and zi R: X ( z1 + z2 + + zm ) n k1 k2 km z1 z2 zm k 1 , . . . km
n

k1 , . . . , k m N k1 + + km = n

Tuesday, February 4, 14

Combinatorial Proofs
Symmetry n n = k nk Pascals Identity n n1 n1 = + k k1 k 3n n = n X
r =0

n r

2n nr

Tuesday, February 4, 14

Conditional Probability

Tuesday, February 4, 14

Conditional Probability
Conditional Probability provides us with a way to reason about the outcome of an experiment, based on partial information. We seek to construct a new probability law that takes into account the new information: a probability law that form ant even A, species the conditional probability of A given B denoted by P ( A| B )

Tuesday, February 4, 14

Conditional Probability
P(A|B ) must constitute a legitimate probability law. The conditional probabilities must be consistent with our intuition in important special cases P( A B ) P ( A| B ) = P( B ) where we assume that P(B ) > 0.

Tuesday, February 4, 14

Conditional Probability
1. (Non-negativity) P(A|B ) 0, for every event A.

Tuesday, February 4, 14

Conditional Probability
2. (Additivity) If A1 and A2 are two disjoint events, then the probability of their union satises P ( A1 A2 | B ) = P ( A1 | B ) + P ( A2 | B ) More generally, if the sample space has an innite number of elements and A1 , A2 , . . . is a sequence of disjoint events then the probability of their union satises P ( A 1 A2 | B ) = P ( A1 | B ) + P ( A2 | B ) +

Tuesday, February 4, 14

Conditional Probability
3. (Normalization) The probability of the entire sample space is equal to 1, that is P( | B ) = 1

Tuesday, February 4, 14

Examples
We toss a coin three successive times. Compute the conditional probability P(A|B ) when A and B are the events: A = { more heads than tails come up} B = {1st toss is a head}

Tuesday, February 4, 14

Conditional Probability for Modeling

When building models for experiments with sequential character, it is natural to rst specify conditional probabilities and the use them to determine unconditional probabilities: P ( A B ) = P ( B ) P ( A| B )

Tuesday, February 4, 14

Examples
If an aircraft is present in a certain area, a radar detects it and generates an alarm signal with probability 0.99. If an aircraft is not present, the radar generates a (false) alarm with probability 0.10. We assume that an aircraft is present with probability 0.005. What is the probability of no aircraft presence and a false alarm? What is the probability of aircraft presence and no detection?

Tuesday, February 4, 14

Examples
Aircraft Present
P(
P(

P
5 0 . 0

) A | (B

.99 0 =

= ) A

P( B c | A)

=0

.01

Missed Detection

Ac )=

0. 9

P
5

A (B |

c)

0 1 . 0

False Alarm

P( B c |A c )=

0.90

Tuesday, February 4, 14

Multiplication Rule
Assuming that all of the conditioning events have positive probability, we have:
n \ n 1 \ i=1

P(

i=1

Ai ) = P ( A1 ) P ( A2 | A1 ) P ( A3 | A1 A2 ) P ( An |

Ai )

Tuesday, February 4, 14

Example
Three cards are drawn from an ordinary 52-card deck without replacement. What is the probability that none of the three cards is a heart, assuming that all triplets are equally likely?

Tuesday, February 4, 14

The Monty Hall Problem

1 2 3

Tuesday, February 4, 14

The Monty Hall Problem

1 2 3

Tuesday, February 4, 14

The Monty Hall Problem

1 2 3

Tuesday, February 4, 14

The Monty Hall Problem

1 2 3

Tuesday, February 4, 14

The Monty Hall Problem

D 1 1 2 3 O 2 3 3 2 P( D )
1 3 1 3 1 3 1 3

P( O | D ) p (1 p) 1 1

P ( D ) P( O | D ) 1 3p 1 3 (1 p)
1 3 1 3

Keep Win Win Lose Lose

Switch Lose Lose Win Win

Tuesday, February 4, 14

Minimum Global Cut

Global min cut. Given a connected, undirected graph G = (V, E ), nd a cut (A, B ) of minimum cardinality. Applications. Identify clusters, network reliability, TSP solvers.

Tuesday, February 4, 14

Contraction algorithm

Minimum Global Cut

- replace u and v by single new super-node w - keep parallel edges, but delete self-loops - preserve edges, updating endpoints of u and v to w

Contraction algorithm. [Karger 1995]

Pick an edge e = (u, v) uniformly at random. Contract edge e.

Repeat until graph has just two nodes v1 and v1. Return the cut (all nodes that were contracted to form v1).

b d e

u
f

contract u-v f

10

Tuesday, February 4, 14

p parallel edges, but delete self-loops

1

at until graph has just two nodes v1 and v1.

Minimum Global Cut n the cut (all nodes that were contracted to form v ).

Reference: Thore Husfeldt

Tuesday, February 4, 14

Minimum Global Cut

Tuesday, February 4, 14

Total Probability Theorem

Let A1 , A2 , . . . , An be events that form a partition of the sample space , that is Ai Aj = for i 6= j and
n [

Ai =

i=1

and assume that P(Ai ) > 0, for all i. Then, for any event B , we have P ( B ) = P ( A1 ) P ( B | A1 ) + P ( An ) P ( B | An )

Tuesday, February 4, 14

Total Probability Theorem

A1 B A3

A4 A2

Tuesday, February 4, 14

Example
You enter a chess tournament where your probability of winning is 0.3 against half the players (call then type I), 0.4 against a quarter of the players (call the type II), and 0.5 against the remainder quarter of the players (call them type III). You play a game against a randomly chosen opponent. What is the probability of winning?

Tuesday, February 4, 14

Inference and Bayes Theorem

Let A1 , A2 , . . . , An be disjoint events that form a partition of the sample space, and assume that P(Ai ) > 0, for all i, then, for any event B such that P(B ) > 0, we have P ( Ai | B ) = = P ( Ai ) P ( B | Ai ) P( B ) P ( Ai ) P ( B | Ai ) P ( A1 ) P ( B | A1 ) + + P ( An ) P ( B | An )

Tuesday, February 4, 14

Inference and Bayes Theorem

Prior
P ( Ai ) P ( B | Ai ) P ( Ai | B ) = P( B )

Posterior

Tuesday, February 4, 14

Example
Probability that an aircraft is present? Suppose that you won. What is the probability you had an opponent of type I? The False-Positive Puzzle.

Tuesday, February 4, 14

Examples
If an aircraft is present in a certain area, a radar detects it and generates an alarm signal with probability 0.99. If an aircraft is not present, the radar generates a (false) alarm with probability 0.10. We assume that an aircraft is present with probability 0.005. What is the probability of aircraft presence given alarm went o ?

Tuesday, February 4, 14

Example
You enter a chess tournament where your probability of winning is 0.3 against half the players (call then type I), 0.4 against a quarter of the players (call the type II), and 0.5 against the remainder quarter of the players (call them type III). You play a game against a randomly chosen opponent. What is the probability of having an opponent of type I given you won?

Tuesday, February 4, 14

Independence

Tuesday, February 4, 14

Independent Events
When the occurrence of B does not alter the probability of A: P ( A| B ) = P ( A) we say that A is independent of B . Equivalently, P ( A B ) = P ( A) P ( B )

Tuesday, February 4, 14

Example
Consider an experiment involving two successive rolls of a 4-sided die in which all 16 possible outcomes are equally likely. Ai = {1st roll results in i}, A = {1st roll is a 1}, Bj = {2nd roll results in j } B = {sum is a 5}

A = {minimum is 2},
Tuesday, February 4, 14

B = {maximum is 2}

Conditional Independence
Given an even C , the events A and B are called conditionally independent if P ( A B | C ) = P ( A| C ) P ( B | C )

Tuesday, February 4, 14

Example

Tuesday, February 4, 14

Independence of a Collection of Events

We say that the events A1 , A2 , . . . , An are independent if Y \ P(Ai ), for every subset S of {1, 2, . . . , n} P( Ai ) =
i S i S

Tuesday, February 4, 14

Pairwise independence does not imply independence

Tuesday, February 4, 14

Example

Tuesday, February 4, 14

Minimum Global Cut

Tuesday, February 4, 14

Network Connectivity
0.8 0.9

E
0. 9

0.9

A
0. 7 5

0.85

0.95

Tuesday, February 4, 14

Network Connectivity
1

2 3 Parallel Connection

Series Connection

Tuesday, February 4, 14

Independent Trials and Binomial Probabilities

If an experiment involves a sequence of independent but identical stages, we say we have a sequence of independent trials. In the special case where there are only two possible results at each stage, we say we have a sequence of independent Bernoulli trials.

Tuesday, February 4, 14

Contention Resolution in a Contention resolution in a distributed system Distributed System

Contention resolution. Given n processes P1, , Pn, each competing for access to a shared database. If two or more processes access the database simultaneously, all processes are locked out. Devise protocol to ensure all processes get through on a regular basis. Restriction. Processes can't communicate. Challenge. Need symmetry-breaking paradigm.

P1

P2 . . . Pn
Tuesday, February 4, 14
4