Lectures
Lectures
Intro lecture
Predicate / propositional logic
Model checking * Automata on infinite words (Buchi automata) * Binary deci-
sion diagrams * SAT solving * DPLL, CDCL * Abstraction refinement * Predi-
cate abstraction * Counter-example guided abstraction refinement
Verifying programs * Invariants * Interpolents * Ranking functions * Hoare logic
Advanced topics: * Timed automata: Automata with clocks. Model real time
behavior of systems. * Verification of deep neural networks
Logic flavors
Careful balance between * how complex systems can be specified * automatically
argue about the correctness, etc
Propositional logic
If the train arrives late and there are no taxis at the station, then
John is late for his meeting. John is not late for his meeting. The
train did arrive late. Therefore,there were taxis at the station.
p: train arrives late
q: there are taxis at the station
r: John is late
If p and not q then r. Not r. p. Then q.
Syntax of propositional logic
• Unbounded supply of atoms (atomic propositions)
𝜙 ∶= 𝑝|¬𝜙|𝜙 ∧ 𝜓|𝜙 ∨ 𝜓|𝜙 → 𝜓 where p is an atom
This is called the BNF form.
Syntax now allows us to write formulas. But we also need to give meaning to
formulas.
Semantics of propositional logic Atoms and formulas take values: either
true or false.
𝑝 ¬𝑝
T F
F T
1
Similarly for others. Defined semantics using truth tables. Using truth tables
we can figure out if complex formulas are true or not. (𝑝 → ¬𝑞) → (𝑞 ∨ ¬𝑝)
• Satisifiability: If there is some assignment of atoms such that the formula
can be true.
• Validity: If the formula is true for every assignment of atoms.
(𝑝 ∨ 𝑞) → 𝑝 is satisfiable but not valid.
𝜙 is satisfiable iff ¬𝜙 is valid.
Normal forms * Conjunctive normal form (CNF) * Disjunctive normal form
(DNF) * Negation normal form (NNF)
Which of these forms make it easier to answer satisfiability? To answer validity?
Lecture 2
Predicate logic semantics
Given a signature 𝜎, we need 𝜎 structure/assignment. * A non-empty universe
over which 𝜎 gets meaning 𝑈𝐴 .
* We have to define for which a predicate is true. k-ary predicate : 𝑈𝐴 ∗𝑈𝐴 ...∗𝑈𝐴
* Every k-ary function : 𝑈𝐴 ∗ 𝑈𝐴 ... ∗ 𝑈𝐴 → 𝑈𝐴 * Constant 𝑐 ∈ 𝑈𝐴 * Variables
𝑣 ∈ 𝑈𝐴
Satisfaction relation 𝐴 is an assignment, 𝐹 is a formula: 𝐴 ⊧ 𝐹
2
Given an assignment, every term can be evaluated 𝐴[[𝑡1]], 𝐴[[𝑡2]], .. 𝐴[[𝑡𝑛]]. It
becomes easy to know if formula is satisfied.
Universe is non-empty but it can be infinite. ∀ and ∃ become difficult. How to
find an assignment?
Semantic entailment: 𝜙1 , 𝜙2 , ..., 𝜙𝑛 ⊧ 𝜓 When LHS is true, RHS should be
true. Semantic entailment is difficult. We have to argue about every assignment.
Easier in propositional logic since assignments can be enumerated (just true or
false).
∀𝑥(𝑃 (𝑥) → 𝑄(𝑥)) ⊧ ∀𝑥𝑃 (𝑥) → ∀𝑥𝑄(𝑥) holds.
∀𝑥𝑃 (𝑥) → ∀𝑥𝑄(𝑥) ⊧ ∀𝑥(𝑃 (𝑥) → 𝑄(𝑥)) does not hold.
This assignment is ok for LHS but not for RHS * 𝑃 (𝑥0 ) is false, 𝑄(𝑥0 ) is true *
𝑃 (𝑥1 ) is true, 𝑄(𝑥1 ) is false
Derivability |− . They are part of the proof rules. Used by automated theorem
provers. Derivability vs entailment?
Model checking
Model the system. Does it satisfy the property? Computer can say here is why
it does not satisfy the property.
x = 0; y = 0;
while(*) // * is a non-deterministic condition
x++; y++;
while(x != 0 /\ y != 0)
x--; y--;
assert(x = 0 /\ y = 0)
Assertion holds. At assertion, either x = 0 or y = 0. And x and y are always
equal.
Are we able to specify the system? Are we able to automatically verify its
properties? What kind of properties are we not able to specify?
3
• G: globally
• U: until
𝐹 (𝑝 → 𝐺𝑟) ∨ (¬𝑞 ∪ 𝑝) is a valid formula.
• Unary primitives bind most tightly: ¬, 𝑋, 𝐹 , 𝐺
• U
• ∧, ∨
• →
Parse tree:
$\lor$
/ \
F ...
|
$\rightarrow$
| \
p G
|
r
𝜓 is a subformula of 𝜙 if the parse tree of 𝜓 is a subtree of the parse tree of 𝜙.
4
• 𝜋 ⊧ 𝑝 if the first state satisifies 𝑝 according to the labelling function, i.e,
𝑝 ∈ 𝐿(𝑆0)
• similar for ∨ etc
• 𝜋 ⊧ 𝑋𝜙 if the second state satisifies 𝑝 according to the labelling function,
i.e, 𝑝 ∈ 𝐿(𝑆1), 𝜋2 ⊧ 𝜙
• 𝜋 ⊧ 𝐹 𝜙 if 𝜙 is true in some state in 𝜋. We assume future includes the
current, i.e, the first state in the path.
• 𝜋 ⊧ 𝐺𝜙 if 𝜙 is true in all states in 𝜋
• 𝜋 ⊧ 𝜙𝑈 𝜓. At some point 𝜓 becomes true, until then 𝜙 must be true. It
demands that 𝜓 is true at some point in the future. It does not demand
that 𝜙 is false after that point.
– I was happy until I got married does not imply that I must get un-
happy after I get married.
𝑀 , 𝑆 ⊧ 𝜙 iff for every path 𝜋 starting from S, we have 𝜋 ⊧ 𝜙. Examples:
• 𝑀 , 𝑆0 ⊧ 𝑝 ∧ 𝑞. True.
• 𝑀 , 𝑆0 ⊧ ¬𝑟. True.
• 𝑀 , 𝑆0 ⊧ 𝑋𝑟. True.
• 𝑀 , 𝑆0 ⊧ 𝑋(𝑞 ∧ 𝑟). False. If 𝑆0 goes next to 𝑆2 .
• 𝑀 , 𝑆0 ⊧ 𝐺(𝑝 ∧ ¬𝑟). True. p and r are never true together.
• 𝑀 , 𝑆2 ⊧ 𝐺𝑟. True. All paths starting from 𝑆2 stay on 𝑆2 .
• 𝑀 , 𝑆0 ⊧ 𝐹 (¬𝑞 ∧ 𝑟) → 𝐹 𝐺𝑟. True. LHS is true only for paths that go to
𝑆2 . Once we go to 𝑆2 , RHS is true.
• 𝑀 , 𝑆1 ⊧ 𝐹 (¬𝑞 ∧ 𝑟) → 𝐹 𝐺𝑟. True.
• 𝑀 , 𝑆2 ⊧ 𝐹 (¬𝑞 ∧ 𝑟) → 𝐹 𝐺𝑟. True.
• 𝑀 , 𝑆0 ⊧ 𝐺𝐹 𝑟 → 𝐺𝐹 𝑝. False. A path that loops infinitely over 𝑆2 satisfies
GFr but not GFp.
• 𝑀 , 𝑆0 ⊧ 𝐺𝐹 𝑝 → 𝐺𝐹 𝑟. True. LHS imply that the path go through 𝑆0
infinite times. RHS imply that the path go through either 𝑆1 or 𝑆2 infinite
times.
Lecture 3
Subformulas of 𝑝𝑈 (𝑞𝑈 𝑟) * p, q, r, 𝑞𝑈 𝑟
Other operators that could be derived from 𝑈 : * 𝑊 is a weak until: 𝜙𝑈 𝜓
demands that 𝜓 should be true at some point in the future. 𝑊 does not demand
𝜓 to ever be true. * 𝑅 is release. 𝜙𝑅𝜓 means 𝜓 remains true. When 𝜙 becomes
true, it releases 𝜙 so 𝜙 can become false. * In 𝑅 both become true in one state.
In 𝑈 and 𝑊 , they need not be true ever in the same state.
It is impossible to get to a state where started holds, but ready does not
hold. * 𝐺¬(𝑠𝑡𝑎𝑟𝑡𝑒𝑑 ∧ ¬𝑟𝑒𝑎𝑑𝑦)
It is possible to get to a state where started holds, but ready does not hold.
* ¬𝐺¬(𝑠𝑡𝑎𝑟𝑡𝑒𝑑 ∧ ¬𝑟𝑒𝑎𝑑𝑦) is not correct as it is saying that on all paths it is
possible to get to such a state
5
In LTL, there is no way to specify “there exists a path”. In LTL, it is possible
̸ ̸
̸ * 𝑀 , 𝑆 ⊧𝜙
that: * 𝑀 , 𝑆0 ⊧𝜙 0
6
• Safety property: 𝐺(¬𝑝1.𝑠 = 𝑐 ∨ ¬𝑝2.𝑠 = 𝑐). Both processes are never
in critical sections at the same time.
• Liveness property: If someone is trying to get into critical section, they
will be able to get into it at some point in the future.
Collective system:
graph LR
n1n2 --> n1t2
n1n2 --> t1n2
n1t2 --> t1t2
t1n2 --> t1t2
t1t2 --> c1t2
t1t2 --> t1c2
t1n2 --> c1n2
n1t2 --> n1c2
c1n2 --> c1t2
c1n2 --> n1n2
n1c2 --> t1c2
n1c2 --> n1n2
t1c2 --> t1n2
c1t2 --> n1t2
Safety holds. Liveness does not hold since process 1 can keep getting the lock
forever.
NuSMV model checking tool!
Lecture 4
You can get safety cheaply. Important to specify liveness.
• Non-blocking: a process can always request.
• No strict sequencing: process need not enter their critical section in
strict sequence. Process 0 may want to enter more often than Process 3.
• Safety: 𝐺¬(𝑐1 ∧ 𝑐2 )
• Liveness: 𝐺(𝑡1 → 𝐹 𝑐1 )
• Non-blocking: can not talk about existence of a path. If you are in a state
where 𝑛1 is true, then there is a path where 𝑡1 is true.
• No strict sequence: 𝐺(𝑐1 → 𝑐1 𝑊 (¬𝑐1 ∧ ¬𝑐1 𝑊 𝑐2))
If 𝑐1 is in a critical section, then it remains in its critical section forever. When 𝑐1
comes out of its critical section, either it stays out of its critical section forever.
No strict sequence is a purely ∃ property, so we could negate it.
7
Non-blocking is a ∀∃ property. From all states, there exists a way to get to
trying to acquire.
Non-blocking and no strict sequencing properties are added to kill trivial proto-
cols that satisfy safety/liveness, such as force strict sequence on acquires.
simple.smv state diagram:
stateDiagram
rb: request, busy
rr: request, ready
nb: no request, busy
nr: no request, ready
rr --> rb
rr --> nb
nr --> nr
nr --> nb
nr --> rb
nr --> rr
Mutex protocol issue: we treated t1t2 as one state. We forgot who was trying
first out of the two processes.
Fix: remember the turn. Who was trying first?
HW: NuSMV does not allow W. As an exercise, verify no strict se-
quencing using NuSMV. You want to find a path that allows strict
sequencing. Manually verify non-blocking property or come up with
counter example.
Lecture 5
LTL has an implicit universal quantification over all paths starting in a state.
We could not express existential path properties that we could not specify in
LTL.
8
Syntax 𝜙 ∶= ⊤|⊥|𝑝|¬𝜙|𝜙∧𝜓|𝜙∨𝜓|𝜙 → 𝜓|𝐴𝑋𝜙|𝐸𝑋𝜙|𝐴𝐹 𝜙|𝐸𝐹 𝜙|𝐴𝐺𝜙|𝐸𝐺𝜙|𝐴[𝜙𝑈 𝜓]|𝐸[𝜙𝑈 𝜓]
You cannot have an X without an A or an E. Example: EGFr is not a proper
formula.
Precedence order * Unary operators bind tightly, ¬, 𝐴𝑋, .. * And/or * Implica-
tion * 𝐴𝑈 , 𝐸𝑈
𝐴𝐺𝑞 → 𝐸𝐺𝑟 is (𝐴𝐺𝑞) → (𝐸𝐺𝑟) and not 𝐴𝐺(𝑞 → 𝐸𝐺𝑟).
𝐴[𝐴𝑋¬𝑝𝑈 𝐸[𝐸𝑋(𝑝 ∧ 𝑞)𝑈 ¬𝑝]]. Build parse tree. Subformulas are again formulas
whose parse tree is a subtree the parse tree of the original formula.
9
• 𝑀 , 𝑠0 ⊧ 𝐸[(𝑝 ∧ 𝑞)𝑈 𝑟]. s0, s2, s2, ..
• 𝑀 , 𝑠0 ⊧ 𝐴[𝑝𝑈 𝑟]. No matter where we go, until holds
• 𝑀 , 𝑠0 ⊧ 𝐴𝐺[⊤ → 𝐸𝐹 𝐸𝐺𝑟]. For each state, there always exists a path
such that r is forever true.
• It is possible to get to a state where started holds, but ready does not
hold. 𝐸𝐹 (𝑠𝑡𝑎𝑟𝑡𝑒𝑑 ∧ ¬𝑟𝑒𝑎𝑑𝑦).
• It is impossible to get to a state where started holds, but ready does not
hold. ¬𝐸𝐹 (𝑠𝑡𝑎𝑟𝑡𝑒𝑑 ∧ ¬𝑟𝑒𝑎𝑑𝑦). Couldn’t do in LTL.
• for any reachable state, if a request occurs, then it will be eventually be
granted. 𝐴𝐺(𝑟𝑒𝑞𝑢𝑒𝑠𝑡 → 𝐴𝐹 𝑔𝑟𝑎𝑛𝑡𝑒𝑑).
• a certain process is enabled infinitely often on every computation path:
𝐴𝐺(𝐴𝐹 𝑒𝑛𝑎𝑏𝑙𝑒𝑑).
• on all paths, a certain process will eventually become permanently dead-
locked: 𝐴𝐹 (𝐴𝐺𝑑𝑒𝑎𝑑𝑙𝑜𝑐𝑘𝑒𝑑)
• If a process is enabled infinitely often, then it runs infinitely often. You
can express it in LTL, but you cannot express it in CTL. 𝐺𝐹 𝑒𝑛𝑎𝑏𝑙𝑒𝑑 →
𝐺𝐹 𝑟𝑢𝑛𝑠. LTL can talk about paths satisfy a certain property. In CTL,
we can not talk about paths that satisfy a certain property.
𝐴𝐺𝐴𝐹 𝑒𝑛𝑎𝑏𝑙𝑒𝑑 → 𝐴𝐺𝐴𝐹 𝑟𝑢𝑛𝑠 says that if in all paths enabled holds in-
finitely often then in all paths run holds infinitely often.
stateDiagram
direction LR
S0: p (S0)
S1: ~p (S1)
S2: p (S2)
[*] --> S0
S0 --> S0
S0 --> S1
S1 --> S2
S2 --> S2
• LTL: 𝐹 𝐺𝑝 holds
• CTL: 𝐴𝐹 𝐴𝐺𝑝 does not hold because in the infinite path S0, S0, S0, …
𝐴𝐺𝑝 never becomes true.
LTL: 𝐹 𝐺𝑝 is also not equivalent to CTL’s 𝐴𝐹 𝐸𝐺𝑝.
Lecture 6
• LTL: what atomic propisitions are true in a state and what is true about
all paths starting from here.
• CTL: we look at the entire tree of computation paths
10
Temporal operators: X, F, G, U. Path quantifiers: A, E.
We can look at each formula as either a state formula or as path formula.
𝜙 ∶= ⊤|⊥|𝑝|𝜙 ∧ 𝜓|𝜙 ∨ 𝜓|¬𝜙 can be thought of as state formulas. They are only
talking about a state.
Any state formula is a path formula. $�:= �| �_1 ��_2 | �_1 ��_2 | ¬�| X �| F �|
G �| �_1 U �_2 $
𝜙 ∶= 𝐴𝛼|𝐸𝛼 are also state formulas. So state and path formulas have recursive
definitions.
This logic is called CTL. Both CTL and LTL are subsets of CTL. In some sense,
LTL only allows state formulas with an implicit A in front of them. CTL does
not allow 𝛼 ∶= 𝜙|𝛼1 ∧ 𝛼2 |𝛼1 ∨ 𝛼2 |¬𝛼. It also does not allow nesting of path
modalities, i.e, you cannot have 𝐴𝑋𝐹 𝜙.
Not allowing boolean combinations of path is not a restriction.
𝐸(𝐹 𝑝 ∧ 𝐹 𝑞) ∶= 𝐸𝐹 (𝑝 ∧ 𝐸𝐹 𝑞) ∨ 𝐸𝐹 (𝑞 ∧ 𝐸𝐹 𝑝). p followed by q or vice versa.
Not allowing nesting of path formulas is a real restriction. We keeep this restric-
tion so that we can algorithmically verify models.
Example: * On all paths, a certain process will eventually become deadlocked
* CTL: `AF AG d`
* LTL: `F G d`
stateDiagram
direction LR
S0: d (S0)
S1: ~d (S1)
S2: d (S2)
[*] --> S0
S0 --> S0
S0 --> S1
S1 --> S2
S2 --> S2
LTL holds. CTL does not hold. 𝐴𝐹 𝐴𝐺𝑑 → 𝐹 𝐺𝑑. It may be possible to take
them both to CTL: 𝐴𝐹 𝐴𝐺𝑑 → 𝐴𝐹 𝐺𝑑. But, we may not have appropriate
derivation rules in CTL, so we might have to still give an argument based on
path enumeration.
Non-blocking: a process can always request to enter its critical section 𝐴𝐺(𝑛 →
𝐸𝐹 𝑡).
• 𝐿𝑇 𝐿 ⊄ 𝐶𝑇 𝐿,
• 𝐶𝑇 𝐿 ⊄ 𝐿𝑇 𝐿
• 𝐶𝑇 𝐿 ⊂ 𝐶𝑇 𝐿∗
11
• 𝐿𝑇 𝐿 ⊂ 𝐶𝑇 𝐿∗
• 𝐴𝐺𝐸𝐹 𝑝 ∈ 𝐶𝑇 𝐿, but there is no way to express this in LTL.
• 𝐸𝐺𝐹 𝑝 ∈ 𝐶𝑇 𝐿∗, but it is not expressible in CTL and not in LTL. There
exists a path with infinitely many ps.
12