Specification and Verification of Programs

The common interface

Specification Yes
Program Verifier
Implementation
No
The same promise
Model checking

M Ψ
Ψ: Specification
M:
M satisfies Ψ ? Implementation

no yes
counter the system is correct!
example

M satisfies
Ψ
Automated Program Verification: 3
methods

 Automated Theorem Proving

 Constraint Solving

 Model Checking
A slightly more complex example
Program P

Requirement: g always terminates

The way you are possibly thinking…
• In the case where y <= 0, execution jumps to the
end of the function g. That’s easy case.

• If, in every iteration of the loop, the value of the

variable x decreases, then eventually, the loop
condition x > y will fail, and the end of g will
be reached

• The value of x always decreases only if y is always

positive, because only then does the update
to x (i.e., x = x - y) decrease x. But y’s positivity is
R: An underflow never happens on x
established by the conditional expression,
so x always decreases.
Automated Program Verification in
action
 If you carried out the three steps yourself, you now have a very
intuitive view of the type of thinking an automated-reasoning tool
is performing on our behalf when reasoning about a computer
program.

 There are many nitty-gritty details that the tools have to face (e.g.,
heaps, stacks, strings, pointer arithmetic, recursion, concurrency,
callbacks, etc.),

 Decades of research papers on techniques for handling these and

other topics, along with various practical tools that put these ideas
to work.
How an automated prover will do it
 Automated-reasoning tools are usually working through the three
steps above on our behalf:
 Item 1 is reasoning about the program’s control structure.
 Item 2 is reasoning about what is eventually true within the
program.
 Item 3 is reasoning about what is always true in the program.

 Generic viewpoint allows us to use the same techniques we use to

reason about C or Python code to answer questions.
The way it all started
 Pioneered by the seminal paper by Tony
Hoare in 1969.

 Deductive Verification to use a set of

inference rules and axioms and apply
them to construct the proof of
correctness.

 Several semi-automated tools (also called

Theorem provers) such as Coq, Isabelle,
ACL2, Lean etc. are widely used today
following this mode of verification
Can they solve all problems?
 Imagine that we have an automated-reasoning API, called
terminates, that returns “yes” if a C function always terminates or
“no” when the function could execute forever.

terminates(g) should return true.

terminates(g2) should return false
g2(5, 0) will never terminate.
What’s the right answer for
terminates(h)?
 Recursive
 The answer cannot be "yes". It also cannot be "no".
 Imagine that terminates(h) were to return "yes". If you read the code of h,
you’ll see that in this case, the function does not terminate because of the
conditional statement in the code of h that will execute the infinite loop
while(1){}.
 In this case, the terminates(h) answer would be wrong, because h is defined
recursively, calling terminates on itself.
 Similarly, if terminates(h) were to return "no", then h would in fact terminate
and return control to its caller, because the if case of the conditional
statement is not met, and there is no else branch.
 Again, the answer would be wrong.
 This is why the “Don’t know” answer is actually unavoidable in this case.
Undecidability of Verification
 Automated verification of programs written in a Turing-complete
language is undecidable

 Rice’s Theorem: There is no Turing machine that can decide

whether the language accepted by a Turing machine has a non-
trivial property

 In practice, automated verifiers either restrict the space of

programs, or the space of specifications
 Even then, verification is computationally expensive
Notes
 The program h is a variation of examples given in Turing’s
famous 1936 paper on decidability and
Gödel’s incompleteness theorems from 1931.

 These papers tell us that problems like the halting problem

cannot be “solved”.

 Solved -> that the solution procedure itself always terminates

and answers either “yes” or “no” but never “Don’t know”.

 Evidently, no verifier can solve such problems!!

To contribute to this area: Need to
know
 Logic: a formal and mechanical system for defining what is true and
untrue. Propositional logic + first-order logic.

 Theorems: A true statement in logic. Example: the four-color theorem.

 Deduction and Proofs: A valid argument in logic of a theorem.

Example: Gonthier's proof of the four-color theorem.

 Inside a mechanical theorem prover: A semi-automated-reasoning

tool that checks a machine-readable expression of a proof often
written down by a human.
 Often require human guidance.
Automated Program Verification: 3
methods

 Automated Theorem Proving

 Constraint Solving

 Model Checking
Program Verification using Constraint
Solving
 Input: Program P; Requirement R;

 Procedure:
 Translate P to a collection of constraints
 Add a constraint for negation of R (Requirement)
 Use constraint solvers to solve the constraint set
 Completely automated

 Scales well: Modern constraint solvers employ powerful

techniques
 A simple Example
Function f (input x, output y)
y = x + 1;
z = 3 * y;
w = 2 * z;
EndFunction
Requirement: For all positive values of x, we always have a positive w at the end
Formal VerificationQuery:
(x > 0) AND (y = x + 1) AND (z = 3 * y) AND (w = 2 * z) AND (w <= 0) + Domain constraints

SAT
For any +ve input x,
Constraint
does the program ever For no +ve input x,
Solver
produce a w a witness of w <= 0
which is <= 0? is possible
UNSAT
A not so simple Example
Function f (input x, output y)
y = x + 1;
if (y < 0) then
z = 3 * y;if-then-else (y<0, 3*y, y – 2)
else z = ite(y<0, 3*y,y-2)
z = y - 2;
Endfunction
Requirement: For all positive values of x, we always have a positive z at the end
Formal Verification Query:
(x > 0) AND (y = x + 1) AND [{(y < 0)  (z = 3 * y)} OR {(y >= 0)  (z = y – 2)}] AND (z <= 0) + Domain
constraints

For any +ve input x, x=1

SAT
does the program ever (Counterexample)
produce a z Constraint
which is <=0? Solver

UNSAT
Using CSP for certification
 Given program A and requirement R, does A satisfy R?
 Option 1: prove that R holds
 Option 2: provide a counter-example showing that it does not

 Create a constraint model of the system

 Check for correctness of the model against requirements
 Completely automated procedure
 Routinely applied in the context of hardware / software programs

20
CSP for Neural Networks
 Can we prove that a given NN satisfies desired properties?

21
CSP for Neural Networks
Input Space I Output Space O
N

y0 Q
x0

For a neural network N : I  O, an input property P(I) and an output

property Q(O), does there exist an input x0 with output y0 = N(x0), such
that x0 satisfies P and y0 satisfies Q?
• P(I) characterizes a pre-condition on the inputs
• Q(O) characterizes a set of undesired / unsafe behaviors
• Negative answer (UNSAT) indicates property holds
• Positive answer (SAT) includes a counterexample
22
CSP for Neural Networks
 Step 1:
 Create a mathematical model of the neural network
Neural Networks (NNs)
 Steps:
1. Weighted sum
2. Activation function
Rectified Linear Units (ReLUs)
A Simple Example
Modeling Networks
ReLU

ReLU
Certifying Neural Networks
 Step 2:
 Formally specify the properties of the system
A Simple Example
Encoding Networks
ReLU

ReLU
The ACAS Xu Example
 Whenever the intruder is distant, network always answers Clear-of-Conflict (CoC)
 5 output labels y0, y1,y2,y3,y4
 For all distant inputs, score of y0 is always the highest

 P(X): Distance ≥ 40000

 Q(Y): Negation of requirement

 For some input Distance ≥ 40000, score of y0 is NOT the highest
 (y0 ≤ y1) OR (y0 ≤ y2) OR (y0 ≤ y3) OR (y0 ≤ y4)

 A collection of 10 such adivsories

31
The ACAS Xu Example
P(X): Distance ≥ 40000

Other input
constraints
Distance ≥
40000
SAT
Constraint
Solver

(y0 ≤ y1) OR (y0 ≤ y2) OR UNSAT

(y0 ≤ y3) OR (y0 ≤ y4)

Negative answer (UNSAT) indicates system behaves as expected

Adversarial Robustness

Pixel matrix
0.1 0.0 0.8..
Input Image 0.5 0.6 0.7.. 28 X 28
(x0)
…

Y0
Y1
Y2
Y3
Y4

Score of yi is the highest

Adversarial Robustness
Pixel matrix

Input Image with

Any 28X28 image
Reference Image slight change but whose pixel values
(x0) very close to x0 are within   of x0

Y0
Y1
Y2
Y3
Y4

Score of yi is the highest

The same setup works
P(X): Image within the  neighbourhood of x0

Other input
constraints
P(X)
SAT
Constraint
Solver

(y0(y0
≤≤y1)y1)
ORV (y0 ≤ y2) OR
V UNSAT
(y0(y0
≤≤y3)y3)
ORV (y0 ≤ y4)

Negative answer (UNSAT) indicates system is adversarially robust for

the given value of 
The same setup works
 Want to ensure: for a given input x0 and a given amount of noise
, classification remains the same

 P(X): ||x - x0||L ≤ 

 Equivalent to: i (- ≤ x[i] - x0[i] ≤ )

 Q(Y):
 OR (y[i ] ≤ y[i]), where y[i ] is the desired label
i 0 0

 NEGATIVE answer means the system is adversarially robust for

the given value of 
36
How it all works
 Create a CSP model of the neural network
 Weighted Sum
 Activation Functions

 Formally specify the properties of the system

 Prove that the system satisfies the properties

 Need a proof engine that can reason about activation functions

37
All is well?

38
Constraint Solvers
 SAT was the original NP-Complete Problem, but decades of research
in SAT solving has led to efficient solvers which can easily handle
thousands of variables and millions of clauses.

 This was complemented by rise of Satisfiability Modulo Theories

(SMT), targeting first-order logic.
 Specialized Algorithms for SMT for various theories such as Linear Arithmetic, Bit
Vector Logic, etc. •
 Z3, CVC4, …

 A number of verifiers (deductive, model-checking, combination…) use

constraint solvers in the backend.
 CBMC, Dafny, Seahorn, VCC, Sage…
To contribute to this area: Need to
know
 How to translate a program as a collection of constraints

 How to write requirements formally

 How constraint solvers work

 Mostly intractable problems
 Lots of ML algorithms working behind today
Automated Program Verification: 3
methods

 Constraint Solving

 Automated Theorem Proving

 Model Checking
Model Checking
Model Checkers
Model Checking
 For finite-state systems, Model Checking reduces the verification
problem to the reachability problem in transition systems.
 Applicable if system generates (finite) behavioral model

 Pioneered by Clarke, Emerson, Quielle and Sifakis in the 1980s

 Initially used for concurrent program

verification and protocol analysis.

 Used by IBM, Intel, Cadence, Synopsys,

 Microsoft, Nvidia, Texas Instruments, Qualcomm…
A motivating example

Program P

Bad States: The value of x is less than 0 or greater than 200

Codes as transition system models
Does this help?

NO
And the answer is…
To contribute to this area: Need to
know
 How to extract transition system models from implementations
 Software code, hardware code, concurrent systems…

 How to express requirements formally

 Builds on logic

 How model checkers work

 Graph algorithms, automata theory
References
 https://vimeo.com/81641895

 https://www.amazon.science/blog/a-gentle-introduction-to-automated-reasoning

 https://www.quantamagazine.org/computer-scientists-attempt-to-corner-the-collatz-conjecture-20200826/

 Principles of Model Checking by J.P. Katoen (Read Chapter 1)

 Model Checking by Doron A. Peled, Orna Grumberg, Helmut Veith, Daniel Kroening, Edmund M. Clarke
About the course
 Requires knowledge in algorithms + preliminary automata theory

 Requires programming skills in C / Python

 Is heavy on programming assignments

Coming up next
 Automata theory based model checking

 A gentle introduction to finite automata