Lecture 11 - Formal Methods

Question 1

How bad can bugs be?

Answer 1

REFER TO NOTES FOR EXAMPELS

Question 2

What can we do about bugs?

Answer 2

REFER TO NOTES FOR THE EXAMPLE

Question 3

What are formal methods?

Answer 3

Formal methods are a collection of methods by which we can take a computer programs and a set of properties that we know should hold of it and obtain strong guarantees that the program satisfies those properties.

Question 4

What is formal verification?

Answer 4

All software is really very complicated mathematical functions.

1. Make a mathematical model of our software
   
   • ideally automatically from your program
2. Write down the property that you want to hold
   
   • using some form of logic (e.g. predicate logic)
3. Proving that the model satisfies the property
   
   • either done manually or automatically or some form of both.

Question 5

What is the formal verification spectrum?

Answer 5

Formal methods exists broadly on the following spectrum.
- 1. Testing – no property, no proof
○ Not property, not proof, just random test case or values
- 2. Property-based testing – property but no proof
○ Write down your property you want to hold, computer will generate will generate the tests, but no proofs (might find bugs in manual testing)
- 3. Model checking – property and unknown proof
○ Write down a property (what you want your program to hold) and the computer will automatically go away and try to find that proof that satisfies that property, but it wont tell you the proof (tell you yes or no and trust the computer that it works)
- 4. Interactive Theorem Provers – property and known proof

Question 6

What are the steps to property based testing?

Answer 6

1. User: Define a property P(x).
2. User: Choose a strategy for generating values for x.
3. Library: Use the strategy to generate x1 , x2 , … and test P(x1 ), P(x2 ), …
   
   • The strategy may use the results of the previous tests to inform the generation of the next test.
4. Library: If a counter-example xi is found, automatically shrink xi to be as simple as possible, and then fail the test.

Question 7

What is Property-based testing?

Answer 7

Key idea:
- You write down the property.
- The computer generates test cases automatically

Question 8

What are the pros and cons of property based testing?

Answer 8

• Pros:
  ○ 1. Write a single test and test many values at the same time.
  ○ 2. Intelligently test a much larger range of values and find many more bugs
  ○ 3. Often quicker to write than normal tests.
  
  • Cons:
    ○ 1. For complex inputs, defining a strategy for generating data can be complicated.
    ○ 2. Still no hard guarantees

Question 9

What are Model-checking tools?

Answer 9

Key idea:
- You write down the property
- The computer proves it automatically

Question 10

How does model checking work?

Answer 10

Procedure for a domain-specific tool:
- 1. User: writes down their property in a high-level language.
- 2. Tool: compiles down the property to a set of SMTLIB queries.
- 3. Tool: calls an SMT solver to answer the queries.
- 4. Tool: converts any counter-example found back into a form understandable by the user.

Question 10

What are the pros and cons of model-checking?

Answer 10

• Pros:
  ○ 1. Formal guarantee of correctness.
  ○ 2. (Sometimes) don’t have to alter your program.
  
  • Cons:
    ○ 1. Cannot prove more complicated properties
    § see CITS2211 for non-computable problems, e.g. the halting problem.
    ○ 2. Sometimes you are forced to rewrite your program to make life easier for the model checker.
    ○ 3. A counter-example doesn’t immediately tell you why your program has gone wrong.

Question 11

What is interactive theorem provers?

Answer 11

Key idea:
- You provide the proof.
- The computer checks it.

Question 12

What is the problem and solution to interactive theorem provers?

Answer 12

Problem: Standard programming languages not designed to write proofs!
Solution: Custom program languages called theorem provers in which you can write both proofs

Question 13

What are the pros and cons of interactive theorem provers?

Answer 13

• Pros:
  ○ 1. Formal guarantee of correctness.
  ○ 2. Can represent pretty much any proof or argument.
  ○ 3. Can be used to prove mathematical theorems as well!
  
  • Cons:
    ○ 1. Writing down a correct proof is 100 times more time-consuming than writing the program!
    ○ 2. You must write your program in specialised languages.

Question 14

Advice on using formal methods

Answer 14

REFER TO NOTES

Question 15

What are the limitations of formal methods?

Answer 15

• All these methods check whether the program obeys your property
  
  • None of them guarantee that the property itself is correct

Question 16

How to Uses of LLMs in Formal method?

Answer 16

What not to do:
- Ask the LLM “Is this code correct?”

Hot research fields:
- Using LLMs to generate formal properties from human text.
- Automatic program repair based on formal properties.
- Using LLMs as generating strategies in property-based testing.

Question 17

What should the LLM do?

Answer 17

When you ask an LLM to “generate code that does X”, the LLM should:
1. Translate X into a formal specification.
2. Generate code that does X.
3. Generate a proof that the code satisfies the specification.
4. Give the proof to an Interactive Theorem Prover to check.
5. (Optional but recommended!) The user checks that the specification means X.