Module 6: Statistical Debugging

Question 1

why do bugs still escape in-house testing and analysis tools?

Answer 1

1. dynamic analysis is unsound
2. static analysis is incomplete
3. there are limited resources (time, money, people)

Question 2

monitor deployed code:

Answer 2

online: collect info from user runs
offline: company analyzes it

Question 3

two key questions

Answer 3

how do we get it?

what do we do with it?

Question 4

practical challenges

Answer 4

1. complex systems - millions of lines of code / mix of controlled and uncontrolled code
2. remote monitoring constraints
3. incomplete information (limits of performance overhead & private and security)

Question 5

approach to statistical debugging

Answer 5

1. guess which behaviors are “potentially interesting”
   - -> compile time instrumentation of program
2. collect sparse, fair subset of these behaviors
   - -> generic sampling framework
   - -> feedback profile + outcome label for each run
3. Analyze behavioral changes in successful versus failing runs to find bugs
   - -> statistical debugging

Question 6

how do we decide to examine or ignore each instrumented site?

Answer 6

randomly
independently
dynamically

for fairness and accurate picture or rare events

Question 7

finding causes of bugs

Answer 7

Question 8

tracking context

Answer 8

Question 9

a useful measure: increase()

Answer 9