Basic principles of black box testing
- Test-case design based on definition of spec (reqs) of software, instead of structure (source code)
- Complete coverage can be applied to black box testing (BBT)
- Rigorous specs have various benefits & help BBT (e.g. categorise inputs & derive expected outputs)
- Specs help test-case generation & test oracles
- Most faults caused by interactions of max 2 factors
- Want to reduce combos to manageable size while still providing effective fault detection power to test suite
What is the purpose of equivalence class partitioning?
Have sense of complete testing and avoid test redundancy
What are equivalence classes?
Partitions of input set in which input data has same effect on program (same output)
What is the purpose of disjoint classes?
Avoids redundancy
What is the process of WECT?
Weak equivalence class partioning
- Choosing 1 var value form each equiv class such that all classes covered - not all cartesian combos.
- Num test cases = max(|A|,|B|,|C|)
- Num WECT test cases = max partition size (D)
What is the process of SECT?
Strong equivalence class partioning
- Based on cartesian product of partition subsets (test all interactions of all equiv classes) - 1 item per partition - all cartesian combos
- Num test cases = |A| x |B| x |C|
- Num SECT test cases = partition size (D) x partition size (M) x partition size (Y)
What is a cartesian combination?
Cross join
What is the process of equivalence class partioning?
- ID input domain (of method/function params)
- Start from reqs
- Draw equiv classes diagram for each param
- Combine equiv classes, using WECT or SECT & spec expected output per test case
What is pair-wise and n-way testing?
- 2 way testing
- test cases = max x 2nd max
- n-way testing is generalisation of pair-wise testing
What is BVA?
Boundary-value analysis
- Behaviour of program not always similar for all input values of equiv class, esp at boundaries between classes e.g. off-by-one error
- Set values for input var at min, just above min, nominal (typical) value, just below max & at max
- Usual strategy for all input vars is holding values of all but 1 var at nominal values, let 1 var assume extreme value
- Function with n vars requires 4n + 1 test cases
What are the benefits of BVA?
- Works well with vars representing bounded physical quantities
- Extendable to robustness testing
What is WCT?
Worst-case testing
- Boundary-value assumes failures usually originate from 1 fault
- Cartesian product of {min, min+, nom, max-, max}
- 5n test cases (n number vars)
- Good strategy when physical vars have numerous interactions & failure costly
- Robust worst cases testing has 7n test cases
What is robustness testing?
- Robustness is degree to which system operates correctly in presence of exceptional inputs/stressful environmental conditions
- Goal of robustness testing to develop test cases & environments where robustness assessed
- Also look at behaviour of system when var extremes exceeded with value slightly above max & slightly less than min
- Function with n vars requires 6n + 1 test cases
What are the benenfits of MBT?
- Improved test-case design practices
- Detecting real fault during regression testing
- Measuring code coverage (client & server side)
- Made work of test engineers more interesting & organised
- Easier to see big picture with model in front of them & being directly executable
Fault Taxonomy
- Missing/incorrect transition by implementation of valid state based on correct input (event)
- Missing/incorrect output (action) by implementation based on correct input & transition
- Sneak path (extra transition) - implementation accepts illegal/unspecific by state input
- Illegal input failure - implementation fails to handle illegal message correctly (incorrect output, state corrupted)
- Corrupt state - based on correct input, implementation output invalid state
Transition coverage for MBT
Needs/requires every transition in state model to be taken min 1x
Full predicate coverage for MBT
Requires each clause in each predicate (composite condition) of transitions to be tested true/false min 1x
Transition-pair coverage for MBT
Test for invalid sequences of transitions. For each pair of adjacent transitions (states), test suite should contain test case traversing pair of transitions in sequence min 1x
Complete sequence coverage for MBT
All poss sequences of transitions on state chart diagram tested, usually v expensive
What are the benefits of decision tables?
- Helps express test reqs in directly usable form
- Easy to understand & review by peers
- Support systematic derivation of test cases
- Support automated/manual gen of test cases
- Particular response (output) selected for testing by evaluating many related conditions
- Ideal for describing situations in which num of combos of actions taken under varying condition sets
What is the condition section in decision tables?
Section listing conditions & combos of them
- Express relationships among decision vars
What is the action/output section in decision tables?
Section lists responses to be produced when corresponding combos of conditions true
- Actions independent of input order & order in which conditions evaluated
Decision tables
- Kind of a type of MBT
- 1 test case from each column of corresponding truth table
Cause-effect graphs
Type of MBT
- Graphical technique helping derive decision tables
- Aimed at supporting interaction with domain experts & reverse engineering of specs for testing purposes
- ID causes (conditions on inputs, stimuli) & effects (outputs, changes in system state)
- Causes must be stated in boolean manner
- Can be used to explicitly specify constraints (environmental/external) on causes & effects
- Can help select more signif subset of input-output combos & build smaller decision tables
