Static Testing is
- Carried out without executing the SUT
- Complements (sometimes drives) dynamic testing
- Can find issues earlier
Error
Human action that produces an incorrect result
Bug
Incorrect step, process or data definition in a program
Failure
The result of a fault ex. crash
Manual vs Automated Static Testing
Manual: Requirements testing, peer review, code walkthrough, code smell analysis…
Automated: Static code analysis, symbolic execution
Requirements Testing
- Complete: All requirements needed to specify the solution to the problem have been included.
- Correct: Each requirement is free from error
- Precise and clear: Each requirement is exact and not vague with single interpretation; easy to read
- Consistent: no requirement conflicts with another
- Relevant: Each item is relevant to the problem and its solution
- Testable: During the program, coding and acceptance testing it will be possible to determine whether the requirement has been satisfied.
- Feasible: The requirement can be implemented with available techniques, tools, resources and personnel, within the specified cost and schedule.
- Free of unwarranted design detail: Requirements should not provide the solution details but just state the requirement
- Manageable: requirements are expressed in such a way that each item can be changed without excessive impact on other items.
Peer Review
Buddy Checking, Walkthrough, Inspection
Buddy Checking
A person other than the code author informally reviews a piece of work. No collection of data needed. Difficult to put under managerial control. No use of checklists, therefore not repeatable. Inexpensive.
Walkthrough
Used to find defects and become familiar with material. The author presents the artefact to an audience who asks questions. Can break down into arguments. Minimal documentation and no prior preparation for the audience. Not easy
Inspection
To find defects and collect data. Doesn’t examine alternative solutions. To communicate important work product information.
- Formally structured and managed peer review process
- Involve a review team with clearly defined roles
- Specific data collected during inspections
- Reviewers check an artifact against an unambiguous set of inspection criteria for that type of artefact.
Data collection promotes process and quality improvements
Inspection Roles
- Moderator/Leader
- Author/Producer
- Reviewer/Reader
- Scribe
Automated Static Testing
Process of extracting info about a program from its source code or artefacts with intent of finding faults. Potential issues flagged.
The parser
- Transforms code into one or more internal representations
- Representations are internal abstractions of the source code
- Different types of representations facilitate different types of analyses.
Internal Representations
- Abstract a particular aspect of the program into a form more suitable for automated analysis.
- Different types of representations facilitate different types of analyses
- Some representations are produced directly by compilers.
Analysis
- Utilizes internal representations to draw conclusions
- Each analysis type seeks to answer one or more questions
Static vs Dynamic Analysis
(Don’t confuse this with static vs dynamic testing)
Static Analysis: Does not take program input into account. Results are applicable to all executions
Dynamic Analysis: Takes concrete inputs into account (typically one input), provides greater precision, result only applicable to that particular execution
Static Code Analysis Tools
- Findbugs
- PMD
- Checkstyle
- Sonar
Typical Inspection Process
- Entry
- Planning
- Kickoff Meeting
- Individual Checking
- Logging Meeting
- Edit
- Follow Up and Exit
Inspection - Entry
- Author of artefact requests inspection
- Artefact checked by inspection moderator to ensure entry criteria is met
- Primary purpose is to ensure inspection time is not wasted on artefacts that contain defects that the author should have found.
Inspection - Planning
The moderator determines the practical aspects of the inspection
- Determining size and composition of inspection team
- Determining goals of the inspection
- Determine the timing and purpose of the meetings
Inspection - Kickoff Meeting
Roles for the inspection team are assigned and clarified (moderator usually does this)
Documents, including artefact and source document, checklist and inspection rules are distributed and checked
Sometimes, the authors may be required to give a quick walkthrough of the artefact and its relation to other documentation.
Inspection - Individual Checking
- Final stage of preparation.
- Majority of defects found in inspection processes are found here.
- During this stage an individual reviewer reads the artefact and with guidance of the inspection checklist attempts to find defects.
- Reviewer should record issues found and make effort to determine what they consider the severity of a defect to be and classify it.
Inspection - Logging Meeting
- A planned and moderated meeting with the primary purpose of logging the issues found by the reviewers
- All reviewers should be given a chance to raise their issues as a a scribe logs the issue being raised.
- Important to only log issues once
- Mod should ensure that discussion on issues is kept to a minimum to maintain continuity of the meeting
- Some variations of this process include group defect finding as an activity at the end of the meeting
Inspection - Edit
- Editor (usually the author) is responsible for addressing all logged issues in the inspected artefact.
- The editor decides if something is a defect or not.
- All defects must be corrected.
- All non-defects should also be addressed in some way
