Exercise 7: Testing and Continuous Integration

Question 1

Terminology for failure

Answer 1

• Failure: Any deviation of the observed behavior from the specified behavior
  
  • A crash is the informal term for some kind of failures
• Error (Erroneous state): The system is in a state such that further processing by the system can lead to a failure
• Fault: The mechanical or algorithmic cause of an error
  
  • Informally also called bug
• Validation: Activity of checking for deviations between the observed behavior of a system and its specified behavior

Question 2

How to deal with faults

Answer 2

• Fault avoidance
  
  • Use methodology to reduce complexity
  • Use configuration management to prevent inconsistency
  • Apply verification to prevent algorithmic faults
  • Use reviews to identify faults already in the design
• Fault detection
  
  • Testing: Provoke failures in a planned way
  • Debugging: Find and remove faults
  • Monitoring: Deliver information about state and unusual behavior => Used during debugging
• Fault tolerance
• • Exception handling, modular redundancy.

Question 3

Taxonomy for Fault Handling Techniques

Answer 3

Question 4

Types of testing

Answer 4

• Integration Testing:
  
  • Groups of subsystems (collection of subsystems) and eventually the entire system are tested
  • Carried out by developers
  • Goal: Test the interfaces of the subsystems
• Acceptance Testing:
  
  • Evaluates the system delivered by developers
  • Carried out by the client. May involve executing typical transactions on site on a trial basis
  • Goal: Demonstrate that the system meets the requirements and is ready to use
• Unit Testing
  
  • Individual components (class or subsystem) are tested
  • Carried out by developers
  • Goal: Confirm that the component or subsystem is correct and carries out the intended functionality
• System Testing
  
  • The entire system is tested
  • Carried out by developers
  • Goal: Determine if the system meets the requirements (functional and nonfunctional)

Question 5

Acceptance Testing

Answer 5

• Goal: Demonstrate that the system is ready for operational use
  
  • Many tests can be taken from integration and system testing
  • Choice of tests is made by the client
  • Acceptance test is performed by the client, not by the developer
• Alpha test:
  
  • Client uses the software at the developer’s environment
  • Software used in a controlled setting, with the developer always ready to fix bugs
• Beta test:
  
  • Conducted at client’s environment (developer is not present)
  • Software gets a realistic workout in target environment.

Question 6

Motivation for Continuous Integration and Continuous Delivery

Answer 6

• Risk #1: The later acceptance testing occurs in a project, the bigger is the risk that unexpected faults occur
• Risk #2: The higher the complexity of the software system, the more difficult it is to integrate its components
• Continuous Integration addresses these risks by building software systems as early as possible and frequently
• Continuous Delivery addresses these risks by delivering software systems as early as possible and frequently

Question 7

Definition: Continuous Integration

Answer 7

• Continuous Integration: A software development technique where members of a team integrate their work frequently. Usually each person integrates at least daily, leading to multiple integrations per day.
• Each integration is verified by an automated build which includes the execution of tests – regression tests – to detect integration errors as quickly as possible.
• Continuous Delivery: topic of next week’s exercise

Question 8

Additional advantages

Answer 8

• There is always an executable version of the system
• Developers and managers have a good overview of the project status
• Automatic regression testing
  
  • Definition: Regression testing is a type of software testing that seeks to uncover new software bugs, that might have been introduced to the system after change has been made

Question 9

Continuous Integration Workflow

Answer 9