1
Q
4.1 Selection of Test Automation Approach and Planning of Deployment
Selection of Test Automation Approach and Planning of Deployment
A
Two main activities involved on the implementation and rollout of a TAS:
- Pilot
- identify a suitable project
- plan the pilot
- Conduct the pilot
- Evaluate the pilot - Deployment
- identify initial project(s)
- deploy the TAS in the selected projects
- monitor and evaluate the TAS in projects after a pre-defined period
- rollout to the rest of the organization/projects
2
Q
4.1.1 Pilot Project
Pilot Project
A
- Tool implementation typically starts with a pilot project
- The aim of the pilot project is to ensure that the TAS can be used to achieve the planned benefits
3
Q
4.1.1 Pilot Project
Pilot Project Objectives
A
- Learn about TAS
- See how TAS fits with organization, processes, procedures, tools. Identify how they might need to change
- Design the automation interface for testers
- Decide on standard ways of using, managing, storing and maintaining the TAS and the test assets
- Identify metrics and measurement methods to monitor test automation in use, including usability, maintainability and expandability
- Assess whether the benefits can be achieved at reasonable cost.
- Determine what skils are required (available/missing)
4
Q
4.1.1 Pilot Project
Pilot Project Process
A
- Identify a suitable project
- Plan the pilot
- Conduct the pilot
- Evaluate the pilot
5
Q
4.1.1 Pilot Project
Identify a suitable project
A
- Avoid critical project
- Avoid trivial project
- Involve necessary stakeholders in the selection process
- Good reference for other projects of organization
6
Q
4.1.1 Pilot Project
Plan the pilot
A
- Have a dedicated team members
- Treat as development project
- Involve developers in the deployment
- Management commitment
7
Q
4.1.1 Pilot Project
Conduct the Pilot
A
Perform the pilot of the deployment and pay attention to the following questions:
- Does the TAS provide the functionality as expected (and promised by the vendor)?
- if not, this needs to be addressed as son as possible
- when the TAS is developed in-house the corresponding developers need to assist in the deployment by providing any missing functionality - Do the TAS and the existing process support each other?
- if not, they need to be alligned
8
Q
4.1.1 Pilot Project
Evaluate the pilot
A
- Use all stakeholders for evaluation
- Evaluate against clear requirements and objective criteria
- Decide on the type of the valuation:
- formative
- summative
- process
- outcomes
- impact
9
Q
4.1.2 Deployment
Company-wide Deployment
Part I
A
- Once the pilot has been assessed, the TAS should only be deployed to the rest of the department/organization if the pilot has been deemed succcessful
- Rollout should be undertaken incrementally and be well-managed
10
Q
4.1.2 Deployment
Company-wide Deployment
Part II
A
Success factors for deployment:
- Incremental rollout
- Adapting and improving processes to fit with the use of the TAS
- Providing training and coaching/mentoring for new users
- Defining usage guidelines
- Implementing a way to gather information about the actual use
- Monitoring TAS use, benefits and costs
- Providing support for the test and development teams for a given TAS
- Gathering lessons learned from all teams
- identifying and implementing improvements
11
Q
4.1.2 Deployment
An Incremental Rollout
A
Perform the rollout to the rest of the organization in steps, in increments:
- Support to the new users in “waves”
- Usage of the TAS to increase in steps
- Possible bottlenecks can be identified and solved before they become real problems
- Licenses can be added when necessary
12
Q
4.1.2 Deployment
Adapting and Improving Processes
A
- When different users use the TAS, different processes come in touch with the TAS
- It may be needed:
- to adapt and improve processes to fit with the use of the TAS, or
- the TAS may need (small) adaptations to the processes
13
Q
4.1.2 Deployment
Providing Training and Coaching / Mentoring
A
- New users need training and coaching in the use of the new TAS
- Training / workshops should be provided to the users before they actually use the TAS
14
Q
4.1.2 Deployment
Defining Usage Guidelines
A
- For the usage of the TAS it is possible to write:
- guidelines
- checklists
- FAQs - This can prevent extensive support questions
15
Q
4.1.2 Deployment
Gather Information About the Actual Use
A
- There should be an automated way to gather information about the actual usage of the TAS
- Ideally not only the ussage itself, but also what parts of the TAS (certain functionalities) are being used
- In this way, the usage of the TAS can be monitored easily
16
Q
4.1.2 Deployment
Monitoring TAS Use, Benefits and Costs
A
- Monitoring the usage of the TAS over a certain period of time indicates whether the TAS is indeed used
- Monitor test automation efficiency (e.g. how much time has been saved)
- Monitor other test automation attributes set as objectives for the TAS (i.e. maintainability, reliability, usability)
17
Q
4.1.2 Deployment
Gathering Lessons Learned
A
- Gather lessons learned from all teams
- Perform evaluation / retrospective meetings with the different teams that use the TAS
- In this way, lessons learned can be identified
- The teams wil feel that their input is necessary and wanted to improve the usage of the TAS
18
Q
4.1.2 Deployment
Identifying and Implementing Improvements
A
- Based on the feedback of the team and the monitoring of the TAS, identify and implement steps for improvement
- Communicate these findings clearly to the stakeholders
19
Q
4.1.3 Deployment of the TAS within the Sofware Lifecycle
Deployment of the TAS within the Sofware Lifecycle
A
- The deployment of a TAS depends greatly on the phase of development of the software project which will be tested by the TAS
- Usually, a new TAS or a new version of it, is deployed either in the beggining of the project, or when reaching a milestone, such as code freeze or the end of a sprint
- Deployment activities, with all the testing and modifications involved, require time and effort
20
Q
4.2 Risk Assesment and Mitigation Strategies
Risk Assesment and Mitigation Strategies
Part I
A
Technical issues can lead to product - or project risks
Typical techincal issues include:
- too much abstraction can lead to difficulty in understanding what really happens (e.g. with keywords)
- in data-driven solution data tables can become too large / complex / cumbersome
- dependency of the TAS to use certaion operating system libraries or other components that may not be available in all the target environments of the SUT
21
Q
4.2 Risk Assesment and Mitigation Strategies
Risk Assesment and Mitigation Strategies
Part II
A
- Typical deployment project risks include:
- Staffing issues: getting the right people to maintain the code base may be difficult
- New SUT deliverables may cause the TAS to operate incorrectly
- Delays in introduction automation
- Delays in updating TAS based on the changes done to SUT
- The TAS cannot capture the (non-standard) objects it is intended to track
22
Q
4.2 Risk Assesment and Mitigation Strategies
Risk Assesment and Mitigation Strategies
Part III
A
Potential failure points of the TAS project include:
- Migration to a different environment
- Deployment to the target environment
- New delivery from development
23
Q
4.2 Risk Assesment and Mitigation Strategies
Risk Assesment and Mitigation Strategies
Part IV
A
- There are a number of risk mitigation strategies that can be employed to deal with these risk areas
- The TAS has a software lifecycle of its own, whether it is in-house developed or an acquired solution
- One thing to remeber is that the TAS, like any other software, needs to be under version control and its features documented
- Otherwise, it becomes very difficult to deploy different parts of it and make them work together, or work in certain environments
24
Q
4.2 Risk Assesment and Mitigation Strategies
Risk Assesment and Mitigation Strategies
Part V
A
There are two distinct cases when deploying a TAS:
- Initial deployment
- Maintenance deployment - TAS already exists and needs to be updated
Before starting with the first deployment of a TAS, it is important to be sure that:
- it can run in its own environment
- it is isolated from random changes
- test cases can be updated and managed
Both the TAS and its infrastructure must be maintained
25
# 4.2 Risk Assesment and Mitigation Strategies
First time deployment
| Part I
The following steps are needed:
1. Define the infrastructure in which the TAS will run
2. Create the infrastructure for the TAS
3. Create a procedure for maintaining the TAS and its infrastructure
4. Create a procedure for maintaining the test suite that the TAS will execute
26
# 4.2 Risk Assesment and Mitigation Strategies
First time deployment
| Part II
The risks related to first time deployment include:
* Total execution time of the test suite may be longer than the planned execution time for the test cycle
- in this case it is important to make sure that the test suite gets enough time to be executed entirely before the next scheduled test cycle begins
* Installation and configuration issues with test environment exist
- in general, the TAS needs an effective way to setup needed preconditions for the automated test cases within the test environment
27
# 4.2 Risk Assesment and Mitigation Strategies
Maintenance Deployments
| Part I
* The TAS itself needs to evolve, and the updates for it have to be deployed into production
* Before deploying an updated version of the TAS into production, it needs to be tested like any other software
- to check the new functionality
- to verify that the test suite can be run on the updated TAS
- that reports can be sent
- that there are no performance issues or other functional regressions
* In some cases the entire test suite may need to be changed to fit the new version of the TAS
28
# 4.2 Risk Assesment and Mitigation Strategies
Maintenance Deployments
| Part II
When maintenance deployment occurs, the following steps are needed:
1. Make an assesment of the changes in the new version of the TAS compared to old one
2. Test the TAS for both new functionality and regressions
3. Check if the test suite needs to be adapted to the new version of the TAS
29
# 4.2 Risk Assesment and Mitigation Strategies
Update Risk and Mitigation
| Part I
An update incurs the following risks and corresponding mitigation actions (1):
* The test suite needs to change to run on updated TAS:
- make the necessary changes to the **test suite** and test them before deploying them on to the TAS
* Stubs, drivers and interfaces used in testing need to change to fit with the updated TAS:
- make the necessary changes to the **test harness** and test it before deploying to the TAS
30
# 4.2 Risk Assesment and Mitigation Strategies
Update Risk and Mitigation
| Part II
An update incurs the following risks and corresponding mitigation actions (2):
* The **infrastrucure** needs to change to acommodate the updated TAS:
- make an assessment of the infrastructure components that need to be changed, perform the changes and test them with the updated TAS
* The updated TAS has addtional **defects or performance issues**:
- perform an analysis of risks vs. benefits
- be sure to create a release note with known issues to notify the test automation engineers and other stakeholders and try to get an estimate on when the issues are going to be fixed
31
# 4.3 Test Automation Maintenance
Test Automation Maintenance
| Part I
* Developing test automation solution is not trivial
* TAS needs to be:
- modular
- scalable
- understandable
- reliable
- testable
* To add even more complexity, TAS has to evolve
32
# 4.3 Test Automation Maintenance
Test Automation Maintenance
| Part II
* To ensure reliable and safe operation of the TAS, maintaining the TAS can be done by:
- adapting it to new types of systems to be tested
- accommodating support for new software environments
- making it compliant to new laws and regulations
* It also optimizes the life span and the performance of the TAS
* Whether due to internal changes or changes in the environment in which they operate, maintenance is an important aspect of architecting a TAS
33
# 4.3.1 Types of Maintenance
Types of Maintenance
| Part I
Maintenance is done on an existing operatinal TAS and is triggered by:
* modifications
* migration
* retirement of the system
34
# 4.3.1 Types of Maintenance
Types of Maintenance
| Part I
* Preventive
* Corrective
* Perfective
* Adaptive
35
# 4.3.1 Types of Maintenance
Preventive Maintenance
Changes are made to:
* Make the TAS support more test types
* Test on multiple interfaces
* Test multiple versions of the SUT
* Support test automation for a new SUT
36
# 4.3.1 Types of Maintenance
Corrective Maintenance
* Changes are made to correct **failures** of the TAS
* The best way to maintain a TAS in operation, thus reducing the risk in using it, is through the execution of regular maintenance tests
37
# 4.3.1 Types of Maintenance
Perfective Maintenance
* The TAS is **optimized** and non-functional issues are fixed
* They can address the performance of the TAS, its usability, robustness or reliability
38
# 4.3.1 Types of Maintenance
Adaptive Maintenance
* As new software systems are launched in the market, it may be required that the TAS supports them
- it may be the case that the TAS needs to comply with new laws, regulations or industry-specific requirements
* In this case, changes are made to the TAS to amke it adapt accordingly
* Conformance to laws and regulaations creates mandatory maintenance with specific rules, requirements and sometimes auditing requirements
* As integrating tools are updated and new versions created, tool integration andpoints need to be maintained and kept functional
39
# 4.3.2 Scope and Approach
Scope
| Part I
* Maintenance is a process that can affect all layers and components of a TAS
* The scope of it depends on:
- the size and complexity of the TAS
- the size of the change
- the risk of the change
40
# 4.3.2 Scope and Approach
Scope
| Part II
* Given the fact that maintenance refers to TAS in operation, an impact analysis is necessary to determine how the system may be affected by the changes
* Depending on the impact:
- the changes need to be introduced incrementally
- tests need to be carried out after each step to ensure the continuous functioning of the Tas
* Maintaining the TAS can be difficult if its specificatins and documentation are outdated
41
# 4.3.2 Scope and Approach
Maintaining the TAS
| Part I
* Time efficiency is the main contributing factor to the success of test automation
* Good practices for maintaining the TAS (1):
- deployment procedures and usage of the TAS must be clear and documented
- third party dependencies must be documented, together with drawbacks and known issues
42
# 4.3.2 Scope and Approach
Maintaining the TAS
| Part II
* Good practices for maintaining the TAS (2):
* Test automation solution:
- must be modular, so parts of it can be easily replaced
- must run in an environment that is replaceable or with replaceable components
- must separate test scripts from TAF itself
- must run isolated from the development environment, so that changes to the TAS will not adversely affect the test environment
- togehter with the environment, test suite nad testware artifact must be under configuration management
43
# 4.3.2 Scope and Approach
Third Party components and Other Libraries
Considerations for the maintenance of the third party components and other libraries:
* TAS can use third party components to run the tests
* TAS depends on third party libraries (e.g. UI automation libraries)
* All the third party component parts of the TAS must be documented and under configuration management
* It is necessary to have a plan in case these external components need to be modified or fixed
* There must be documentation regarding the licence under which the third party components are used, so that there is information on whether they can be modified, to what degree and by whom
* For each of the third party components, it is necessary to get infomration about updates and new versions, which pays-off in the long-term
44
# 4.3.2 Scope and Approach
Naming standards and Other Conventions
| Part I
* The idea of naming standards and other conventions help the test suite and the TAS itself has to be:
- easy to read
- understand
- change
- maintain
* Benefits of introducing naming standards and other conventions:
- saves time in maintenance process
- minimizes risks of introducing regression, or wrong fixes
- easier introduction of new pepole to the test automation project
45
# 4.3.2 Scope and Approach
Naming standards and Other Conventions
| Part II
The naming standards can refer to:
* variables
* files
* test scenarios
* keywords
* keyword parameters
46
# 4.3.2 Scope and Approach
Naming standards and Other Conventions
| Part III
Other conventions refer to:
* Pre-requisites and post-actions for test execution
* content and format of test data
* test environment
* status of test execution
* execution logs and reports
When starting a test automation project, all standards and conventions must be:
* agreed upon
* documented
47
# 4.3.2 Scope and Approach
Documentation
| Part I
* Documentation for both, test scenarios and the TAS, should be good and up-to-date
* There are challenges related to documentation:
- someone has to write it
- someone has to maintain it
* The code of the test tool can be:
- self-documenting
- semi-automatically documented
48
# 4.3.2 Scope and Approach
Documentation
| Part II
* Other documents need to be updated manually:
- design
- components
- integrations with third parties
- dependencies
- deploying procedures, etc.
* It is a good practice to introduce the writing of documentation as part of develop[ment process
- a task should not be considered as done unless it is documented or the documentation is updated
49
# 4.3.2 Scope and Approach
Training Material
| Part I
* If the documentation for the TAS is well-written, it can be used as a basis for training material of the TAS
* Training material is a combination of:
- functional specifications of the TAS
- design and architecture of the TAS
- deployment and maintenance of the TAS
- usage of the TAS (user manual)
- practical examples and exercises
- tips and tricks
50
# 4.3.2 Scope and Approach
Training Material
| Part II
* The maintenance of the training material consist of:
- initial writing
- periodical review
* Done by the team members designated as trainers on the TAS and it most likely happens towards the end of a lifecycle iteration of the SUT (e.g. at the end of sprints)
ISTQB TAE
flashcards
Decks in class (2)
# Cards
