Performance testing should start at the ________ level and continue until _________ __________.
component (unit) level; system deployment
What is the risk of waiting until the end of a project to implement performance testing?
Performance problems may not be solvable due to time, money, and technological constraints.
List three goals/purposes of performance testing.
- Measuring system performance under given conditions.
- Determining max concurrent user load or transactional volume a system can handle before it fails.
- Providing info to assist in capacity planning for a system.
How are performance requirements normally recorded? Provide an example.
Normally recorded in over-arching requirements (ex. all system responses to users that require longer than 3 seconds must display a wait notification).
How might performance requirements be defined in Agile?
May be documented in non-functional epics.
What is the role of stakeholders when it comes to determining performance requirements? (i.e., what happens without it?)
Lack of stakeholder performance requirements make it difficult for testers to assess test results and determine acceptable performance.
What are the four key tasks in aligning performance testing with the SDLC?
- During requirements definition: specific requirements for performance clearly defined and agreed upon by all stakeholders
- During design and coding: performance factors considered and built into design and subsequent code
- During integration and system testing: any new integration introduces new performance testing opportunities
- During acceptance testing: assess performance within the business or operational context
List some of the challenges with performance testing.
- obtaining definition and agreement from stakeholders
- obtaining funding for testing tools
- obtaining best fit solution for performance testing tool
- accurately profiling load levels
- acquiring skilled test engineers to plan, design, and conduct a realistic performance test
- building a representative performance test environment
What are some of the risks of performance testing? (Long list)
- inadequate performance test design
- incomplete coverage of protocols and connectivity
- inadequate test environments
- inability to apply performance tool correctly
- inadequate coverage of functions
- inadequate amount of user and load data
- lack of defined stakeholder requirements for performance
- lack of agreement on defined stakeholder performance requirements
- lack of appropriate performance metrics
What are some of the benefits of performance testing?
- opportunity to “right-size” the system to handle expected load
- opportunity to plan mitigation steps if loads exceed expected levels
- early test results can help define acceptable levels of performance
- expectations of future growth can be tested to see if system can support it
- system performance weaknesses can be identified and fixed before occurring in prod
- long lead time items can be addressed
What are the six areas of defining a performance test plan?
- Scope
- Strategy and approach
- Risk assessment
- Definition of test objectives
- Responsibilities
- Reporting metrics
When defining a performance test plan, how is scope framed? (List and define five items.)
- Functionality: what features will be included/excluded from performance testing?
- Architecture: which aspects of the system architecture will be included?
- Transactions: what typical user transactions will be included?
- Users: which classes of users should be included on selection factors?
- Data: how much data should be used and how should it be accessed?
When defining a performance test plan, what does strategy and approach define? What might it include?
Defines the way performance testing is to be conducted (may include outsourcing).
When defining a performance test plan, what purpose does risk assessment serve?
Performance testing can be risk-driven to focus on areas that carry the highest risk.
When defining a performance test plan, what is the purpose of defining test objectives?
At a higher level, objectives describe what the performance test is intended to achieve.
When defining a performance test plan, what is the purpose of defining responsibilities?
Both internal and external team roles must be defined; performance testing often requires support from system architects, database analysts, network engineers, etc.
When defining a performance test plan, what is the purpose of defining reporting metrics?
Important to define which metrics are most meaningful to stakeholders to ensure those metrics are tracked.
When defining a specific performance test within an overall test plan, what four areas must be considered? List and define.
- Environments: full-scale environments often difficult to build due to cost and complexity; can use a cloud-based test environment that is rapidly scaled to simulate prod
- Load and throughput profiles: current and predicted profiles must be understood for both load levels and data throughout (peak load times and load levels)
- Operational profiles: describes the functions users are expected to perform on the system and the frequency; profiles are implemented as scripts to be executed by virtual users
- Test data: need representative amounts and types of test data
What are the four areas of conducting performance testing? (Test ________)
- Test preparation
- Test execution
- Test evaluation
- Test reporting
In the performance test preparation stage, what components should be verified as ready for testing? List and define.
- Test environment: run preliminary tests to make sure basic test environments are in place; validate co-existing applications, verify test environment capacities
- Data: all databases and files should be populated with the designed volumes of test data to ensure test are not blocked
- Testware: test scripts, procedures, etc. should be in place and executing correctly
What needs to occur during the performance test execution phase? When are performance anomalies tested?
Test tools are run and execution is monitored. Interpretation of results requires human intelligence. Performance anomalies can be tested in real time or after test is complete.
During the performance test evaluation phase, are test results more definitive or comprehensive? What is the main question performance testing seeks to answer?
Performance testing is more comprehensive and informative, providing a snapshot view.
“Does the system or application meet stated performance goals or requirements?”
What are the two types of reporting in the performance test reporting phase? List and define.
- Risk-based reporting: helps identify which aspects of system performance may carry the most relative risk; helps stakeholders understand where to focus efforts.
- Objectives-based reporting: performance test results are reported with traceability to performance test.
Why are tools essential for performance testing? List four.
- Creating simulated concurrent user load (used for concurrency testing)
- Sustaining high levels of load
- Accurately measuring load levels and system response
- Repeating performance testing whenever needed
