Chapter 2

Question 1

Vocab

Measurement

Answer 1

The process of quantifying and evaluating the attributes of a software product or process to assess its quality, performance, and progress.

Used to assess key-performance-indicators (KPIs) and metrics

Examples include:
* Time To First Byte (TTFB)
* Peak Response Time
* Throughput
* Error rates
* Resource utilization (e.g., CPU, memory, disk I/O)

Question 2

Vocab

Metric

Answer 2

Derived from measurements

A standardized unit of measurement that provides a snapshot of the system’s behavior under various loads, scenarios, or conditions.

Question 3

Why are measurements and metrics necessary?

Answer 3

• Defines goals for
• Evaluates the results of

performance testing

Question 4

What are the 6 project risks that can occur without well defined measurements and metrics?

Note project vs product risks from the CTFL material

Answer 4

• Unknown if performance levels are acceptable for operational objectives
• Performance requirements aren’t defined in measurable terms
• May not be possible to ID trends that can predict lower levels of performance
• Cannot compare actual to expected test results due to lack of baseline
• Results are subjective rather than objective
• Results are not understandable, thereby breaking one of the principles

Question 5

For Fun

Define SMART goals

Helpful for testing and even personal/professional growth

Answer 5

• Specific - Clearly define what you want to achieve, avoiding vague or general statements.
• Measurable - Quantify your goal, allowing you to track progress and measure success.
• Achievable - Ensure the goal is realistic and attainable based on your resources and capabilities.
• Realistic - Align your goal with your values and long-term objectives, making it meaningful and relevant.
• Time-bound - Set a specific deadline or timeframe for achieving the goal, providing focus and motivation.

Question 6

What are the three contextual environments of perf testing?

Answer 6

• Business
• Operational
• Technical

Question 7

What are some examples of technical environments?

Answer 7

• Web-based
• Mobile
• IOT
• Desktop client devices
• Server-side processing
• Mainframe
• Databases
• Networks
• Embedded

Question 8

What are some examples of technical metrics?

Answer 8

• Response time
• Resource utilization
• Throughput rate of key transaction
• Batch processing time
• Numbers of errors that would impact performance
• Completion time
• Background load on shared resources
• Software metrics (such as complexity)

See 2.1.2

Question 9

What are some examples of business metrics?

Answer 9

• Business process efficiency
• Throughput of data, transactions, and other units of work
• Service level agreements (SLAs) and any violations
• Scope of usage
• Concurrency of usage
• Timing of usage

See 2.1.2

Question 10

What are some examples of operational metrics?

Answer 10

• Operational processes
• System restoration
• Alerts and warnings

Things that aren’t user-facing in nature

See 2.1.2

Question 11

What is the goal-question-metric (GQM) approach to metric selection?

Answer 11

1. Establish the goals
2. Ask questions to know when the goals have been achieved

Metrics are associated with each question to ensure the answer is mesurable.

The process may be iterative, as more or better metrics might need to be derived.

Question 12

Define the “probe effect”

Answer 12

When the metrics collection effort impacts the performance of the system. The effort should no more than minimally impact.

Question 13

Is it ok to use more than one tool to collect metrics?

Answer 13

Multiple tool usage is common enough, but care must be taken. Redundant metrics might be collected and other problems introduced.

Question 14

What are 3 sources of performance metrics?

Answer 14

• Performance Test tools
• Performance Monitoring tools
• Log analysis tools

Question 15

Describe Performance Test Tools

Answer 15

• They all provide measures and metrics.
• Some report in a text format, others use graphics.
• Many allow the ability to export results for reporting.

Question 16

Describe Performance Monitoring Tools

Answer 16

• Often used to supplement performance test tools.
• Can monitor on an ongoing basis and alert sys admins to threshold violations and errors.
• These can also be used to detect and report suspicious behavior.

Question 17

Describe Log Analysis tools

Answer 17

These can scan server logs and compile metrics.

Errors, alerts, and warnings are usually found in server logs. Some of the events recorded are:
* High resource usage
* Memory errors and warnings
* Deadlocks and multithreading issues (especially for databases)
* Database errors

Question 18

Difference between functional and non-functional (performance) test requirements?

Answer 18

Performance requirements usually aren’t directly articulated by the business the way functional requirements are.

Additionally, performance issues can be hidden under seemingly good results, such as resourcing issues.

Results from performance tests will always rely on the context and type of test being run.