Usability Evaluation (Week 4)

Question 1

Five Components of Usability (according to Nielson)

Answer 1

1. Learnability
2. Efficiency
3. Memorability
4. Errors
5. Satisfaction

Question 2

Learnability

Answer 2

How easy is it for users to accomplish basic tasks the first time they encounter the design?

Question 3

Efficiency

Answer 3

Once users have learned the design, how quickly can they perform tasks?

Question 4

Memorability

Answer 4

When users return to the design after a period of not using it, how easily can they reestablish proficiency?

Question 5

Errors

Answer 5

How many errors do users make, how severe are these errors, and how easily can they recover from the errors?

Question 6

Satisfaction

Answer 6

How pleasant is it to use the design?

Question 7

Usability

Answer 7

How well a system satisfies its intended use.

Question 8

Usefulness

Answer 8

How much value a system offers to its user.

Question 9

Desirability

Answer 9

How desirable the system is for its user.

Question 10

Why Things are not usable (according to Rubin & Chisnell)

Answer 10

1. Development focuses on the machine or system.
2. Target audiences change and adapt.
3. Designing usable products is difficult.
4. Team specialists don’t always work in integrated ways.
5. Design and implementation don’t always match.

Question 11

How to make things usable (according to Rubin & Chisnell)

Answer 11

1. Early focus on users and their tasks
2. Evaluation and measurement of product usage
3. Iterative design

Question 12

Empirical Methods of Usability Evaluation

Answer 12

1. Usability Testing
2. Performance Evaluation
3. Behavior Evaluation
4. Measuring Satisfaction

Question 13

Non-empirical methods of Usability Evaluation

Answer 13

1. Cognitive modeling
2. Heuristic evaluation
3. Cognitive walkthrough

Question 14

Five Characteristics of Usability Evaluation according to Dumas and Redish (1999)

Answer 14

1. The goal is to improve usability
2. The participants represent real users.
3. The participants do real tasks.
4. You observe and record what the participants do and say.
5. You analyze the data, diagnose the real problems, and recommend changes to fix the problems.

Question 15

Why usability evaluation is NOT a scientific experiment

Answer 15

1. Experimental control is not as necessary
2. Data measurement is not as precise
3. Changes can be made mid-test to explore alternatives
4. The exact same test does not have to be repeated precisely for each user.
5. The number of participants is fewer in usability testing than in scientific experiments.
6. Hypotheses and inferential statistics are used less often.

Question 16

Experimental design (Research) according to Lazar, Feng, & Hochheiser

Answer 16

1. Isolate and understand specific phenomena with the goal of generalization to other problems
2. A larger number of participants is required

Question 17

Experimental design (Usability testing) according to Lazar, Feng & Hochheiser

Answer 17

1. Find and fix flaws in a specific interface, not goal of generalization
2. A small number of participants can be utilized

Question 18

Ethnography (Research) according to Lazar, Feng, & Hochheiser

Answer 18

1. Observe to understand the context of people, groups, and organizations
2. Researcher participation is encouraged
3. Longer-term research method

Question 19

Ethnography (Usability testing) according to Lazar, Feng, & Hochheiser

Answer 19

1. Observe to understand where in the interface users are having problems
2. Researcher participation is not encouraged in any way
3. Short-term testing method

Question 20

Ethnography and experimental design (Research) according to Lazar, Feng & Hochheiser

Answer 20

1. Used to understand problems or answer research questions
2. Used in earlier stages, often separate from (or only partially related to) the interface development process
3. Used for understanding problems

Question 21

Ethnography and experimental design (Usability testing) according to Lazar, Feng & Hochheiser

Answer 21

1. Used in systems and interface development
2. Typically takes place in later stages, after interfaces (or prototypes) have been developed
3. Used for evaluating solutions

Question 22

Formative vs Summative Tests

Answer 22

Formative tests help “form” a design.
They are quick and dirty and run during many stages of development.

Summative tests are run “at the sum” of the project.
They give final results about a more advanced prototype.
Usually more formal and thorough than a formative test.
Might lead to a report that you pass to a product management team.

Question 23

Stages of a Usability Test (according to Rubin & Chisnell)

Answer 23

1. Develop the test plan
2. Set up the test environment
3. Find and select participants
4. Prepare test materials
5. Conduct the test sessions
6. Debrief the participants
7. Analyze data and observations
8. Report findings and recommendations

Question 24

Parts of a Test Plan (according to Rubin & Chisnell)

Answer 24

1. Purpose, goals, and objectives of the test
2. Research questions
3. Participant characteristics
4. Method (test design)
5. Task list
6. Test environment, equipment, and logistics
7. Test moderator role
8. Data to be collected and evaluation measures
9. Report contents and presentation

Question 25

Types of Data

Answer 25

1. Objective measurements 2. Behavioral measurements 3. Subjective measurements

Question 26

Objective measurements

Answer 26

Things that can be measured objectively the same by everyone Time, errors, confusions, breakdowns, workarounds, successes, and failures.

Question 27

Behavioral measurements

Answer 27

Mostly your observations Notes about where, when, why and how the above things occurred.

Question 28

Subjective measurements

Answer 28

Users' comments and feedback Often a questionnaire is used at the end.