A user test with paper prototyping is an example of a:
a. Down the hall-way usability test
b. Summative Usability Test
c. Low-fidelity Usability Test
C. Low-fidelity Usability Test
Is usability testing largely a “qualitative or quantitative” research technique?
Qualitative
What are some examples of quantitative observable data?
Time on Task, Success/failure rates, Effort e.g. # of clicks/perception of progress
What are some examples of qualitative observable data?
Stress responses, Subjective Satisfaction, Perceived effort or difficulty
Why metrics?
Metrics is really important because usability testing is not just for you to help refine your design, but it’s also a tool to influence the rest of your team members.
If you can show, you know, 60% of users were struggling with this, or that this feature has a 10% success rate, that can be very powerful – as opposed to saying, “Oh, the users didn’t like it. The users found it difficult.” It’s less believable than if you can actually show the metrics, so I think it helps ground teams in the reality around how users will probably use a design
“Okay, well, let’s look at the stats.” And they also basically turn conversations into fact-based – as opposed to opinion-based – conversations. Again, if people are arguing over a design, you can sort of say, “Well, let’s look at the data; let’s look at how users did.” So, what are we measuring? We’re measuring behavior, opinions and the actual data.
Why shouldn’t you ask your testers “Do you like this design”?
If you ask someone what they like, they’re likely to tell you what they like or what they “think” they like.
What users think and what they do are…
two different things
What are we measuring in usability tests?
- Behavior: Task performance, speed, efficiency, goal fulfillment, expectation matching
- Opinions: How it looks, Thoughts & opinions
- Data: Visitors, Pages, Documents
What are we looking for when we’re observing user testers?
We’re observing if a user has not seen something; see if a user is going in the wrong direction; see if the user is thinking it’s correct when it isn’t correct, or maybe the user is missing a rule.
Not seeing something, Going in the wrong direction, Thinking it’s correct when it isn’t, Missing a “rule”
What is the think-aloud protocol?
When you ask the user to think aloud to tell you what they’re thinking and feeling.
- Test the interface not the user
- Ask them to externalize thoughts/feelings
- Verify researcher mental model
According to Frank Spillers, why is time on task not always the best measure of usability?
a. It’s difficult to measure precisely when a user starts performing a task.
b. How fast a task should be done depends on the context.
c. It is not always easy to divide the users’ actions into tasks.
b. How fast a task should be done depends on the context.
What are the top 3 metrics according to Frank Spillers?
- Success rate - do they get the task?
eg.) 7/10 users completed the task aka “Got it right” - Failure rate - do they not get it?
eg.) 3/10 users committed errors - Partial Success - do they get part of it?
6/10 users had confusions
How many users should you test with?
- You only need 8-15 people for most (Formative) tests. Summative (statistical) tests 20-50+
- The number depends on your user types
- User testing is Qualitative Research (the research rules are different)
If you test with 5 users, that’s called…
Agile test
weekly usability test
Lean UX, agile UX, then do 3 to 5 users one week.. 3 to 5 the next week, 3 to 5 the next week.
and by the end of the month, you have a nice sample of about 15-20 users.
When you’re doing more statistical testing, you will want to use a higher sample size of…
15 to 20 users
even all the way up to 50 or 100 users
What does Ethnographic mean?
It means the study of people.
What does study of people mean in ethnographic?
Means you go out to “their” environment with a field study.
What should you prepare for in a test plan?
- Decide what areas to concentrate on
- Determine potential usability issues
- Write a Test plan
- Determine what tasks will be tested (~12)
What should your test plan include?
- Objective
- Test Task(scenario)
- End state(answer)
- Assets needed (for managing time up to the test. What’s needed for you to prepare for the test)
When you create your test plan, run your scenarios from ____ to ____.
Easy to difficult.
The first task should be an orientation task.
Put difficult tasks in the middle of the Test Plan
*After each user test, ask the user if the tasks were realistic- this will help give you corrective feedback. Writing good test tasks takes time and practice.
When is a good time to ask test participants for their subjective opinions on the product they are testing?
a. Pre-test
b. During test
c. Post-test
c. Post-test
You are running a usability test and your test participant has been trying to solve a task for more than 5 minutes. Finally, the participant gives up and asks you how to solve the task. According to Frank Spillers, you should:
a. Tell the participant what the correct solution is.
b. Ask the user to move on to the next task or encourage him/her to keep trying.
c. Give the participant a simple hint and encourage him/her to keep trying.
b. Ask the user to move on to the next task or encourage him/her to keep trying.
