What is the difference between a usability test and experiments?
Usability Testing
- Goal: Check that the system being developed is usable by the intended user population for their tasks
Experiments
- Goal: Test hypotheses to discover new knowledge by investigating the relationship between two or more things
- Involves things like conditions, variables, etc.
What is the experimental Process?
- Formulate a hypothesis
- Identify independent, dependent variables
- Design a controlled experiment
- Check for:
- Confounds
- Validity
- Reliability
- Select representative participants
- Randomly assign to conditions
- Run experiment, collect data
- Analyze results
What is an experiment Hypothesis?
A suggested explanation of a phenomenon
- “If I change A, then B will change in this manner…”
In experimentation, want a hypothesis to be as specific as possible
- makes it easier to test
To test hypothesis, must identify what variables we think will lead to expected outcome
Must identify how manipulating these variables will result in expected outcome
- “If I provide keyboard shortcuts, users will be able to complete the tasks faster than with just menus”
- “If I use pie menus rather than vertical context menus, users will be able to select items faster”
Clearly identify which variables will influence what outcomes, and how
What are independent variables?
Independent variables are those directly manipulated as part of the experiment
Examples
- Menu type: pie or vertical context
- Keyboard shortcuts: available, not available
Everything else should be kept constant
They are not dependent on anything in the experiment, you change them on purpose.
What are dependent variables?
Dependent variables are those that change in response to the independent variables
ExamplesCompletion time
- Error rate
- User preference
- Quality of user response
Their value is dependent on the changes you made to theindependent variables
What is the relationship between independant and dependant variables?
Independent variables are assumed to produce an effect on dependent variables’ values when manipulated
- “If I use pie menus rather than context menus, users will be able to select items faster”
- Pie menus vs. context menus (independent variables)
- Item selection speed (dependent variable)
Only manipulating independent variables increases our confidence that any observed changes in dependent variables due to changes in independent variables
What is the difference between causation versus correlation?
you notice that people seem to be faster with your interface for people who use a mouse, and slower for people with hotkeys
Therefore, the mouse is more effective for your interface than hotkeys. True?
no – what if it was luck? what if there were other variables that you missed? What if you just noticed what you wanted? What if some other reason is why you have mouse vs hotkey users
This is a correlation – you noticed that two things seem to be linked. This does not mean that one caused the other
You can use experimental design to, on purpose, change one variable to see if the other is impacted. Test causation
Causation vs correlation
What is a null hypothesis?
In testing hypothesis, we are seeking to reject the null
hypothesis
Null hypothesis
- There exists no relationship between manipulating the independent variables and the resultant changes in the dependent variables
- Example:
- “There is no difference in selection speed between pie-menus and vertical context menus”
What are nuisance variables?
Any other factors that can affect the dependent variables
Examples
- Time of day
- Handedness
Goal is to have as few of these as possible
Use techniques to mitigate the effect (e.g., counterbalancing)
What are the details needed for experimental design?
Need at least two conditions
- Control condition
- Experimental condition
Control condition
- No experimental manipulations performed
Experimental condition
- Experimental variable is manipulated
Results are compared between two conditions
- Statistics tell you whether the differences are expected randomness or not
Validity
- Are we measuring what we say we are measuring?
Reliability
- If we run the experiment multiple times will we get the same results?
Confounds
- Are there variables we didn’t control for which may be influencing the results we’re obtaining?
What are the types of validity?
- Internal
- External
- Ecological
What is Internal validty and External validity and what is the tradeoff between the two?
Internal validity
- The changes in the dependent variables are caused by the independent variables
External validity
- Results can be generalized to other settings, populations, tasks, etc.
There is often a tradeoff between the two
- The more tightly you control the experiment (to increase internal validity), the less generalizable the results
What is ecological validity?
To what extent do the study conditions mimic those in the real world
Related to external validity, but not the same
What are the options for how participants will be assigned conditions?
Between-subjects
- Each participant does one of the experimental conditions
- Doesn’t account for individual variability
- Need more participants
- No learning effects (good)
Within-subjects
- Each participant completes all experimental conditions
- Better able to account for individual differences Requires fewer participants
- Allows participants to make direct comparative statements
- Learning effects are possible
- To account for these, order of conditions are usually counterbalanced
How many participants should be used?
The more participants the better
- Increases confidence of findings
For large effects, fewer participants needed to achieve statistical significance
- The smaller the effect, the more participants needed
Which is faster?
We don’t know with what’s shown in the previous picture.
Statistics!
General Idea: compare to random chance
What can you do with your data once you have it?
Consult your friendly statistician…
Different tests include:
- t-tests, ANOVA, CHI-squared, sign test important to
- pick the right one for your data
What are the advatages vs disadvantages for controlled experimentation?
Advantages:
- High confidence in the phenomena that we observed
Disadvantages:
- Confidence applies to only narrow phenomena
- Can be difficult to create the right motivation for participants
- Can’t see things like workarounds that participants develop over time in the field
- Results can be statistically significant without being practically significant
Proper study design is very hard – you need expertise, mentorship, iteration with trial and error, etc.
- get a mentor, do an MSc!
Often, a study fails to find a result, but in retrospect you can see confounds, nuisance variables:
- re-design and re-conduct the study to fix it
What are the variables vs dependent variables for the pictures experiment: Keyboard vs Touchscreen
independent variable?
- Interface: existing vs new touch screen
dependent variables?
- Time, error rate, cognitive load (nasa task-load index)
