1
Q
Evaluation 1
A
- Evaluation is a process of systematically assessing a system, guided by questions about the system and criteria
- Does the system work as it is supposed to work?
- Does it meet certain standards?
- Is the performance improved over a previous version?
2
Q
Evaluation 2
A
In HCI, evaluations are concerned with how well users can learn and use a system to achieve their goals, and how satisfying the system is to use.
- Can users complete the tasks the system is meant to support?
- Can users achieve their goals faster or with less effort, compared to earlier versions, or competing products?
- Would a first-time user be able to figure out how to use the system?
- Has a change made in interface had the desired impact?
3
Q
Why Evaluation is important
A
- Motivation
- Developers are often not aware of the effect their system has on users
- People are often vague about their goals and expectations
- Developers often see no need for evaluation when the system “works”, but there are always ways to improve it!
- Getting the right design / getting the design right
- Comparing different ideas against each other, for example by making prototypes and simulating functionality
- Identifying usability problems in the design of an interface, to improve it
- Economical importance
- Evaluating usability as differentiating factor / Assesing return on investment
4
Q
Objectives & Criteria
A
Evaluations are a goal-directed process for which objectives and criteria need to be clear\
- What aspects of the system and user experience is being evaluated? For example:
- Accessibility of the system
- Design alternatives for the interface
- What the user thinks and feels about the system
- Impact of small changes
- Criteria (for example)
- Compliance with rules, guidelines, standards
- Performance against metrics
- Identification of problems
- Insight into how users use a system and reason about it
5
Q
Forms of Evaluation #1
A
- Reviews
- Informal or formal, for example using scenarios, guidelines or checklists
- By the design team and/or external reviewers (usability experts)
- Using models
- Formal models of interactive systems, for example to analyse complexity of tasks, and whether goals are reachable
- Models that predict joint performance of human and system
- Evaluation with experts
- Having usability experts try out and critique with the system
- Expert walkthrough, putting themselves in the shoes of a user
- Evaluation with users
- Usability tests, interventions, experiments
6
Q
Forms of evaluation
A
7
Q
Formative / Summative
A
8
Q
Evaluating an Axe
A
- Analytical evaluation identifies the object properties
- “If the axe does not cut well, what do we have to change?”
- Empirical evaluation helps to understand the tool and it’s properties in the context
- “Why does the axe have a special-shaped handle?”
9
Q
Evaluation - Key Points
A
- Evaluation is fundamental in HCI research
- Can be approached analytically or empirically
- with users, with experts, or model-based
- Can have a focus on qualitative and/or quantitative data
- Allows us to assess how users perform with a system and how the system affects users
- It enables improvement of a system, at all stages of the development
- … must be carefully considered to be able to draw the right conclusions!
10
Q
Models
A
- Models are useful for understanding and analysing behaviour of interactive systems
- Models are not perfect: as simple possible, as complex as necessary
11
Q
Descriptive models
A
- Describing a system or phenomenon
- Providing a structure for understanding, designing, specifying, analysing (classification, taxonomy, process model, …) - e.g. 7 Stages of Action, Layered Model of Input
12
Q
Predictive Models
A
- Mathematical model of system behaviour
- Makes it possible to simulate and predict future behaviour
13
Q
Cognitive Modelling
A
- Cognitive models in HCI are based on viewing humans as rational informational processors
- Modelling goal-directed interaction
- Applies for routine tasks with clear goals
- Cognitive modelling supports analysis of tasks
- How can the user accomplish a goal?
- What steps are involved?
- How complex is the task?
- Are there different ways of doing it?
- Some models extend to the prediction of performance
- How long does it take to complete a task?
14
Q
Hierarchical-Sequential Organisation
A
15
Q
GOMS
A
GOMS is a formal model that describes interactions in terms of Goals, Operators, Methods and Selection rules.
16
Q
Goals in GOMS
A
Description of what the user aims to accomplish
Action-object pair, e.g. print-document, delete-word
17
Q
Operators in GOMS
A
Actions that cause change in the system
- Elementary operations on the interface
- Action-object pair, e.g. voice-command, click-button
- Defined by the basic operations an interface supports
18
Q
Methods in GOMS
A
Sequence of operators to achieve a goal
- Well learned sequence of steps to accomplish a goal
- Specific way to complete a (sub-) task with the system
19
Q
Selection Rules in GOMS
A
Rules that define the choice of method
- Only when there are different methods for achieving the same goal
20
Q
GOMS Model
A
21
Q
Example: ATM
A
22
Q
Cognitive modelling - Key Points
A
- Cognitive modelling is useful for analysing and predicting human performance with interactive systems, without actually having to observe users
- Using models of human performance as approximation
- GOMS models task structures and support task analysis:
- Capturing procedural knowledge -> which steps in which order
- More steps to accomplish a goal -> indicates more effort
- More depth in the goal structure -> indicates higher complexity and demand on short-term memory
23
Q
Evaluation using GOMS
A
- Analytical
- Analysing procedures using formal models
- Formative
- Generally motivated for improvement
- Finding problems in task structures
- Analyse performance of alternative solutions to inform choices
- Possible early in design but also on (parts of) complete system
- Qualitative
- GOMS provides qualitative description of procedural knowledge required for system operation (KLM, in comparison, is quantitative)
- Objective
24
Q
Expert Evaluation #1
A
- Expert evaluations involve somebody with experience in interaction design and usability look over the system
- Expert evaluation utilize the knowledge of specialists such as usability professionals to find usability problems in a formative process.
- Expert evaluations are not a replacement for evaluation with users, but they can be easier to arrange and help address basic issues before evaluation with users
- Experts will find problems based on their experience, and will suggest improvements and make recommendations
- Depending on method, expert reviews might also include review other materials, such as product specifications or system logs.
25
Expert Evaluation #2
Expert evaluations are inspection methods in which experts walk through the system
* Using representative tasks
* Concrete examples of what a user would
want to accomplish with the system
* Considering the context of use
* e.g. based on scenarios about people using the system, conveying people’s motivation for using the system, and the kinds of situation in which this would happen
* Adopting the perspective of a user
* e.g. based on personas that are concrete representations of the type of person the
system is designed for
26
Heuristic Evaluation #1
- Heuristic evaluations referes to examination of a user interface by experts against a list of principles or guidelines
- Originally developed by Jakob Nielsen as discount usability engineering
- In the early days of HCI, more reluctance of business to invest in usability and concern over costs of studies with actual users
- Instead, a small number of experts examine the interface and judge its compliance with "heuristics" for good design
- Relatively quick and effective for finding most of the problems
27
Heuristic Evaluation #2
- Now used for any "quick and dirty" evaluation where the aim is to get useful and informed feedback as soon as possible
- Quick review against usability principles by a single reviewer
- Rigorous process with multiple evaluators, and rating of problems
- Not really effective for evaluating one's own design
28
Heuristic Evaluation #3
- A small group of experts examine the interface and judge its compliance with recognized usability principles (the "heuristics")
- Either just by inspection or by scenario-based walkthrough
- Evaluators independently produce a list of critical issues, weighted by severity grade
- Evaluators only communicate afterwards
- Opinions are consolidated into one report
29
Heuristic Evaluation #4
- Implicit assumptions
- There is a fixed list of desirable properties of user interface (the "heuristics") that can be checked by experts with a clear result
- The issues that experts identify are a good prediction of the problems that users would have
- If the number of evaluators is sufficiently large, most problems will be found
30
Nielsen's 10 Heuristics
1. Visibility of system status
2. Match between system and the real world
3. User control and freedom
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
8. Aesthetic and minimalist design
9. Help users recognize, diagnose, and recover from errors
10. Help and documentation
31
Other Heuristics
- Heuristic evaluation can be done against any list of heuristics, and designers and evaluators can adapt these - For example, Cognitive Design Principles by Jill Gerhardt-Powels
1. Automate unwanted workload
2. Reduce uncertainty
3. Fuse data
4. Present new information with meaningful aids to interpretation
5. Use names that are conceptually related to function
6. Group data in consistently meaningful ways
7. Limit data-driven tasks
8. Include in the displays only that information needed by the user at a given time
9. Provide multiple coding of data when appropriate
10.Practice judicious redundancy
32
How many Evaluators?
Nielsen & Landauer's model for the finding of usability problems
- Suggest use of 3-5 experts for a Heuristic Evaluation
- Widely used to argue that it is sufficient to evaluate a user interface with 5 people
33
Limitations of Heuristic Evaluation
Woolrych and Cockton (2000) conducted a large-scale trail of Heuristic Evaluation showing limitations to finding problems users actually experience.
- Evaluators were trained in using HE, and then evaluated the interface of a drawing editor. The interface was then trialled with actual users
- Many of the issues the experts identified were not experienced by users (false positives)
- Many false positives stemmed from a tendency by the experts to assume people had no intelligence, or even common sense
- Some more severe problems were missed by the evaluators
- Problems users experienced as a result of a series of mistakes and misconceptions rather than isolated misunderstandings
34
Heuristic Evaluation - Key Points
* Heuristic evaluation is quick, cheap and easy to do
* Effective as formative method
* Identifying issues for improvement early in development
* HE is subjective and depends on experience
* Relies very much on the knowledge, creativity and experience of the evaluator
* HE has limitations in predicting problems
* “Usability checklists and inspections can produce rapid feedback, but may call attention to problems that are infrequent or atypical in real worlds use.” (Rosson, Carroll & Hill, 2002)
* HE is not a replacement for evaluation with users
35
Cognitive Walkthrough
Cognitive walkthrough is a formal method for checking through the detailed design of steps in an interaction
- Focuses on first-time use
- What happens when somebody uses a system for the first time
- Evaluates learnability of a system
- How well does the system support learning through exploration
- Task-oriented
- Requires tasks and walkthrough scenarios
- Fundamental question : "Will users be able to follow this scenario"
- Can you tell a believable story that users would be doing this
- Requires that evaluators have good understanding of user capabilities
36
Input to the Process
Cognitive walkthrough follows a rigorous procedure, starting with gathering input to the process, followed by the walkthrough
* Input to process are:
* A clear understanding of the people who are expected to use the system (can use on personas)
* A set of concrete scenarios representing both common and uncommon but critical sequences of activities
* A complete description of the interface
* Can be a paper prototype
* Must have a comp
37
Walkthrough Procedure
For each scenario, and each individual step (=action) in the scenario, the analyst asks the following four questions:
- WIll the people using the system try to achieve whatever effect the action has?
- Does the user understand that this step is needed to reach their goal? (Mental model)
- Will they be able to notice that the correct action is available?
- Visibility
- Will they associate the correct action with the effect they are trying to achieve? (or might they select a different action instead?)
- Labels and signifiers
- If the correct action is performed, will people be able to tell that progress is being made towards the goal of their activity?
- Feedback
38
Outcome
Learning about initial user experience and how to improve it
- Do users understand how to carry out tasks
- WIll users really see the right control
- Will they understand this is the right control
- Will they understand labels, signifiers
- WIll they understand the feedback
By the end of this simple procedure, the evaluators will have found some number of missing goals, missing affordances, gulfs of execution, and gulfs of evaluation
39
Research & Ethics
- By conducting research ethics reviews and adhering to guidelines we ensure that the work we are going to do is fair, safe, and does no harm
* (BAD!) Examples:
* Milgram (1963) – Electric Shocks
* Harlow (1958) Rhesus Monkeys – Maternal Deprivation
* Zimbardo (1971) Stanford Prison Experiment – Situational Variables
39
Research & Ethics
- By conducting research ethics reviews and adhering to guidelines we ensure that the work we are going to do is fair, safe, and does no harm
* (BAD!) Examples:
* Milgram (1963) – Electric Shocks
* Harlow (1958) Rhesus Monkeys – Maternal Deprivation
* Zimbardo (1971) Stanford Prison Experiment – Situational Variables
40
Introduction to Ethics #1
* By conducting research ethics reviews and adhering to guidelines we ensure that the work we are going to do is fair, safe, and does no harm
* Promotes moral and social values
* Promotes the principal aim of research works (truth, knowledge, avoid error)
* Promotes value vital for collaborative work (trust, fairness, respect)
* Ensures researchers are held accountable (often research uses public money)
* Builds public support for research work
41
Introduction to Ethics #2
Honesty: Be honest when stating research methods, procedures, or reporting data or results. Avoid any urge to falsify data or methodology. It could come back to haunt you.
Integrity: Keep to your agreements and promises. Be consistent.
Openness: You have to be open to suggestions, new ideas, and criticisms (constructive or not). Share data, resources, tools, and ideas.
Confidentiality: Do not expose confidential records.
Respect intellectual property: Honor copyrights and other intellectual property forms. Seek permission before using unpublished results, data or procedures.
Respect each other!
42
Consent
These principles make sure that participation in studies is voluntary, informed, and safe
- Voluntary: Do not coerce people into taking part, or offer substantial rewards
- Informed: Provide accurate information about what the study entails
- Safe: Do not expose participants to unsafe situations or practice.
We also need to gain consent when using human participants
43
Consent Example
- Participants must be able to make an informed decision
- Understand what the study involves, what data it will collect and how the data will be used, and any risks
- This is often a formalized process
SCC 202 HCI
flashcards
Decks in class (20)
# Cards
-
HCI Introduction17
-
MHP and Time Scales27
-
Vision and Colour21
-
Visual Perception19
-
Movement and Fitts' Law17
-
Input Models20
-
Human Memory19
-
Human Attention19
-
Cognition, Action and Knowledge25
-
Mental Models, Metaphors, Errors23
-
Design Principles24
-
Design Rules, Patterns and Guidelines35
-
Analytical Evaluation44
-
User Feedback43
-
User Studies and UX Metrics35
-
Study Design and Analysis40
-
Intelligent User Interfaces17
-
3D User Interfaces23
-
User Interface Technology24
-
Eye Movement and Gaze as Input18
