User Feedback Flashcards

Question

Pitfalls #2

Answer 1

- Data reported by the users themselves is important as it provides information on their satisfaction with a system, and perception of the interaction with it - Self-reported data can be - Qualitative: for example by asking users open-ended questions about their experience - Quantitative: By using questionnaires as instrument for collecting quantitative data from users - The general format for self-reported metrics is to give users a question (or statement) and ask them to select an answer on a scale - Self-reported metrics are rarely useful just by themselves - Combined with performance data, such as task success and times - Combined with qualitative feedback, which can provide explanations for ratings

Answer 2

- Data reported by the users themselves is important as it provides information on their satisfaction with a system, and perception of the interaction with it - Subjective feedback by users - Self-reported data collected with questionnaires - Asking users a pre-defined set of questions - Similar to structured interview, but on paper or on a computer - Use of rating scales for quantitative analysis - Also qualitative data that can be coded for quantitative analysis

Answer 3

-Questionnaires are a method for data collection from study participants - In general, the term refers to collection of data by giving users a set of pre-defined questions similar to a structured interview but on paper or on a computer - Useful for gathering data from a larger number of people - Can only gather data you know about (unlike observation and interviews that can uncover data)

Answer 4

In quantitative research, the term refers to instruments that measure specific phenomena (perceptions, attitudes, ...) by asking people questions that are carefully designed to meet three criteria - Validity: the question measures what is intended to be measured - Reliability: users will consistently answer the question in the same way - Sensitivity: the questions detects meaningful differences

Answer 5

Igroup Presence Questionnaire (IPQ) - Develop for virtual reality experiences - Measuring the user's sense of presence in the virtual environment - 14 items on 3 factors: - Spatial presence: Sense of being physically present in the VE - Involvement: Measuring attention devoted to the VE - Experienced Realism: measuring subjective experience of realism

Answer 6

- Open-ended questions ("can you suggest any improvements") - Good for general subjective information - Difficult to analyse - Closed questions - single or multiple choice - Restrict responses by supplying alternatives - Easy to analyse - Watch out for 'hard-to-interpret' responses - Alternative responses should be - Mutually exclusive - Exhaustive

Answer 7

- Collection of demographic information on users, and any information about users that is relevant for a study and analysis of the results - Age, gender - e.g., prior experience with the type of interface or application - e.g., handedness - Collection of qualitative feedback - For example using SEQ (Single Ease Question) combined with asking users to give a reason for their rating - Post-test feedback / comments - e.g., "Can you suggest any improvements"

Answer 8

- Always collect demographic data: age and gender - Concise: keep questions simple and as short as possible. - Relevance: each question must be relevant to your study goal. - Precision: don't use vague terms. - Avoid 'loaded' or 'leading' questions that hint at the answer you want to hear - Avoid 'and' questions -> split. - Avoid negative questions (and double-negatives!) - Avoid jargon and abbreviations

Answer 9

- The most common rating scale are Likert Scales, composed of statements to which respondents rate their agreement. - Developed by Rensis Likert, 1932, as general psychometric scale. - A Likert item can be a positive ("The labels used in the interface are clear") or a negative statement (" I found the navigation options confusing") - Respondents specify level of agreement with a statement on a symmetrical agree-disagree scale. - The original Likert scale has 5 points, each with a response anchor: 1 - Strongly disagree; 2 - Disagree; 3 - Neither agree nor disagree; 4 - Agree; 5 - Strongly agree - The range captures the intensity of the subjects' feeling for a given item

Answer 10

Likert Scales - User judge a specific statement on a numeric scale - Usually agreement or disagreement with a statement - Provides quantitative data - Typically 5-point or 7-point scales Also other types of scales, e.g., semantic differential

Answer 11

Statements for Likert scales need to be worded carefully, using unmodified adjectives. - Modifiers such as "very", "extremely", "absolutely" bias the response - e.g., "the UI is extremely easy to use" makes strong agreement less likely than "the UI is easy to use"

Answer 12

- * In usability evaluation, questionnaires and ratings are categorized as as post-task versus post-test * Post-task ratings are completed immediately after finishing a task, to capture impression of the tasks, and are often just a single task-difficulty question: e.g. 7-point “Single Ease Question” (SEQ) * Post-test questionnaires are administered at the end of a session, after completion of all tasks with an interface, to capture how users perceive the usability of the interface as a whole * Post-task and post-test rantings can be complementary

Answer 13

- Widely used scale - Developed by John Brooke, 1986 - 10 statements - 5 Worded positively, 5 negatively - Responses converted to score 0...4 - Added up - Multiplied by 2.5 - Total out of 100

Answer 14

* NASA-TLX (Task Load Index) is a post-task questionnaire for complex interfaces * Developed by NASA for measuring the perceived workload of highly technical tasks of aerospace crew * 6 Questions on an un-labelled 21-point scale, from Very Low to Very High * Complex to score * In HCI it is common to just adopt the mental demand and physical demand questions into custom questionnaires

Answer 15

* Post Study System Usability Questionnaire (PSSUQ) and Computer System Usability Questionnaire (CSUQ) * Like the SUS for post-test rating of any type of interface. Originally PSSUQ, minor changes in CSUQ * 16 items, 7-point scale, positively worded * Provides overall usability score, but also scores subfactors: System Usefulness; Information Quality; Interface Quality * High sensitivity: able to detect differences across a large number of variables (different user groups, types of systems used, years of experience, etc.) * Effective at smaller sample sizes (because of higher sensitivity) * Strong correlation with SUS

Answer 16

* Some aspects of people‘s interaction with technology cannot be measured * Qualitative studies are useful for obtaining rich data from a smaller number of subjects * Qualitative methods can be used for evaluation * … but are often also used to understand the domain and the people who will be using a system before design and construction of the system begins * Qualitative approaches are not by definition better than quantitative approaches (or vice-versa) – What is appropriate depends on what you want to find out… * Qualitative and quantitative data can complement each other

User Feedback Flashcards

(43 cards)