Focus Groups
Guided group discussions of 6-12 people
Often used to collected attitudes and relevant experiences related to a specific product, service, marketing campaign, or brand.
Advantages
Good for desires, motivations, values, memory
Participants can bounce ideas off one one another
Limitations
Have to be careful to reduce risk of group think, conversation being dominated by one stakeholder
RITE (Rapid iterative testing and evaluation)
Iterative usability method in which the assets are modified or changed between tests based on each session’s feedback.
Advantages
Great for early stage, lean feedback
Encourages team alignment in design decisions
Limitations
Requires strong team commitment
If testing higher fidelity or functional prototypes - making quick changes can be a challenge
Fixing one problem might introduce several new problems
Literature Review
Provides a concise summary of existing research about a research questions. Can also extend to more detailed synthesis that re-organizes literature and creates an argument.
Advantages
Relatively quick, great return on investment
Limitations
External validity
Paywalls for sources
Structured interviews
Interviews with fixed sequence of questions that are not flexible either by moderator or participant.
Advantages
Can be more efficient in moderation required
Continuity in questions allows for session comparability, and metrics to be collected
Limitations
May be more difficult to build rapport
Limits surfacing of unexpected stories
Intercept Study
A quick way to get a product or concept tested in the wild when you have a specific, narrow set of topics to explore
Advantages
Best for short studies with very little screening required
Limitations
Risky with unreleased products or related assets
Probing not allowed
Dyad (Paired) Interviews
An interview with 2-3 participants in a session together who have a pre-existing relationship.
Advantages
- Captures social dynamics that would otherwise be missed with an individual participant; they can build off of each others answers and keep each other honest
- Easier to build rapport and comfort with participants; especially proven effective with young children and teens
Limitations
- Cancellation, snowball sampling
- Careful about what you introduce (e.g., couples, arguments)
Unstructured Interviews
Interviews with the most conversational flexibility; often the researcher has a set of questions or topics to address, but the conversation may change direction so long as the topics are relevant.
Advantages
- Great for exploration, deep understanding, especially when tackling spaces, experiences, cultures, or settings that aren’t yet fully understood
- Often easier to build rapport
Cons
- Inability to compare sessions or quantify
- Setting stakeholder expectations (don’t know fully what we’ll have answered or surfaced)
KANO model
Method for measuring the effect of different product or service attributes on user satisfaction. Includes a functional and dysfunctional scenario, leades to placement into one of 5 categories (e.g., tablestake, performance, delighters)
Pros:
- Ability to categorize and prioritize product features / roadmaps
- Provides a metric for the interest level of a feature
Cons:
- Short shelf live; expectations change quickly in fast moving markets (what delights today may be a tablestake tomorrow)
- Features need to be non-technical and easy for participants to comprehend
- Issues with hypotheticals in surveys
Playtest
Observing participants interacting with a product or service in an unstructured fashion. Typically used with live builds that have already been extensively QAed.
Advantages
- Allows the holistic study of the natural context of use (progression, likeability), and to surface unexpected painpoints or attitudes.
- Efficiently run a study with a greater sample size
Limitations
- Not great at more deeply understanding “why,” since researcher intervention in the study is limited
- Lots of concurrent and retrospective moderation required; need lab space and technical support for a wider range of participants
Remote, unmoderated research
Method that automates the collection of peoples attitudes and behaviors. A collection of methods (e.g., diary studies, click testing, usability evaluation, card sorting)
Advantages
- Can conduct studies with slightly higher statistical power
- Reduces researcher hours and resources
- Avoids some forms of researcher bias
- Perhaps more ecological validity than lab-based testing
Limitations
- Inability to incorporate probes
- All assets need to be heavily QAed
Individual-centered behavioral mapping
Method that maps the activities and travel of a specific individual or group over time and location.
Advantages:
- Great for illuminating social behaviors and interactions (rather than, say, an office or set space)
Limitations:
- Slightly intrusive, need to allow time for participants to become less reactive to being followed
Cognitive Walkthrough
Usability inspection method that examines the learnability of a new product or system from the perspective of a first-time or one-time user.
You break down the “first session,” including all onboarding, into a series of steps. A team of evaluators then evaluate each step according to 4 key questions (e.g., will users want to produce the action of the step? will users discover the signifier for the affordance (e.g., button, label)?).
Advantages:
- Speedy, lightweight method for quick evaluation
- Helps to prioritize topics for usability testing, since you often won’t have time to evaluate each step
Limitations:
- Complementary to usability testing, not meant to replace
- Requires “thinking caps” and role playing, which may be challenging depending on evaluators
Contextual Inquiry
An ethnographic method that places researcher on-site as a participant in research. The researcher observes and interviews (asks questions), in an apprentice-like fashion, in order to understand communication flows, tasks sequences, tools and artifacts, and physical environments.
Advantages:
- Allows for the examination of tacit knowledge and underlying (and invisible) work structures
Limitations:
- Need to CONSTANTLY check your interpretation to ensure it aligns with the participants
Crowdsourcing
A quantitative method where you allow a large population of people to voluntarily opt-in to complete microtasks.
Microtasks can range from classifying data, answering questions, to providing feedback.
Advantages:
- Quickly and cheaply collect a large volume of data
- When testing hypotheses, representation doesn’t matter (as much)
Limitations:
- Limited ability to collect demographic information, intentions, expertise. Thus, harder to generalize to an overall population.
Narrative Interview
Researcher asks a series of open-ended questions that direct the participant to stories of lived experiences.
Follow-up guidance: who, what, when, where, how. Less of a question-answer format, and more of a narrator-listener format.
Advantages
- A way to collect rich experiences from participants when time constraints (e.g., no time for longitudinal methods or direct observation)
Limitations
- Requires rapport and sensitive handling
Desirability Testing
Iterative method where participants are provided with a fixed vocabulary to select from in order to describe a design. Participants typically select 3-5 cards, and are asked to explain their choices.
Advantages
- Can build consensus when selecting alternative design directions based on how well each elicits intended emotional responses
- Helps non-designers express their feelings towards a product or service
Limitations
- Need clear design hypotheses and goals, from which ideal attributes can be selected
- Interpretations of vocabulary words can vary; good to hear explanations and pick unambiguous terms
Diary Studies
Longitudinal research method where you expose participants to a series of questions and tasks over the course of ~1 week. Participants submit entries throughout the course of that time.
Advantages
- A way to collect in-situ feedback without observation
- A useful way to collect infrequent experiences, or experiences that may be too sensitive for intrusive observation
Limitations
- Ensuring participant adherence
- Lacks the richness of observation
- Best paired with other qualitative methods to be able to use probes
Experiments
A method used to test for causal relationships between variable. Typically you have a control group, followed with 1 or more treatment groups. All other factors should remain constant between the groups, aside from the experimental variable.
3 types of experiments: lab-based, internet-based, and field-based experiments
Advantages:
- The best way to determine causal relationships
- Can be used both to test hypotheses and mitigate risks when developing product updates
Limitations
- Consider the estimated effect size; if small, then consider the extensive sample required to detect the effect.
- If lab or field-based, consider the drawbacks; need for consistency in protocol, whether you need to conduct it blindly, whether you can actually control assignment of participants to groups.
Difference between exploratory and generative research
Foundational / Exploratory / Pathfinding
- Foundational research, sometimes referred to as “exploratory” or “pathfinding” research, centers on a problem or topic that hasn’t yet been clearly defined. It is critical in all research fields, helping teams generate initial understanding, establish priorities, and anticipate and adapt to changes in the competitive landscape.
- User-centered, rather than product or paradigm centered
Generative Research
- Typically comes into gear when you’ve defined a problem space and you’re now looking to prioritize relevant opportunities and solutions
- Very slightly more product centered, since you’re looking for viable product solutions ultimately
Two kinds of generative research
- Projective, Constructive
Projective
- Expressive exercises used at very early stages of design - thoughts, feelings, desired elicited
- Artifact creation can support engaged and comfortable conversation (e.g., collage, drawing, diagramming)
Constructive
- Used at later stages of design and concept development, after parameters have been set for ideating.
- E.g., we’re going to design a calendaring system - flexible modeling can be used to determine appropriate components; set of components will be varied, fixed, and not too wide as to overwhelm them.
Heuristics evaluation
A usability inspection method where you inspect a set of designs according to a set of selected usability heuristics (principles).
Advantages:
- Lightweight, efficient
- Can identify major usability painpoints prior to a usability test (similar to RITE method)
Limitations:
- Ideally requires 3-5 evaluators who are trained on heuristics
- Don’t just call out violations; call out good usage of heuristics as well
Field Research (In-Home Visits)
A method for collecting in-situ feedback within the contexts that people may use or consider products of interest.
Often includes exercises such as home tours, out of box experience, card sorts, generative methods. Debriefs critical between sessions
Advantages
- Participants in their own turf (ecological validity, rapport, comfort)
- Deep, in-situ observation with ability to probe; can cite artifacts and “show” you what they mean
- Stakeholder buy in and participation (developing empathy through immersion)
Limitations
- Time required (planning, stakeholder preparation, travel, participant recruitment)
- Lighter methods to get in-situ feedback; know when this commitment is required
- Personal safety risk (e.g., doing them alone, alergens, etc)
- Observer effect (participants have no preconception of what a home visit entails - may act differently when being observed; place recording devices discretely)
Love Letter and Breakup Letter
Participants are recruited to write letters to product in personified form, and express feelings of dislike and like toward the product.
Advantage
- Can elicit deeper feelings about a product that can help to ground how products and brands should position themselves against empathy and needs
- Works great as part of other qualitative methods or in group workshops
Limitations
- Important to capture their feelings (not just the letter) through video and audio
