1
Q
The Butterfly Ballot
A
- Confusion over Palm Beach Count ballot
- The central problem: Poorly designed questionnaire measure which changed course of US history (Bush won over Al Gore)
- Similar challenge for Psychology Research: assess beliefs/moods/intentions/etc., make unobservable into “countable,” assign “values” to psychological states
2
Q
Measuring Psychological Constructs
A
- Converting feelings, attiudes, beliefs, and thoughts into numbers
- tallied, averages, divided, correlated; reported verbally; or observed indirectly (neuroscience)
3
Q
Construct Validity of Surveys and Polls
A
- Many people may have been asked to participate in surveys/polls, method of questioning people
- Online surveys, in person, written interviews
4
Q
What is construct validity?
A
The degree to which a test or measurement tool accurately captures the theoretical concept it is intended to measure
5
Q
Choosing Question Formats
Open-ended questions
A
- People can respond in any way they like and can say anything
- Use their own words
- Unpredictable responses
- An example of a self-report question format
- Ex: As you know, this study is primarily concerned with quality of life. What does quality of life mean to you?
6
Q
Choosing Question Formats
Structured Self-Report Questionnaire
A
- Converts ideas to numbers
- Self report inventories and behavioral ratin scales
- Characteristics: Standard questions, everyone sees the same test, forced-choice format, highly reliable
7
Q
What are the strengths of structured self-report?
A
- Specific administration rules
- Standardized methods for scoring
- Strict guidelines for interpretation
- Strict rules for reporting of results
8
Q
Fixed Alternative Question aka Fixed Choice
A
- respondent selects answer from a set of specified responses
9
Q
Question Formats: Forced Choice
Likert Scales
A
Respondent indicates extent to whcih they agree or disagree with the statement (strongly disagree, disagree, somewhat disagree, etc)
10
Q
Question Formats
Semantic differential format
A
- When there is an anchor at both ends
- Nothing in middle, left to interpretation
11
Q
Wording Questions Well
Question Wording Matters- Delaware Voters ex.
A
- Looked at support for voter ID law
- Randomly assigned one of 3 versions of a question
- Leading questions: wording leads people to answer a particular way
12
Q
Wording Questions Well
Double-barreled Questions
A
- How much do you love vacation and final exams?- these questions have poor construct validity
- Should instead ask one thing at a time, don’t ask participants to evaluate 2 different things with one response
- Break complex questions into 2-3 simpler questions
13
Q
Wording Questions Well
Keeping it simple
A
- Avoiding complexity: one thought or concept at a time
- Avoid negations: “Not” can cause confusion, especially double negatives
- Use similar words instead (not standing–> sitting)
Easter egg: bad one= i do not-should not-have not
14
Q
REMEMBER
A
need to watch ted talk on are we in control of our decisions by dan ariely
15
Q
Wording Questions Well
Question Order
A
- The order of questions can also affect responses
- One of the best ways to control for this is to randomize question order and create several versions of your survey
16
Q
Wording Questions Well
Use informal Language
A
- Consider your intended audience: age/generation, reading skill/level of education, don’t patronize/oversimplify
- No psychology jargon: Hallucinations–>hearing voices, anorexic–>body image, aspirations–>wishes/dreams
17
Q
Wording Questions Well
Forced Choice and Variance
A
- Choosing yes or no: one of two responses is the only option
- Problems: may not provide enough information, may not have enough variance, liking apples vs oranges
18
Q
Wording Questions Well
Variance is Desirable
A
- Don’t want questions that are answered the same way by everyone. Ex. Do you find puppies cute?
- Recall the Goal: to study differences/changes. People’s answers to questions must vary. Hard if everyone gets 100% or 0% on the test
19
Q
Wording Questions Well
Variance is desirable- restriction of range
A
- Restriction of Range: people’s score measure little variation/ few differences
- floor effects: everyone responds at a low level on your measure of DV
- ceiling effects: everyone responds at a high level on your measure of DV
20
Q
Why is restriction of range bad? What is the solution?
A
- Restriction of range is not desirable because it is hard to predict anything based on scores that measure a non-varied DV
- Could be your construct (ex puppies) or your measure
- Solution is to pilot your DV measure
21
Q
Wording Questions Well
Response Bias
A
- Solutions to the problem:
- reverse-worded items
- neutral language in questions
- use “normalizing” questions (questions designed to make a participant feel like their experiences are common or typical)
- stressing anonymity
22
Q
Wording Questions Well
Response Bias
A
- Social desirability bias: answering the way you think is appropriate/expected/desirable, to loaded/sensitive topics
- Response set bias (non differentiation), always giving middle-of-the-road answer (fence sitting), always agreeing (acquiescnce/yea-sayer), alsways disagreeing (naysayer)
23
Q
Wording Questions Well
How to prevent response bias?
A
- Guarantee anonymity: participant answers never linked to their actual identity
- Researchers sees only the groups’ answers or see only participants’ secret code
24
Q
Wording Questions Well
How many questions is enough?
A
- Sometimes: need only ask one questions
- Often-times: need several times to capture construct
- More is better for reliability, idiosyncratic answers tend to average out
25
# Wording Questions Well
Sensitive to diversity
* Not taking typical for granted: your representative sample is heterogeneous; culture/gender/sexual orientation
* Language can't alienate sub-groups: mother --> primary caregiver, spouse --> significant other
26
# Wording Questions Well
Sensitive to personal topics
* Your wife is on the roof: when dealing w/ sensitive topics tread lightly (wording) and use questions that progress in intensity
* Easy/innocuous first, and then difficult/sensitive later to get more honest responses
27
Observer bias
* Observations can often be better than self-report
* Although it's still work to make you observations reliable and valid
* Observer bias- when expectation influence interpretations of behaviors
28
Observer bias example
* participants (psychotherapists) watched the same video of a person answering questions about work experiences
* Group 1 was told this man was a patient, and his behaviors were rated as defensive ad frightened of his aggressive impulses
* Group 2 was told the man was a job applicant, rated him as attractive, candid, and innovative
29
Observer effects examples
* Students w/ maze bright and maze dull rats- rats labeled as bright performed significantly better simply because the researchers expected then to
* Clever hans: when someone or an animal sense what another person wants them to do without being given any deliberate signals. Named after clever hans, a horse that appeared to perform intellectual tasks like arithmetic and reading but was just responding to subtle cues from his trainer
30
How to prevent observer bias?
* Clear codebooks: instructions on how to rate behavior
* Using multiple observers: for interrater reliability
* Masked (blind design): researchers are unaware of purpose of study and/or who is in which group
31
Reducing reactivity
* People change behavior when they know they are being watched
* Blend in: unobtrusive observations, researcher is less noticeable
* Wait it out: let participants "get used" to the researcher, works in weight loss studies
* Measure behavioral results: measure traces of behavior/indicators of behaviors
Psyc 344- Research Methods
flashcards
Decks in class (10)
# Cards
