Scientific Methods

Question 1

Pre-Scientific Methods?

Answer 1

Astrology
- Personality assessment based on birth date

Physiognomy
- Personality assessment baed on shape of body, particularly the face

Phrenology
- Personality assessment based on morphology (shape) of skull

Question 2

4 major types of descriptive methods?

Answer 2

LOTS of data!

Life history data
Observer-reports
Test data
Self-reports (surveys)

Question 3

Self-Report?

Answer 3

Asking people questions about their beliefs and behaviours

• Provided by the participant
• Responses to questionnaires

Question 4

Ten-Item Personality Inventory-(TIPI)?

Answer 4

I see myself as: (rate on a likert scale)
1. _____ Extraverted, enthusiastic
2. _____ Critical, quarrelsome.
3. _____ Dependable, self-disciplined.
4. _____ Anxious, easily upset.
5. _____ Open to new experiences, complex.
6. _____ Reserved, quiet.
7. _____ Sympathetic, warm.
8. _____ Disorganized, careless.
9. _____ Calm, emotionally stable.
10. _____ Conventional, uncreative.

Question 5

Self-Report Data, advantages and disadvantages?

Answer 5

Advantages
– Allows study of difficult-to-observe behaviors,
thoughts and feelings
* Who knows better?
– Easy to distribute to large groups

Disadvantages
– Respondents may not be representative
(convenience sampling is tempting)
– Responses may be biased or untruthful

Question 6

Observer report?

Answer 6

Observing behaviour of others

Question 7

Observer Reports:
Who are the Observers?

Answer 7

• Parents, friends, teachers
  – Usually collected by questionnaire or rating
  form
• Trained observers
  – Systematic observations of behavior
• Untrained, participant-observers
  – Class ratings of Trudeau

Question 8

Observer report, advantages and disadvantages?

Answer 8

Advantages
– Capture spontaneous behaviors
– Avoid bias of self-reports

Disadvantages
– Researcher interference
* How naturalistic (vs. artificial) is the observation?
– Rarity of some behaviors
* Research on criminality
– Observer bias & selective attention
– Time consuming

Question 9

Test Data?

Answer 9

Assessing an individual’s
abilities, cognitions, motivations,
or behaviors, by observing their
performance in a test situation
Tests may be written,
physical (e.g.,
cardiogram),
experimental, or
physiological

Question 10

Examples of Kinds of Test Data?

Answer 10

Questionnaire tests
– E.g., IQ

Experimental tests
– Megargee (1969) study of dominance
* Does trait dominance (high vs. low) or gender
predict leadership?
* Paired high and low dominant men and women in
“box repair” task
* 4 kinds of groups:
(1) high dom ♀, high dom ♂
(2) high dom ♀, low dom ♂
(3) low dom ♀, low dom ♂
(4) low dom ♀, high dom ♂

Question 11

Test data, advantages and disadvantages?

Answer 11

*Advantages
* Allows measurement of characteristics
not easily observable, or known to the
participant

Disadvantages
* Must infer that the test measures what
you think it measures
– Validity issue

Question 12

Life history (Case studies)

Answer 12

Intensive examination
of a single person or
group

Question 13

Case Study Methods?

Answer 13

• Obtained from life history (interviews,
  autobiography)
• Other life records (Life Outcome Data)
  – School grades
  – Criminal records
  – Work record
  – Facebook page, tweets, instagram, etc.

Question 14

Case study, advantages and disadvantages?

Answer 14

• Advantages
  – Rich source of hypotheses
  – Allows for studies of rare behaviors
• Disadvantages
  – Observer bias
  – Difficult to generalize (N = 1)
  – Difficult to reconstruct causes from complexity
  of past events

Question 15

Reliability?

Answer 15

Extent to which scores on the measure
are stable and replicable, vs. amount of
error or randomness in the measure

Question 16

Validity?

Answer 16

• Degree to which measure assesses what it is
  supposed to assess

Question 17

Bulls eye analogy?

Answer 17

• Reliability = are you hitting the same spot
  each time?
• Validity = are you actually hitting the bulls
  eye?

Question 18

Measuring Validity (4)?

Answer 18

• Face validity
  – Does it measure what you think it measures?
• E.g., shyness questionnaire
• Predictive validity
  – Does it predict an external criterion?
• Does shuffling predict self-reported shyness?
• Convergent validity
  – Relation to other measures of same variable
• Self-report and observer report should be related
• Construct validity
  – All of the above

Question 19

Inter-rater Reliability and Validity
An Example: Measuring Height (without a ruler)

Answer 19

• Reliability of ratings of height
  – Average correlation between two judges = .76
  – Reliability of 5 judges is about .90
• Validity can only be high if reliability is high
  – If measures are more reliable, they provide a more valid
  assessment!
  – By combining the judgments of multiple people (or using
  multiple items on a personality test) we can get fairly reliable and valid measures of personality (reliabilitiesnabout .80-.90)

Question 20

The Problem:
Incentives Structure?

Answer 20

• Published research is important for
  getting a job, getting tenure, getting
  grants, and being viewed favorably in
  our field
• Result: scientists try to publish as
  much as they can
• Balancing act: need to stay truthful to
  psychological science, but also
  publish
• This results is researchers taking
  shortcuts and sometimes worse…

Question 21

Questionable Research Practices
(QRPs)?

Answer 21

Decisions in design, analysis, and
reporting that increase the likelihood of
achieving a positive result
– And a positive response from editors and
reviewers

Question 22

What should researchers do to avoid QRPs?

Answer 22

• Increase disclosure in methods,
  results, and hypothesis presentation
• Pre-register hypotheses and studies
  – Data collection rules, analytic strategies
• Share data
• Be a responsible scientist regardless of
  outcome

Question 23

Center for Open Science?

Answer 23

Founded to increase to openness,
integrity, and reproducibility of scientific research
– Brian Nosek and Jeff Spies
* Open source software platform for
pre-registering hypotheses,
archiving study materials,
depositing data and syntax

Question 24

Good Research?

Answer 24

• Good research is open research
  – Materials and data are shared publicly
• Good research features experimental
  methods that are strong and isolate a
  question of interest
• Good research is adequately
  “powered” research

Question 25

Power?

Answer 25

* Most psychological effects are small, so you need a lot of participants – Some say “around 200”; others say it depends on what you’re studying and your design – If you’re studying an effect that’s likely to be small, you need a big sample * E.g., Are UBC or SFU students more liberal? – If you’re studying an effect that’s likely to be big, a smaller sample is ok * E.g., Are UBC students or Texan oil tycoons more liberal?

Question 26

What does it mean that Power is generally set at 80%?

Answer 26

– This means there’s an 80% chance of finding an effect that exists – However, studies are often run with much lower power * Researchers underestimate how much data are needed * Effects are smaller than they think * It’s hard and expensive to collect large samples