Reliability
the consistency or repeatability of measures
Reducing random measurement error improves reliability
Validity rides on the back of reliability
Reliability does not guarantee validity
Validity
are we measuring what we are trying to measure?
Test-retest reliability
Used to assess the consistency of a measure from one time to another
The correlation between scores across two administrations of the measure
Parallel forms reliability
Used to assess the consistency of the results of two tests constructed in the same way from the same content domain
- Split-half reliability
- Item-total reliability
Internal consistency reliability
Used to assess the consistency of results across items within a test
Cronbach’s alpha – the average correlation among all possible pairs of items (.80 or above)
Types of validity
- measurement (construct) validity
- external validity
- internal validity
Construct validity
A ”construct” refers to a behaviour or process that we are interested in studying
- > e.g., Depression, Short term memory, Social prejudices
- Do our measures and manipulations (of both IV’s and DV’s) reflect the theoretical constructs of interest
- > Operationalisation of measurement – how we measure a construct
- > Operationalisation of experimental manipulation – how we manipulate an IV
- Both our measures and our manipulations must be valid
- For example - how well does the Beck Depression Inventory measure Depression?
Measurement validity - convergent
- Do scores on the measure correlate with scores on other similar measures related to the construct?
- Relates to the degree to which the measure converges on (is similar to) other constructs that is theoretically should be similar to
Measurement validity - discriminate (divergent)
- Do scores on the measure have low correlations with scores on other different measures that are unrelated to the construct?
- Relates to the degree to which the measure diverges from (is dissimilar to) other constructs that it should be not similar to
Measurement validity - face
- On its face value, does the measure seem to be a good translation of the construct?
- If you ask participants to do some sums – will they understand this will measure their arithmetic ability?
Measurement validity - content
- Does the measure assess the entire range of characteristics that are representative of the construct it is intending to measure?
Measurement validity - criterion
Concurrent
- Do scores on the measure align with scores on other measures (recognised and reliable) as they should
Predictive
-Are scores on the measure able to predict future outcomes (i.e., attitudes, behaviours, performance)?
Manipulations can be
Instructional:
Experimental conditions are defined by what you tell participants
Environmental:
Stage an event, present a stimulus, induce a state
Stooges:
Use fake participants to alter experimental conditions
When doing it right, manipulations will
- Reduce random error (replicate procedure)
- Reduce experimenter bias
- Reduce participant bias
- Ensure manipulation has construct validity
- Do a manipulation check – ask participants about various aspects, beliefs, attitudes etc.,
External validity
- Extent to which the results can be generalised to other relevant populations, settings or times
- Studies have good external validity when results can be replicated:
- > Using alternative operationalisation of variables
- > Measuring a different sample of participants
- > Conducting the research in another setting
External validity - ecological
The extent to which the results can be generalised to real-life settings
External validity - population generalisation
Applying the results from an experiment to a group of participants that is different and more encompassing than those used in the original experiment
External validity - environmental generalisation
Applying the results from an experiment to a situation or environment that differs from that of the original experiment
External validity - temporal generalisation
Applying the results from an experiment to a time that is different from the time when the original experiment was conducted
The replication crisis in Psych - why does this happen
- Replications are uncommon
- In the top 100 psych Journals between 1900-2012, only 1.6% were replications (Makel et al., 2012)
- Substantial bias towards publishing significant findings (and not null findings)
- Times change – especially in terms of areas like social psych
- Alpha cutoffs are arbitrary
The replication crisis in Psych - what to do about it
- Increase in replications
- Pre-registration of studies (if methods are good they will be published regardless of significance)
- Suggestion that we might consider probability based analyses rather than null hypothesis significance testing
Internal validity
- Able to conclude causal relationships from results
- Extent that any effects on the DV were caused by the IV
- Elimination of alternative explanations for observed relationships
- Inferences of cause-and-effect require three elements
- > Co-variation
- > Temporal precedence
- > Elimination of alternative explanations
- Strong internal validity requires an analysis of these three elements
Selection bias
- A threat to internal validity that can occur if participants are chosen in such a way that the groups are not equal before the experiment
- Differences after the experiment may reflect differences that existed before the experiment began
- Differences after the experiment may reflect differences that existed before the experiment began plus a treatment effect
Maturation
- Changes in participants during the course of an experiment or between measurements of the DV due to the passage of time
- > Permanent – e.g., age, biological growth, cognitive development
- > Temporary – e.g., fatigue, boredom, hunger
- Most common is naturally occurring developmental processes (i.e., children)
