What is the replication crisis?
Psychology studies weren’t replicable. 96% of studies had significant results (r = .4) when they were originally conducted, but only 36% of those studies were able to be replicated (r = .19.
What is replication?
Conducting a study again, usually with the aim of recreating the results of the original study.
3 types of replication
direct, conceptual, replication-plus-extension
direct replication
original study repeated as closely as possible.
purpose: determine whether the original effect is found in new data
conceptual replication
explore the same research question but use different procedures. conceptual variables are the same, but operationalization is different
replication plus extension
researchers replicate the original study but add some variables to test additional questions
what is a meta analysis?
statistical analysis that yields a quantitative summary of scientific literature. examines average effect across all studies conducted on a particular topic.
allows us to make sense of multiple replications
what can lead to failed replications?
contextually sensitive effects.
some original effects are contextually sensitive, and when the replication context is too different, some argue that replication is more likely to fail
how do we counter failed replications?
Many Labs Project (MLP). they conducted 36 replications of each study across many different contexts and found an 85% replication rate
type 1 error
false positive
what can lead to failed replications?
type 1 error, small samples, fraud, questionable research practices
types of questionable research practices (QRPs)
underreporting null effects, p-hacking, HARKing
what is p-hacking
researchers try many different ways of analyzing data, increasing the chance of a false positive
what is HARKing
Hypothesizing After the Result is Known
researchers find an unexpected result, but write about it as if they predicted it from the start
solutions to QRPs
open data & open analysis, power analysis, pre-registration
define open data & open materials
share all data and materials (i.e. surveys, code) publicly
define power analysis
determine sample size in advance to make sure you have enough participants
define pre-registration
publicly registering your hypotheses, methods, and planned analyses in advance
define power
the likelihood of finding a statistically significant effect when the IV truly has an effect in the population
affected by sample size and population effect size
type 2 error
false negative
types of power analysis
post-hoc, sensitivity, a priori sample size analysis
post-hoc power analysis
how much power did we have to test our effects (minimum 80%)
sensitivity analysis
what is the smallest effect we could have detected, given our sample size
a priori sample size analysis
before conducting our study, we can ask how many participants we need to have adequate power to detect an effect if there is one (set parameters to 80%)
