Two kinds of resampling
1) Randomisation
–> Each resample is obtained by randomising the actual data without replacement, as if the null hypothesis were true (getting rid of all group effects by randomising the sample).
2) Bootstrapping
–> Essentially the same but:
o Sample with replacement
o Do it in such a way we preserve any effect in the data
o Effectively creates a large “pseudopopulation” of statistics that is distributed exactly like the sample
o Use confident limits (either of the extreme values of a confidence interval) rather than p values
Advantages/disadvantages of parametric/non parametric tests
• Non-parametric advantages:
o Tests place less restrictive assumptions on data
o May be more powerful if parametric assumptions badly violated
o • Parametric tests also test for significant differences between group means, whereas resampling methods can test significant differences on other statistics such as the median.
• Non-parametric disadvantages:
o Tests less powerful if parametric assumptions met approximately
o Tests more complex null hypotheses
What is the logic of randomisation tests?
- The randomisation process (generally) is done in such a way that it will, on average, remove the hypothetical effect of interest in these resampled samples.
- One calculates the statistic of interest in each of the resampled samples and so can construct a distribution of the statistic, across the many resamplings, which thus generates a distribution of the statistic under the null hypothesis One can then test whether the value of the statistic in the actual sample is sufficiently unlikely (judged by its percentile value within the resampled distribution) to have occurred by chance.
5 things that affect power in resampling
running 1 or 2 tailed test- 1 tailed increases power
effect size- larger effect size increases power
changing alpha level- lower alpha level decreases power
standard deviation- smaller standard deviation increases power
uneven groups- uneven groups reduce power
