Types of multiple comparisons
- before study
- after study
Before study multiple comparisons
Planned comparisons (contrasts)
- A priori
- Break down variance into component parts
- Test specific hypotheses
After study multiple comparisons
Post-hoc analyses
- Compare all groups using stricter alpha values
- This reduces Type I error rate
Planned comparisons
Involves breaking down the variance according to hypotheses made ‘a priori’ (i.e., before the data were collected)
RULES
- Once a group has been singled out – it cannot be used in another contrast
- Each contrast must only compare 2 “chunks” of variation
- There should always be 1 less comparison than the number of groups (i.e., number of comparisons = 𝑘−1)
Planned comparisons - weights and magnitudes
Weights
- Sign
- Magnitude
Rules
- Compare positive against negative weights
- The sum of weights for a comparison should be zero
- If a group is not in a comparison it should be assigned zero
- For any contrast, the weights assigned to the groups or group in one chunk of variation should be equal to the number of groups in the opposite chunk of variation
Types of planned comparisons
Orthogonal Contrasts
Non-orthogonal Contrasts
Standard Contrasts
Polynomial Contrasts
Orthogonal contrasts
- Compare unique “chunks” of variance
Non-orthogonal contrasts
- Overlap or use the same “chunks” of variance in multiple comparisons
- Require careful interpretation
- Lead to increased type 1 error rate
Standard contrasts
- Orthogonal: Helmert and Difference
- Non-Orthogonal: Deviation, Simple, Repeated
Helmert:
Compare each category to the mean of subsequent categories (based on the order they are coded in SPSS, which might be alphabetical!)
Difference:
Compare each category to the mean of previous categories
Polynomial contrasts
- Linear, Quadratic, Cubic and Quartic trends
- Used only when your IV is ordinal
Post-hoc tests
- Involves comparing all possible differences between pairs of means
- Good approach with exploratory research or where there are no pre-defined specific hypotheses
- Simplest post-hoc test is Bonferroni
- Bonferroni correction means:
Bonferroni 𝛼 = 𝛼 / (𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑡𝑒𝑠𝑡𝑠)
Post-hoc tests what is Tukey’s HSD
- Called Tukey’s HSD (Honestly Significant Difference)
- The cumulative probability of a type 1 error never exceeds the specified level of significance (p < .05)
- Supplies a single critical value (HSD) for evaluating the ‘significance’ of each pair of means
Tukey’s HSD details
- The critical value (HSD) increases with 𝑘 (i.e., each additional group mean)
- It becomes more difficult to reject the null hypothesis as a greater number of group means are compared
- If the absolute (i.e., obtained) difference between two means exceeds the critical value for HSD, the null hypothesis for that pair of means can be rejected
- Based on a measure of error variance (𝑀𝑆_𝑊𝑖𝑡ℎ𝑖𝑛), group sample size and 𝑞 value:
𝐻𝑆𝐷 = 𝑞√(𝑀𝑆_𝑊𝑖𝑡ℎ𝑖𝑛 / 𝑛)
Which Post-Hoc?
*look up image
One-Way ANOVA Steps
*look up image
