Meta-analysis (managing type-2 error)

Question 1

Define meta-analysis

Answer 1

Meta-analysis is a systematic method for combining the results of multiple similar studies addressing a similar clinical question. It allows for more accurate conclusions to be drawn from a larger pooled number of participants.

Question 2

State the aims of meta-analysis

Answer 2

To estimate the treatment effect with the greatest possible power and precision

Meta-analysis increases the sample size. This reduces the risk of type 2 error/ false negative results.

Meta-analysis produces estimates that better approximate the population parameters

Question 3

State the steps/process of meta-analysis:

Answer 3

1. Set the study question
2. Literature search
3. Study selection
4. Data extraction and quality assessment
5. Statistical analysis

Question 4

Describe Publication bias

Answer 4

• Journals prefer to publish trials with significant positive findings rather than negative or indifferent trial results

Question 5

What meta-analysis tool can be used to identify publication bias?

Answer 5

• Funnel plot – used to try and identify the existence of publication bias

Question 6

Explain funnel plots

Answer 6

Funnel Plots:
* Purpose: Detect publication bias or other small-study effects.
* Description: A scatter plot of effect sizes against a measure of study precision (e.g., standard error).
* Each dot represents a single study.
* X axis = mean difference, (result of study)
* Y axis = standard error, (larger studies are at the top)
* *All studies should fall symmetrically between the blue lines.
* Asymmetry can indicate = Reporting bias / Chance / Study heterogeneity
(asymmetry can suggest studies that showed no effect are missing)

*the bigger the study the smaller the standard error

• Interpretation: A symmetric inverted funnel shape suggests no publication bias; asymmetry suggests potential bias or systematic differences.

Question 7

How are funnel plots interpreted?

Answer 7

• Interpretation: A symmetric inverted funnel shape suggests no publication bias; asymmetry suggests potential bias or systematic differences.

Question 8

Describe forest plots:

Answer 8

• Purpose: Visualize the effect sizes from individual studies and the overall combined effect.
• Description: A graphical display where each study’s effect size is represented by a square (with size proportional to the study’s weight) and a horizontal line (confidence interval). The combined effect is shown as a diamond.
• Interpretation: Provides a clear summary of individual study results and the meta-analysis result, showing consistency or variability among studies.

Question 9

Describe/state the landmarks on a forest plot

Answer 9

Squares – risk ration for each study
Square size – weight of the study results
Lines – 95% confidence interval of risk ratio (`no difference = 1)
Diamond- pooled estimate
Deepest point of the diamond = point estimate of the treatment effect
Vertical line – no difference if 95% confidence intervals

*The forest plot is on a log scale
The risk ratio favours colchicine on the left
Favours control on the right
No difference is 1

Question 10

Describe fixed effects

Answer 10

Assumes meta-analysis is trying to estimate one overall treatment effect.
– One common ‘true’ treatment effect
– Study results vary randomly around this effect

Used where studies match closely in design and methodology.

Trials contribute to estimate according to their weight (bigger trials contribute more information)

Trials contribute to estimate according to their weight (bigger trials contribute more information)
Variability within (but not between) studies is included in the model.

Question 11

Describe random effects

Answer 11

Assumes a different underlying treatment effect for each study.
– A population of treatment effects – of which the studies are a sample

Used where studies do not match (have heterogeneity)

Gives more weight to smaller studies, overall estimate has wider confidence intervals

Variability within AND between studies is included in the model.

Question 12

Describe pooled estimates in fixed effects models

Answer 12

• Pooled estimates
  – For each trial, estimate of treatment effect is multiplied by its trial weight
  – Weighted estimates are added; then the total divided by the sum of the weights
  – Standard error and confidence interval are calculated
  – Null hypothesis = that pooled estimate is zero – (is tested)

Question 13

How are trials weighted?

Answer 13

• Trials are weighted
  – Weight = 1/variance of trial estimate = 1/standard error squared
  – Big trials have low standard error, therefore carry more weight

Question 14

Describe the purpose of
Heterogeneity Tests:

Answer 14

• Purpose: Assess the degree of variability in effect sizes across studies.
  This can determine which model to use.
  Lots of heterogeneity = use random effects model
  Not much heterogeneity = Use fixed effects model

Question 15

State common Heterogeneity Tests:

Answer 15

• Common Tests:
  o Cochran’s Q Test: Tests the null hypothesis that all studies have the same effect size. (whether there is heterogeneity) A significant result suggests heterogeneity.
  o I² Statistic: Quantifies the percentage of total variation due to heterogeneity rather than chance. Values range from 0% (no heterogeneity) to 100% (high heterogeneity).
• Interpretation: Helps determine the appropriateness of using a fixed-effect or random-effects model.
• The test for overall effect = tests the null hypothesis (that the effect size is 0)

Question 16

State the limitations of meta-analysis

Answer 16

Only as good as the available studies
Many potential sources of bias
Pooled estimates may not be directly meaningful/ applicable to real life clinical practice.
Complex

Question 17

How can the limitations of meta-analysis be addressed?

Answer 17

Addressing limitations: clear protocol before starting / full transparent reporting / discussion of bias.